Contribution feedback system

[Dylan Hsu]

Currently, Zarel is working on a feedback system for the addons section of the website. Users selected by Zarel will have the ability to review other users' contributions and rate on a 5-star scale following rating guidelines established and maintained by Zarel himself.

I'm posting this topic to suggest an alternative method to provide helpful feedback for contributions.

In my system, a group of users are selected as contribution critics, handled conveniently by forum usergroups. These users provide feedback to other users in the form of a like/dislike system - thumbs up, thumbs down, if you will - and can leave reviews for contributed items. Once a non-critic user has received enough thumbs ups and reviews, he becomes a critic and gains the ability to provide feedback on contributions. It thus follows that only individually selected users, along with experienced contributors, will have the ability to provide feedback for user contributions in the addons section.

Like and dislike instead of stars
5-star scales can become very tough to interpret because people have different standards. One critic may rarely if ever give 5's, whereas another may give 5's frequently. A system that allows a critic to either like or dislike a contributed item, akin to RottenTomatoes, allows the user to make a very clear decision regarding whether they like or dislike a contributed item.

In addition, the feedback system can display both the numbers of likes and dislikes very succinctly. Users can draw their own conclusions from the numbers rather than only having an average of 5-star ratings to look at.

User promotion
A system where feedback is provided by only a small group of people is bound to be biased. Feedback will be skewed, especially if there is only another small group of people choosing who gets to give feedback. While it is simply foolhardy to allow everyone to give feedback, users who contribute positively and significantly should with time gain the ability to provide critical feedback as well. Thus, a user can become a critic either by special selection or by contributing positively and significantly.

[KukY]

Cool, a rating system is being worked on!

Now let me comment this:
The fact that rating ability will be limited to selected users is great, as it will prevent a group of newbies coming and crushing a rating of a very good product.
That simple like or dislike rating system is the best, IMO.
And at the end, what I don't agree with:
The frequent contributors getting access to rating. With time, number of those people will grow, and some of those will be in the group of newbies.

[duckblaster]

maybe allow contributors to be removed from the list if they are not fair

[KukY]

maybe allow contributors to be removed from the list if they are not fair

That can only be determined after several unfair ratings, and then you not only have to remove the contributor from the raters list, but also rollback their votes, and by then maybe their unfair votes have rejected some people from someone's work...

[Zarel]

While I will take into account things said in this thread, keep note that things like these are not popularity contests, and I will not necessarily do whatever is voted up the most.

That said, let me give my reasoning for why the system is designed the way it is.

First of all, I noticed YouTube went from a five-star system to a like/dislike system a while ago. They did this because they noticed most their users would vote something either 5, 1, or not vote at all.

The problem with a like/dislike system, though, is that it only gives you a general idea of whether or not something is good or bad. A star system will tell you exactly how good or bad it is, as long as the star rating is accurate. The problem is, most people aren't going to vote accurately. Like on YouTube, they'll vote for what they feel strongly about, and ignore the rest.

I don't want that to happen. A mediocre addon should have a review detailing why it's mediocre. And a good addon doesn't need 100 reviews all saying paraphrases of "This rocks!" A single review that provides details on why it's such a nice addon is far preferable.

The other thing is standardization. It's difficult to teach a constantly-growing pool of people how to review accurately, and make sure that they follow guidelines, and how to deal with special cases, etc etc. YouTube doesn't have the luxury, but we're small enough that a group of 5-6 reviewers can be coached to review all our addons fairly.

[milo christiansen]

Let every registered user vote, this "selected users" is some thing I can't approve of.

[Zarel]

Let every registered user vote, this "selected users" is some thing I can't approve of.

You're going to have to explain why you can't approve of it. Like I said, Warzone Project decisions aren't driven solely by votes.

[KenAlcock]

You're going to have to explain why you can't approve of it. Like I said, Warzone Project decisions aren't driven solely by votes.

First, it is exclusionary and elitist. It basically tells everyone who isn't selected to rate contributions that their input is neither welcome nor worth welcoming. This is not an effective way to win friends, influence people, or grow a gaming community. It goes against everything an online forum is about.


Second and more importantly, it renders the rating system completely useless. Since the size of the data collected is too small, and skewed to the points of view of a few people. The larger the data set collected, the more accurate the rating system. That's just basic Probability and Statistics. I'm mean I could always ask 5 people their view on any subject, but if my population is 500 people, how accurate would my poll be? Answer: It would be completely useless.

It think your rating system should get more detailed and ask questions about the contributions, as well as the the rater. Let's take rating a map for example. The raters should be required to answer questions about themselves and their preferences first.

QA: How many months have you played Warzone?
QB: How many games a month do you average?
QC: Do you prefer: A) Low power maps, B) Medium power Maps, or C: High power maps?
QD: Do you prefer: A) flat maps, B) Maps with lots of varied terrain Features?
...

Then in a second line of questioning, the raters are asked about the map they played.

Q1: From your experience, how do you rate the power on this map?
Q2: From your experience, how do you rate the Terrain on this map?
...

Then you have useful rating information when you aggregate the two sets of answers.

This map is most popular with people who prefer high oil, flat maps, who have played Warzone between 0 and 12 months.

This map is least popular with people who prefer medium and low oil maps with terrain, who have played Warzone 12 months and over.

In addition to the scale of answers (1 through 5 or whatever you use), you should add an option for "I'm not sure/ No opinion" to every question, which essentially tosses out that datum.

[Rman Virgil]

.

My 2 cents on ANY review system...

Its usefulness will depend on it's transparent structure as well the explicit mind-sets and talents those who review bring to the table whether the model is pure egalitarianism & a high statistical sampling OR that of specialized pro staffers from journalistic orgs in print, broadcast or e-media.

Ultimately, however, even the best conventional review structure will mostly serve end-users and those who do the reviewing and not really the creators of Maps & Mods and this is precisely were it differs from You Tube, Amazon, Net Flix or Rotten Tomatoes, to name but a few of the more prominent orgs with review systems.

Let me explain the difference briefly as i see it.

Mappers & modders generally create alone and for pure love of the game with absolutely no possibility of tangible rewards.

If they are diligent, and most are, they will also test their work to the best of their ability. But here's the dilemma - no matter how much mappers and modders test their work alone it will never be enough. Just like the Projects Developers and their work, they need serious competent Beta Testers who can provide positive constructive feedback based on the Modders or Mappers clearly stated, & detailed, design goals so that such further modifications as needed can be made BEFORE the work is showcased as FINISHED - this is the great divide. From there Reviewers can specifically assess the success or shortfalls in the resultant work based on the Mappers or Modders own stated goals rather than the Reviewers own subsumed and unstated biases .... personal likes or dislikes presumed as objectivity.

Now if the Review Structure could somehow manage to serve that really essential need through it's policy, procedure, metrics & the work of its Reviewers then I would be inclined to support or participate.

The review structure must also be explicitly aware of Disraeli's admonition: "There are lies, damn lies... & statistics."

The biggest issue with folks who are actively vocal in the community but do not work on the source or create add-ons or assets is that they are inherently attracted to self-aggrandizement opportunities like the "Ideas & Suggestions" BB which are real easy and require very little effort, time or work - serious beta-testing is the complete opposite of that.

An other thing related to all this:

Even though WZ is open source and a "free game" the basic economic maxim TANSTAAFL still applies. That acronym translates as the saying "there ain't no such thing as a free lunch" which has a 150 year history that began as a 19th century western saloon custom but has since come to be understood in the broader context of meaning that things which appear to be free are always paid for in some way and that this is fundamental to civil & equitable relationships; trading values without coercion, as opposed to barbarism wherein you take what you want by force, the barbaric acquiring valuables they are incapable of creating or making for themselves.

In a FOSS community it's all about actively respecting the values of reciprocity & equity, to my mind, and genuinely appreciating that there is no such thing as a "free lunch". This is were contributions in the form of serious beta-testing come to the fore as a structure that provides for those community of folks who do not code, map, mod, create art or any new game assets .....they can indeed make very, very, important contributions and fulfill their end of the reciprocity equation.

All organizations can foster a set of cultural values either by wide-eyed commission or by negligent omission. In the case before us it can be easy self-aggrandizement opportunities for those who make nothing or opportunities for those who are not makers to reciprocate in a most tangible way via tasks absolutely essential to promote the creative vigor that is at the heart of making a growing body of New Assets or Add-ons.

- Regards, RV

.

[Zarel]

First, it is exclusionary and elitist. It basically tells everyone who isn't selected to rate contributions that their input is neither welcome nor worth welcoming. This is not an effective way to win friends, influence people, or grow a gaming community. It goes against everything an online forum is about.

This is probably the biggest problem I'm worried about. I think what I'll end up doing is having two rating systems - an "Official Warzone Rating" and a "Community Rating" that anyone can vote.

Second and more importantly, it renders the rating system completely useless. Since the size of the data collected is too small, and skewed to the points of view of a few people. The larger the data set collected, the more accurate the rating system. That's just basic Probability and Statistics.

This, on the other hand, isn't true at all. Here, let me teach you Statistics 101. Larger datasets only result in more accurate results if your sample is a random sample.

If you ask some people to rate something 1-5, and you average their ratings, that's a random sample you will come up with an accurate score. You can usually expect at least 90% of people you ask to give a sincere answer, so that's a selection bias of at most 10%.

Note, however, that a vote button on a website is not the same as asking someone. Most people will not vote unless they feel strongly about a map, which will be maybe 5% of viewers, and that is what we, in statistics, call a 95% selection bias. Having a carefully selected reviewer pool and instructing them to rate all addons regardless of how strongly they feel about them prevents that form of bias.

I'm mean I could always ask 5 people their view on any subject, but if my population is 500 people, how accurate would my poll be? Answer: It would be completely useless.

It depends on who those 5 people were. If, for instance, your question was "Which of the five axioms of Euclidean geometry hold in a hyperbolic plane?" you would have far better results asking 5 mathematicians than 500 people selected randomly from the human population.

It's the same idea here. Honestly, as much as I want everyone to participate, I really don't want someone who thinks the map preview is ugly but has never actually played the map to vote. And the guy who voted 5.0 just so he could get his review of "help how do i install maps" out there? I wouldn't mind if he was left off the sample, too.

Making sure the sample consists of people who understand the review guidelines, can be trusted to have played a game or two on the map/mod, will result in far better results than the famed inaccuracy of what would essentially be an online poll. Do you have any ideas about how to prevent the map creator from creating 50 accounts and voting his map up? Because it's impossible to prevent that sort of thing if you let everyone vote.

I mean, the choice is between having the honest opinions of a few reviewers, or a number that represents the mean of what a bunch of people clicked for reasons undisclosed. To me, the former number is more likely to be accurate.

Ultimately, however, even the best conventional review structure will mostly serve end-users and those who do the reviewing and not really the creators of Maps & Mods and this is precisely were it differs from You Tube, Amazon, Net Flix or Rotten Tomatoes, to name but a few of the more prominent orgs with review systems.

I agree. The reviews are meant to assist users decide whether or not to download a map. Their assessment may also provide feedback to the mapmaker, but that is not its primary purpose.

Mappers/modders looking for feedback should ask in the Mapping/Modding forum.

[Dylan Hsu]

I agree. The reviews are meant to assist users decide whether or not to download a map. Their assessment may also provide feedback to the mapmaker, but that is not its primary purpose.

Mappers/modders looking for feedback should ask in the Mapping/Modding forum.

RV's entire post seems to have flown right over your head. It's supposed to serve mappers and modders too; they should be able to get beta feedback, etc. ...

Do you have any ideas about how to prevent the map creator from creating 50 accounts and voting his map up? Because it's impossible to prevent that sort of thing if you let everyone vote.

Restrict multiple forum accounts on the same IP.
Restrict voting attempts from the same IP.
Not everyone should be able to review. Knowing we are on the same page about this, if a user must have received significant reviews and thumbs-ups to have the ability to review in the first place, how is a devious user going to promote 50 multi-accounts to contribution critic status, then bump himself up?

Note, however, that a vote button on a website is not the same as asking someone. Most people will not vote unless they feel strongly about a map, which will be maybe 5% of viewers, and that is what we, in statistics, call a 95% selection bias. Having a carefully selected reviewer pool and instructing them to rate all addons regardless of how strongly they feel about them prevents that form of bias.

This situation only constitutes bias if you accept the opinions of those apathetic to an item as valid and desired. If you only want strong or supported opinions, as opposed to "meh" reviews, then providing a vote button provides exactly what is wanted. You are also forgetting that instructing a reviewer pool handpicked by yourself on how to review inserts an entirely different kind of bias which is much worse. You are not impartial, and you will never be - nor will anybody who is tasked with handpicking and indoctrinating select reviewers.

It's the same idea here. Honestly, as much as I want everyone to participate, I really don't want someone who thinks the map preview is ugly but has never actually played the map to vote. And the guy who voted 5.0 just so he could get his review of "help how do i install maps" out there? I wouldn't mind if he was left off the sample, too.

If someone votes 5 and leaves an off-topic review like that, you can ban them from rating/reviewing and delete the review. Simple moderation tools.

It depends on who those 5 people were. If, for instance, your question was "Which of the five axioms of Euclidean geometry hold in a hyperbolic plane?" you would have far better results asking 5 mathematicians than 500 people selected randomly from the human population.

We are not asking questions of true or false, or a question where there is a right and wrong answer. This is a matter of personal opinion. If judging popularity via political polls, I'd rather have 500 random people selected from the population than 5 mathematicians, political scientists, economists, you name it...

I mean, the choice is between having the honest opinions of a few reviewers, or a number that represents the mean of what a bunch of people clicked for reasons undisclosed. To me, the former number is more likely to be accurate.

Again... It is a matter of opinion, not "accuracy." Every opinion that is well-supported, original, and valid regarding an item has a place. People shouldn't be obligated to always disclose their reasons because it subjects them to community scrutiny and has the potential to start flame wars (and I don't mean with cyborgs .) What is accuracy, anyways? If you have the honest opinions of a few reviewers, then it is accurate only so far as you consider those few reviewers. What about everyone else who isn't being heard?

List of condescending statements

This is probably the biggest problem I'm worried about. I think what I'll end up doing is having two rating systems - an "Official Warzone Rating" and a "Community Rating" that anyone can vote.

This, on the other hand, isn't true at all. Here, let me teach you Statistics 101. Larger datasets only result in more accurate results if your sample is a random sample.

Honestly, as much as I want everyone to participate, I really don't want someone who thinks the map preview is ugly but has never actually played the map to vote.

[cybersphinx]

the honest opinions of a few reviewers

Sounds like that should be in a map review blog or something, maybe auto-linked on the addons page.

[Zarel]

RV's entire post seems to have flown right over your head. It's supposed to serve mappers and modders too; they should be able to get beta feedback, etc. ...

Erm... why?

I see no reason why a review system and a feedback system should be conflated. The Addons system is a review system, not a feedback system.

Restrict multiple forum accounts on the same IP.
Restrict voting attempts from the same IP.

It's called proxies.

Not everyone should be able to review. Knowing we are on the same page about this, if a user must have received significant reviews and thumbs-ups to have the ability to review in the first place, how is a devious user going to promote 50 multi-accounts to contribution critic status, then bump himself up?

I am arguing against opening it to everyone (what KenAlcock was suggesting), not against opening it to contributors. Seriously, stop being so egocentric, not every sentence I type here is directed at you.

This situation only constitutes bias if you accept the opinions of those apathetic to an item as valid and desired. If you only want strong or supported opinions, as opposed to "meh" reviews, then providing a vote button provides exactly what is wanted. You are also forgetting that instructing a reviewer pool handpicked by yourself on how to review inserts an entirely different kind of bias which is much worse. You are not impartial, and you will never be - nor will anybody who is tasked with handpicking and indoctrinating select reviewers.

For an unpopular item, people who have "strong or supported opinions" are generally people who like/hate the creator, and frequently haven't actually tried it out. Obtaining a review from an "apathetic" uninvolved third party who can be trusted to play and evaluate the map would indeed be better.

There will always be bias, but bias against people who haven't actually played the map is generally considered a good thing. Perfect impartiality doesn't exist, but it is not necessary to obtain a useful review.

If someone votes 5 and leaves an off-topic review like that, you can ban them from rating/reviewing and delete the review. Simple moderation tools.

Yeah, well, moderation tools are annoying to code. They're all hacks upon an imperfect system. I have a system that doesn't require them - why not use it?

We are not asking questions of true or false, or a question where there is a right and wrong answer. This is a matter of personal opinion. If judging popularity via political polls, I'd rather have 500 random people selected from the population than 5 mathematicians, political scientists, economists, you name it...

But we're not judging popularity. Like I said, I'll have a separate vote system for popularity. The purpose of reviews, on the other hand, is to help decide who to vote for. I'm not going to just vote for whoever's the most popular. I'm going to listen to people who know what they're talking about, and can describe the candidates for me to help me make an educated assessment.

Again... It is a matter of opinion, not "accuracy." Every opinion that is well-supported, original, and valid regarding an item has a place. People shouldn't be obligated to always disclose their reasons because it subjects them to community scrutiny and has the potential to start flame wars (and I don't mean with cyborgs .) What is accuracy, anyways? If you have the honest opinions of a few reviewers, then it is accurate only so far as you consider those few reviewers. What about everyone else who isn't being heard?

Do we really need the opinions of everyone, especially if the majority will just say the same thing as the reviewers?

List of condescending statements

This is probably the biggest problem I'm worried about. I think what I'll end up doing is having two rating systems - an "Official Warzone Rating" and a "Community Rating" that anyone can vote.

wtf. How is "That's a good point; I have modified my proposal to resolve your issue" at all condescending?

This, on the other hand, isn't true at all. Here, let me teach you Statistics 101. Larger datasets only result in more accurate results if your sample is a random sample.

If someone wrongly accuses me of misunderstanding statistics, of course I'm going to be a bit condescending.

Honestly, as much as I want everyone to participate, I really don't want someone who thinks the map preview is ugly but has never actually played the map to vote.

And this is just my assessment of what will happen to a vote system opened to public voting. It's not directed at anyone, specifically, so it doesn't exactly condescend to anyone. Try looking up the word: condescend

If I were being condescending, I'd say something like "lol i bet ppl who vote r all liek 'map iz ugly' n vote b4 playin on it"

[Dylan Hsu]

Let me restate my position that contributors who receive positive feedback on their work should be inducted into the reviewer pool. The amount of positive feedback required for upward social mobility can be set quite high so that it doesn't end up being a lot of people. If it's set high enough, the reviewer pool will stabilize within enough time: as users become inactive, their accumulation of received feedback (most likely just a counter) resets, while the feedback on their work does not reset. This ensures a contained pool of reviewers who are subject to public scrutiny in the community and thus are obliged to give thorough and well-thought reviews.

A 5 star system bears much similarity to the thumbs up, thumbs down system. If we were to implement permanently the 5 star system, might I suggest that we add a sort of bar graph for each rating and quantize ratings (i.e. you can't give someone 3.14159 stars.) Here's an image of what I mean, ripped from here

With this, I feel one gets the best of both worlds. You get the best data possible - an accurate average that can take into account tepid feedback, but also a fairly descriptive spectrum of all feedback received.

[Zarel]

Sounds like that should be in a map review blog or something, maybe auto-linked on the addons page.

The addons database itself is culled. We don't exactly accept any map or mod that's submitted. It was originally designed as a way to sort through all the stuff in the Showcase forum.

That's honestly what this review system was originally meant to be. What I have in place now is just a preliminary system because I wanted people to help me decide which maps and mods to accept.

I'm not entirely sure what the end product will be, but the system I'm currently planning will be:

- "Everyone" will be able to "vote", with the following caveats:
- - "Everyone" may be restricted to "everyone with a forum account and at least 4 posts". The "at least 4 posts" restriction is intended to prevent spamming votes with multiple accounts, and I am open to removing the "at least 4 posts" restriction if either the forum or the development team prioritize vote freedom over vote fidelity.
- - There will be separate "official rating" and "unofficial rating" statistics, and maybe a "combined" rating.
- "Everyone" will be able to submit reviews
- - There will be an "official reviews" section with reviews from reviewers, but good reviews by regular users may also be promoted there

If this isn't enough freedom, I could move the addons database offsite and change the addons tab to link to the Showcase forum. Or you guys could fork the addons database - the code is open enough.