Contribution feedback system

[Rman Virgil]

.

Zarel: Believe me I know the official function of the "Mappers & Modders" is to serve in that way, I'm not an idiot, but the reality is that it's borderline dysfunctional.

What you see from your reality tunnel as no reason to conflate functions I see as an opportunity to address serious shortfalls in the actual reality - an extant system that mostly don't work & never really has..

My point was that this review system however it ultimately takes shape may be an opportunity to supplement the official function of the "Modders & Mappers" BB and perhaps we will get something that actually serves the very important needs of the folks that are actually creating all the new stuff. Get it now.

Otherwise I see no reason to contribute or participate in any way. Simple as that.

- RV
.

[Zarel]

What you see from your reality tunnel as no reason to conflate functions I see as an opportunity to address serious shortfalls in the actual reality - an extant system that mostly don't work & never really has..

Well, you know, Rman, I generally trust your judgment.

I'm not [usually] a mapper, I don't know what mappers want. I would be happy to design a system to accommodate mappers, or modify the review system to do so, but I would need someone to tell me what they want, specifically. Can you do that for me?

[cybersphinx]

I don't know what mappers want.

Maybe they want their maps to be accepted without a biased preselection.

[Rman Virgil]

.

Well Zarel this would apply to both Maps & Mods.

The biggest hurdle for both are Balance issues which a mapper or modder cannot alone sufficiently address without an external review process. (The "M & M" bb is not at all really serving this function.)

Keep it KISS. Make a distinction in the submission protocol.

1.) This is a FINISHED Mod or Map for review.

2.) This is a late BETA Map or Mod for review.

In both cases have the creater give a brief detail of their design goals.

This way the Reviewers can base their assessment on the very goals of the creator.

And in the case of (2.) the Reviewers can make constructive & specific suggestions to the Modder or Mapper for how to achieve their goals so that they can submit their work for a (1.) Finished work review.

What I am also looking at here is for a system that avoids what amounts to a Zero Sum game for it's creative contributors.

Cybersphinx's comment was pointing in this direction if I understood it correctly.

- Regards, RV
.

[cybersphinx]

Actually my comment was pointing at

We don't exactly accept any map or mod that's submitted. It was originally designed as a way to sort through all the stuff in the Showcase forum.

That's honestly what this review system was originally meant to be. What I have in place now is just a preliminary system because I wanted people to help me decide which maps and mods to accept.

[Zarel]

Actually my comment was pointing at

You said yourself that maps without HQ's should be rejected. My criteria are along the same lines - the only addons to be rejected are ones that don't do what they claim to do, such as if they crash Warzone or something.

What I mean is that when I originally started coding this, it was because I didn't want to have to download and test all maps for this, so I made a way to outsource that approval process to other people. And then I figured that it's not too much work to turn the binary "yes/no" into a review system, so that's what I did. A rating of "0" was for "this map crashes Warzone" and to be rejected, and ratings of "1-5" were to be accepted.

I was going to eventually add code to allow everyone to vote; I just haven't gotten there yet.

P.S. This process is rapidly getting annoying. I add a feature, and then everyone freaks out because it's not the feature they wanted. Interesting to see that those are the people complaining about entitlement mentality.

Well Zarel this would apply to both Maps & Mods.

[system description]

Sure, I can do all of that.

[Rman Virgil]

.

Cybersphinx: I see. What's called "cherry-picking". Classic Zero-Sum. Didn't read that. Oh well. Said my peace. Can do no more. Back to work.

- RV
.

[cybersphinx]

Well, apparently it was a misunderstanding because Zarel failed to mention that the current ratings are basically an extended "works/doesn't work" check, and not meant as a public rating system.

[JDW]

I'm just trying to think out of the box here, how about ratings which go something like this,

• Uber-Fun
  Definitely Must Try
  Playable
  Meh... Seen Better
  Once was enough, forever!

Maybe you guys can come up with better phrases, if you like this idea. IMHO, this method of defining levels will make a voter try to match his/her mental state (after playing a certain map) with each level before placing the final vote.

I mentioned just five levels, you could try and make more if you like.

The final overall ratings could be displayed in terms of stars (turrets, canons, bullets, whatever).

[Rman Virgil]

.

Well, apparently it was a misunderstanding because Zarel failed to mention that the current ratings are basically an extended "works/doesn't work" check, and not meant as a public rating system.

Ahh yea, see now. I shouldn't have tried to engage this discussion while distracted at work. My apologies.

- RV

.

[EvilGuru]

Hi all,

As this topic is quite meaty it is probably better if I respond to general points as opposed to posts.

Firstly, the claim that the system is elitist. Indeed it is. However, how is this a bad thing? By making the review process selective it not only gives people something to aspire to but also ensures competence in the reviewers. All to often things are down-rated just because people can not figure out how to use them or exploit their full potential.

Moreover I also agree with Zarel as to why a free-for-all 5-star system does not work in practise. Reviewing a collection of products is not something anyone can do. You need to not only be prolific in order to facilitate comparisons between similar products but also consistent. It takes time, effort and experience. (And, incidentally, why I do not plan on reviewing anything, maps, mods, or otherwise.)

Finally, and possibly the most important point, is that having a small team of reviewers/commentators is crucial because it allows for readers to negate any perceived bias. I'll give an example: Robert Christgau is a well known American rock critic. He has reviewed thousands of albums over the past 30 years or so. He is also incredibly biased and has an extremely unique taste in music. (He gave Pink Floyd's 1973 album Dark Side of the Moon -- one of the best selling albums of all time -- a B and Making Movies by Dire Straits a C+.) So why do people still value his opinion? Simple: because he is biased and he knows it. If you know someone is biased you can compensate for it. Read enough material from a reviewer and you develop the ability to relate their bias to your own.

The problem arises when either the source is not biased (due to many respondents) or has an unknown bias. At this point your opinion is worthless. You say it is "great," and "allows for an excellent multi-player match", but I don't know you, I don't know if we have similar views on what is great and what is average. This kind of relationship between commentator and reader takes time to develop.

In conclusion: ratings are only meaningful when supplemented with comments (which are also helpful for those developing the content in question) and only helpful when you know who made the comments.

As for if this system will be beneficial to content creators. Not without lots of commentary and feedback from fellow content creators, and experienced players (developers or otherwise). Unless, that is, someone has developed a way of converting a number from 1-5 into constructive criticism.

Edit: Just saw this on Slashdot and consider it to be quite poignant: http://thehill.com/blogs/hillicon-valle ... -overrated entitled "Fark creator says wisdom of crowds is overrated."

Polemically yours, Freddie.

[Rman Virgil]

Hi all,

As this topic is quite meaty it is probably better if I respond to general points as opposed to posts.

Firstly, the claim that the system is elitist. Indeed it is. However, how is this a bad thing? By making the review process selective it not only gives people something to aspire to but also ensures competence in the reviewers. All to often things are down-rated just because people can not figure out how to use them or exploit their full potential.

Moreover I also agree with Zarel as to why a free-for-all 5-star system does not work in practise. Reviewing a collection of products is not something anyone can do. You need to not only be prolific in order to facilitate comparisons between similar products but also consistent. It takes time, effort and experience. (And, incidentally, why I do not plan on reviewing anything, maps, mods, or otherwise.)

Finally, and possibly the most important point, is that having a small team of reviewers/commentators is crucial because it allows for readers to negate any perceived bias. I'll give an example: Robert Christgau is a well known American rock critic. He has reviewed thousands of albums over the past 30 years or so. He is also incredibly biased and has an extremely unique taste in music. (He gave Pink Floyd's 1973 album Dark Side of the Moon -- one of the best selling albums of all time -- a B and Making Movies by Dire Straits a C+.) So why do people still value his opinion? Simple: because he is biased and he knows it. If you know someone is biased you can compensate for it. Read enough material from a reviewer and you develop the ability to relate their bias to your own.

The problem arises when either the source is not biased (due to many respondents) or has an unknown bias. At this point your opinion is worthless. You say it is "great," and "allows for an excellent multi-player match", but I don't know you, I don't know if we have similar views on what is great and what is average. This kind of relationship between commentator and reader takes time to develop.

In conclusion: ratings are only meaningful when supplemented with comments (which are also helpful for those developing the content in question) and only helpful when you know who made the comments.

As for if this system will be beneficial to content creators. Not without lots of commentary and feedback from fellow content creators, and experienced players (developers or otherwise). Unless, that is, someone has developed a way of converting a number from 1-5 into constructive criticism.

Edit: Just saw this on Slashdot and consider it to be quite poignant: http://thehill.com/blogs/hillicon-valle ... -overrated entitled "Fark creator says wisdom of crowds is overrated."

Polemically yours, Freddie.

Well articulated.

I can subscribe to this position in its entirety.

I also hold that the wisdom of crowds is overrated even though there has been the last couple years a very obvious trend in works supporting the thesis that crowds are wiser than the individuals who comprise them.

Just a few of the more popular works being:

~ The Wisdom of Crowds by James Surowiecki

~ Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business... by Jeff Howe

~ Smart Mobs: The Next Social Revolution by Howard Rheingold

~ Groundswell: Winning in a World Transformed by Social Technologies by Charlene Li

~ Herd: How to Change Mass Behavior by Harnessing Our True Nature by Mark Earls

But all these are biased to the extent that they reflect the crowd manipulations possible in for profit, commercial ventures and WZ is not a commercial venture trying to make as much money as possible.

What I believe more accurately applies in this context, views that are NOT marketplace money driven, and convey the nature of crowd behavior in a full 360 degrees are:

~ True Enough: Learning to Live in a Post-Fact Society by Farhad Manjoo

~ Escape from Freedom by Eric Fromm

And last but not least, the towering masterwork of them all -

~ Crowds and Power by Elias Canetti (for which he won the 1981 Nobel Prize)

- RV

.

[JDW]

In conclusion: ratings are only meaningful when supplemented with comments (which are also helpful for those developing the content in question) and only helpful when you know who made the comments.

Just made me realize how much I rely on he Pros and Cons entries of products on Newegg.com . But only of those that have atleast an average number of eggs.

[Saberuneko]

Restrict multiple forum accounts on the same IP.
Restrict voting attempts from the same IP.
Not everyone should be able to review. Knowing we are on the same page about this, if a user must have received significant reviews and thumbs-ups to have the ability to review in the first place, how is a devious user going to promote 50 multi-accounts to contribution critic status, then bump himself up?

Sorry, I came late to comply about this, but.
It's still impossible to prevent, this methods can still be bypassed using proxies.

[Rman Virgil]

Restrict multiple forum accounts on the same IP.
Restrict voting attempts from the same IP.
Not everyone should be able to review. Knowing we are on the same page about this, if a user must have received significant reviews and thumbs-ups to have the ability to review in the first place, how is a devious user going to promote 50 multi-accounts to contribution critic status, then bump himself up?

Sorry, I came late to comply about this, but.
It's still impossible to prevent, this methods can still be bypassed using proxies.

That is true. Very easy to do, in fact. This is where the detailed comments may be something of a deterrent... or maybe not for those dedicated to cheating.

- RV

.