"Philosophical" Questions: Rankings, OpenSource and Cheating

[T_X]

One of the things I'm currently missing a lot is a decent rankings system in Warzone2100. So far, with just these few people who are usually regularly on the IRC channels as well its not that much of an issue and you can somehow remember and estimate how good another player is. However, the issue with this approach is that it does not scale very well, which will hinder the growth of warzone at some point. Also a working, sane ranking system will challenge and therefore encourage both new and veteran players alike, leading to a faster increase in popularity.

However while thinking about this I am currently stuck at one (hopefully just theoretical) point: If such a ranking system were implemented and if the popularity grew quickly, these two things would of course encourage people to find ways to boost their ranking not only in a legitimate way. While "obvious" cheating (e.g. cheating which breaks the rules of the game) can technically be prevented even in Open Source games I'm currently wondering how to prevent not that obvious cheating:

It is relatively easy to write your own, improved AI code in warzone. And it is also not that difficult to bluff any other player in believing that it is you and not an AI playing. Maybe you'd even take control over (parts) of your AIs army from time to time to make it even less obvious.

My question is whether I am currently missing possible solutions to such a problem. Or whether in the future warzone gaming between humans will only be possible while sitting behind certified, locked-up computers (DRM on your not-anymore-general-pupose personal computer at home or while being at a tournament with non-personal, prepared computers, kind of arcade-stylish again).

Or will in 2100 the gameplay of warzone2100 have radically changed and it'd be totally normal and legitimate to not command units anymore, but command/improve AIs instead, maybe with a very intuitive interface making it easy even for non-developers to do so?

.oO(And will those then highly sophisticated AIs and simulation capabilties of computer games then lead militaries of the world to start plugging in their own computers in computer games to train their own drones' AIs - and even base their decisions to attack or not on such simulations - "funnily" making one of the oldest hacker movies, i.e. WarGamse, kind of a reality? Exploiting unsuspecting people to indirectly, but actively financing and contributing to murder all around he world - Ok, these last thoughts might be a little far fetched, no need to comment )

PS: Google also finds some forum posts with similar thoughts and discussions, for instance:

http://forums.wesnoth.org/viewtopic.php?f=13&t=14865
http://www.gamedev.net/topic/368086-ope ... nti-cheat/
http://lists.debian.org/debian-devel/20 ... 01492.html

[Emdek]

Something that could help is using lobby (using that board based authentication) to store such data in central way instead of current solution.

[aubergine]

Here's my simple solution: Make it social.

So you have a lobby where people have to register a password-protected account. At the end of each game, their opponents get chance to rate them.

So it's not the computer that's rating (ranking) people, it's other people. Simples.

Of course, a player could set up a gazillion lobby accounts and use them all to rate themself highly. But this is easily thwarted by ensuring that ratings can only be made after a game is played, and only by the players in that game. And, for games that are too short (less than 10 mins?) ratings will not be allowed.

[T_X]

Hi Emdek, hi aubergine,

Thanks for your replies. But, hmm, I think it still cannot fix my main concern. Even if either a server with registered users or a human would rate another human, how can they make sure that they've been playing just against a human and not an AI?

[Mafiaboss]

Here's my simple solution: Make it social.

So you have a lobby where people have to register a password-protected account. At the end of each game, their opponents get chance to rate them.

So it's not the computer that's rating (ranking) people, it's other people. Simples.

Of course, a player could set up a gazillion lobby accounts and use them all to rate themself highly. But this is easily thwarted by ensuring that ratings can only be made after a game is played, and only by the players in that game. And, for games that are too short (less than 10 mins?) ratings will not be allowed.

I don't think this would be a good idea for saying that you can't rate someone because of a game that ended quickly because your opponent was boss or something else happened.

Also you would have to seperate the ranking systems into two different views, such as a ranking system for friendlyness and how good they play, but even that will not play out so well if a player thinks you cheat when you really are a good player and he/she decides to rank you poorly regardless of how well you played.

The automated system ranking system which is in place as of right now is a fine way to rank up ignoring all the problems that you may face of biased people that you come across who want to see your ratings suffer.

So in conclusion I and i'm sure other people are fine with the ranking system as it is now and don't want it to change in the future unless theres a very good way of handling the problems that may occur from biased people, but even if that happened it would be like going back to the automated system because people wouldn't have a choice of what to choose if that was how the system was set up.

[effigy]

Pre-3.1 master snapshots included a login system. It was reverted because it was incomplete and the person who wrot the code has been inactive.

IIRC the new login system was going to include rankless guest accounts. Accounts that could receive rank would be attached to this forum. Rank stats were going to stored server side. Also I think it was going to have some kind of clan support.

Anyhow, I think this code is somewhere in the fit repo if anyone wants to finish where the last person left off.

[aubergine]

The reason I'd prevent short games from being rated is that if one player was a noob and the other was expert, the noob will likely not be in a good state of mind to give a considered rating.

As for playing against an AI, if I had a good game I don't really care whether it's against human or AI. It doesn't really matter to me if HAL9000 becomes an expert at warzone and gains high ratings as a result, so long as it doesn't try and kill me IRL.

[Mafiaboss]

The reason I'd prevent short games from being rated is that if one player was a noob and the other was expert, the noob will likely not be in a good state of mind to give a considered rating.

As for playing against an AI, if I had a good game I don't really care whether it's against human or AI. It doesn't really matter to me if HAL9000 becomes an expert at warzone and gains high ratings as a result, so long as it doesn't try and kill me IRL.

Yea but there is still always room for people to be biased, and if the noob doesn't want to give the experience player some credit then its only because they were mad that their a** got whooped, and thats being biased, so I say the system could be upgraded on ranking, but who ever does it would have to put in a really good system to try and prevent people being biased

[Emdek]

Single rating impact (or weight) could depend on rank / amount of played games / total played time by the one who gives it (sometimes in similar way are weighted points on sites using digg / reddit like rankings).
Also those with low rank or not enough played games (or too small total played time) could be not allowed to vote.
In this scenario the main issue would be that so nobody would be able to start voting at beginning (and could only start voting if we would use amount of played games or total played games time).

[T_X]

Here's my simple solution: Make it social.

Is this a solution to the problem of which algorithm to use for the ranking? If so, this wasn't really my (main) concern so far, there seem to be nifty algorithms already which have evolved from thousands of games of chess or Go (like the Elo or Glicko rating system) and have been thoroughly, mathematically analized to minimize the risk of an unfitting ranking. One of the latest inventions in this field seems to be TrueSkill (which is based on Glicko), invented by Microsoft Research and used for many xbox360 games. Such algorithms should be able to create a fair ladder (which I would love to have in warzone2100, maybe even different ladders for various game types). One of the major advantages of TrueSkill compared to Glicko is that it's even able to rate multiplayer, multi team games.

With any of these algorithms if at some time the lobby got crowded enough, the lobby server could even automatically match you with other people who are having about the same skill as you, resulting in about a 50:50 chance for both players or any teams, avoiding too frustrating or non-challenging and too short games. These algorithms are even able to tell you what your chance of winning against any other opponent is although you two have never played against each other. So, hm, I'm not quite convinced yet whether social voting would have an advantage compared to these algorithms.

Single rating impact (or weight) could depend on rank / amount of played games / total played time by the one who gives it (sometimes in similar way are weighted points on sites using digg / reddit like rankings).
Also those with low rank or not enough played games (or too small total played time) could be not allowed to vote.
In this scenario the main issue would be that so nobody would be able to start voting at beginning (and could only start voting if we would use amount of played games or total played games time).

TrueSkill (and Glicko, but not Elo I think) already have an automatic weighting incorporated. Next to the current average skill/rating (μ - mean) it has a uncertainity factor (σ - standard deviation) which usually gets smaller, so your rating becomes more "true", the more games you play. Elo has a manually adjusted k-factor instead. Due to these things your rating as an experienced player will not be effected that much by a win/loss against a noob yet (the blog post from moseware.com at the bottom has some nice visuals to explain that).

As for playing against an AI, if I had a good game I don't really care whether it's against human or AI. It doesn't really matter to me if HAL9000 becomes an expert at warzone and gains high ratings as a result, so long as it doesn't try and kill me IRL.

Hmm, and that's what I had been worried about at first... that you might not have "good" games anymore, due to 99.9% of all players actually being an AI which you cannot beat. If you, aubergine, were able to achieve high rankings you'd very likely be an AI then. But if I lost against you I couldn't prove it. Human players would never be able to achieve high ranks.

At first my main concern was, that not only the ladder could be pretty much useless thanks to this, but also that you might not have fun playing anymore due to constantly loosing although no one broke any game rules. But, hm, on second thought, at least with one of the rating systems as described above, you'd be matched with people (or "stupid" AIs) of equal skill, so it might still be kinda fun to play a game against strangers.


https://en.wikipedia.org/wiki/Glicko_rating_system
https://en.wikipedia.org/wiki/TrueSkill
+ http://www.moserware.com/2010/03/comput ... skill.html (very good and easy introduction into TrueSkill!)
https://en.wikipedia.org/wiki/Elo_rating_system (also relatively easy, just 2, 3 short formulas and has nice calculation examples)

https://en.wikipedia.org/wiki/Chess_rating_systems

[camellia0x++]

My complicated solution:

Making each person register a forum account would be a PITA for many people.

The solution, I think, is to use an RSA keypair to handle ranks. A public key is used as a public identifier, and a private key is used by the client to authenticate it's identity. Thus, the server would not need each person to register, and yet no one can be impersonated at the same time.

Trickier question is how to rank people. I would not worry about it. Honest people will be honest, and cheaters WILL find a way to get around the system, unless each server (not the lobby) keeps track of ranks itself. This would need specific servers to be created though. 2 servers could possibly choose to share ranking information, but it'd need to be manual to keep out cheating servers.

[JDW]

Making each person register a forum account would be a PITA for many people.

That is why it was removed. I think in the future they plan to allow registered users and also anonymous login without password. I guess those with anonymous login would not need a forum account and they would probably not have any rank.

[Lord Apocalypse]

Best way to work rankings would be a hard number (ie actual wins) and a social ranking, say a 1-5 scale or 1-10 based on a few things. Was the player any good or did he/she win by attrition (sit back while better players bash each other into oblivion then stroll in and clean up the mess). Was the person a sore loser/ polite (can't think of something better) winner? And you could also apply if the player was a real jerk (includes those who rage quit) or kicks out any experienced player just to bash noobs all day (high wins low social). Its possible you could also track how many times a particular host kicks people out.

Side note: You coul stick with the above or continue with data collection. Number of wins per map, favored map or game configuration etc. Most of this could be collected when you click play or after the game is complete. Save all this is an SQL server (as well as private data such as IP address & provider) and to prevent multiple accounts, have the client send in the computers mac address.

[camellia0x++]

I don't think votes are a good way to keep track of ranking, from experience; people will just upvote their friends.

A better way is by wins/losses/kills/points.

[Lord Apocalypse]

thats why you run both.