Automated testing

For code related discussions and questions

Automated testing

Postby Per » 29 Dec 2016, 02:20

I am currently trying to come up with some goods ways to improve our automated testing. (As in, actually do some automated testing.)

We currently have some command-line facilities that allow us to run this semi-automated, from scripts. With --loadskirmish=savegame and --loadcampaign=savegame, we could jump straight into action for a while now. Recently, I added --autogame, which will cause the game to skip straight through hosting menues, set current player to AI, run at 500x speed, and exit once game is done. This is useful with savegames, or with the --game option, or with --host and --join. Yesterday, I added --saveandquit=savegame, which allows to start a game, in combination with loading a savegame or --game, then immediately saving and quitting. This allows automated testing that saving/loading works for a variety of scenarios. I've started putting this into a script, and it has exposed some errors I'd like to fix before sharing it... :-)

However, the big question is, how do we test campaign. In particular, we need to test every level transition in some automated manner. We already have the "let me win" cheat, which works most of the time. It does not work for transport missions, though. I'm thinking of adding a hack to load up the transport with whatever units are nearby and launch it, on "let me win", and then add another command line option to repeatedly issue "let me win" until the campaign is over. But there could be complications, and other situations where "let me win" will get stuck. Ideas on how to do this in the best way would be welcome.
Per
Warzone 2100 Team Member
Warzone 2100 Team Member
 
Posts: 3561
Joined: 03 Aug 2006, 19:39

Re: Automated testing

Postby MIH-XTC » 29 Dec 2016, 21:08

Per wrote:I am currently trying to come up with some goods ways to improve our automated testing. (As in, actually do some automated testing.)

We currently have some command-line facilities that allow us to run this semi-automated, from scripts. With --loadskirmish=savegame and --loadcampaign=savegame, we could jump straight into action for a while now. Recently, I added --autogame, which will cause the game to skip straight through hosting menues, set current player to AI, run at 500x speed, and exit once game is done. This is useful with savegames, or with the --game option, or with --host and --join. Yesterday, I added --saveandquit=savegame, which allows to start a game, in combination with loading a savegame or --game, then immediately saving and quitting. This allows automated testing that saving/loading works for a variety of scenarios. I've started putting this into a script, and it has exposed some errors I'd like to fix before sharing it... :-)

However, the big question is, how do we test campaign. In particular, we need to test every level transition in some automated manner. We already have the "let me win" cheat, which works most of the time. It does not work for transport missions, though. I'm thinking of adding a hack to load up the transport with whatever units are nearby and launch it, on "let me win", and then add another command line option to repeatedly issue "let me win" until the campaign is over. But there could be complications, and other situations where "let me win" will get stuck. Ideas on how to do this in the best way would be welcome.


I'm not sure it's worth the effort to try and automate this because manually using "let me win" works pretty well. I wasn't aware of "let me win" until just now but I just tried using it and finished all 3 campaigns in 2 minutes a piece. <10 minutes to finish all 3 campaigns isn't too bad. You shouldn't need to verify/test the transitions that often.

You're right the transporter missions don't work using "let me win" but as soon as the transporter leaves the map and loads the next one then it does work, you just have to select all units, send to transport and launch transport then crank up speed to 100x.

This actually isn't as painful as testing balance changes to multiplayer. A flawless research game needs to be simulated to test multiplayer balance changes and there is no way to automate that but with the right map and debug mode I can simulate the entire tech tree in about 15 minutes and I've done that hundreds of times.
MIH-XTC
Trained
Trained
 
Posts: 153
Joined: 31 Jan 2014, 07:06


Return to Coding

cron