Automated testing

For code related discussions and questions
Post Reply
Per
Warzone 2100 Team Member
Warzone 2100 Team Member
Posts: 3753
Joined: 03 Aug 2006, 19:39

Automated testing

Post by Per » 29 Dec 2016, 02:20

I am currently trying to come up with some goods ways to improve our automated testing. (As in, actually do some automated testing.)

We currently have some command-line facilities that allow us to run this semi-automated, from scripts. With --loadskirmish=savegame and --loadcampaign=savegame, we could jump straight into action for a while now. Recently, I added --autogame, which will cause the game to skip straight through hosting menues, set current player to AI, run at 500x speed, and exit once game is done. This is useful with savegames, or with the --game option, or with --host and --join. Yesterday, I added --saveandquit=savegame, which allows to start a game, in combination with loading a savegame or --game, then immediately saving and quitting. This allows automated testing that saving/loading works for a variety of scenarios. I've started putting this into a script, and it has exposed some errors I'd like to fix before sharing it... :-)

However, the big question is, how do we test campaign. In particular, we need to test every level transition in some automated manner. We already have the "let me win" cheat, which works most of the time. It does not work for transport missions, though. I'm thinking of adding a hack to load up the transport with whatever units are nearby and launch it, on "let me win", and then add another command line option to repeatedly issue "let me win" until the campaign is over. But there could be complications, and other situations where "let me win" will get stuck. Ideas on how to do this in the best way would be welcome.

MIH-XTC
Trained
Trained
Posts: 251
Joined: 31 Jan 2014, 07:06

Re: Automated testing

Post by MIH-XTC » 29 Dec 2016, 21:08

Per wrote:I am currently trying to come up with some goods ways to improve our automated testing. (As in, actually do some automated testing.)

We currently have some command-line facilities that allow us to run this semi-automated, from scripts. With --loadskirmish=savegame and --loadcampaign=savegame, we could jump straight into action for a while now. Recently, I added --autogame, which will cause the game to skip straight through hosting menues, set current player to AI, run at 500x speed, and exit once game is done. This is useful with savegames, or with the --game option, or with --host and --join. Yesterday, I added --saveandquit=savegame, which allows to start a game, in combination with loading a savegame or --game, then immediately saving and quitting. This allows automated testing that saving/loading works for a variety of scenarios. I've started putting this into a script, and it has exposed some errors I'd like to fix before sharing it... :-)

However, the big question is, how do we test campaign. In particular, we need to test every level transition in some automated manner. We already have the "let me win" cheat, which works most of the time. It does not work for transport missions, though. I'm thinking of adding a hack to load up the transport with whatever units are nearby and launch it, on "let me win", and then add another command line option to repeatedly issue "let me win" until the campaign is over. But there could be complications, and other situations where "let me win" will get stuck. Ideas on how to do this in the best way would be welcome.
I'm not sure it's worth the effort to try and automate this because manually using "let me win" works pretty well. I wasn't aware of "let me win" until just now but I just tried using it and finished all 3 campaigns in 2 minutes a piece. <10 minutes to finish all 3 campaigns isn't too bad. You shouldn't need to verify/test the transitions that often.

You're right the transporter missions don't work using "let me win" but as soon as the transporter leaves the map and loads the next one then it does work, you just have to select all units, send to transport and launch transport then crank up speed to 100x.

This actually isn't as painful as testing balance changes to multiplayer. A flawless research game needs to be simulated to test multiplayer balance changes and there is no way to automate that but with the right map and debug mode I can simulate the entire tech tree in about 15 minutes and I've done that hundreds of times.

User avatar
Berserk Cyborg
Code contributor
Code contributor
Posts: 615
Joined: 26 Sep 2016, 19:56

Re: Automated testing

Post by Berserk Cyborg » 31 Mar 2018, 01:53

Yeah, I know, topic is somewhat old.
Per wrote: However, the big question is, how do we test campaign. In particular, we need to test every level transition in some automated manner. We already have the "let me win" cheat, which works most of the time. It does not work for transport missions, though. I'm thinking of adding a hack to load up the transport with whatever units are nearby and launch it, on "let me win", and then add another command line option to repeatedly issue "let me win" until the campaign is over. But there could be complications, and other situations where "let me win" will get stuck. Ideas on how to do this in the best way would be welcome.
Should be possible now. Pre-away missions need to load up at least one unit into the transporter, launch it and get into the next mission, and then "let me win". Gamma 8 should not fail instantly when entering it from cheat/debug mode anymore (the nuke silos disappear and that caused the WZscript version to issue a fail). There are some special cases, however.

1. "let me win" needs to be called twice for Alpha 3 (skip the power module part).
2. Move a unit towards player 1's factory in Gamma 6 to donate the structures to the player for the next missions (all objects at the top half of the map are explicitly destroyed at the start of Gamma 8).

User avatar
NoQ
Special
Special
Posts: 6189
Joined: 24 Dec 2009, 11:35
Location: /var/zone

Re: Automated testing

Post by NoQ » 31 Mar 2018, 18:51

*dreams of an AI that could beat campaign missions for automated testing purposes*

Post Reply