Automated testing
Posted: 29 Dec 2016, 02:20
I am currently trying to come up with some goods ways to improve our automated testing. (As in, actually do some automated testing.)
We currently have some command-line facilities that allow us to run this semi-automated, from scripts. With --loadskirmish=savegame and --loadcampaign=savegame, we could jump straight into action for a while now. Recently, I added --autogame, which will cause the game to skip straight through hosting menues, set current player to AI, run at 500x speed, and exit once game is done. This is useful with savegames, or with the --game option, or with --host and --join. Yesterday, I added --saveandquit=savegame, which allows to start a game, in combination with loading a savegame or --game, then immediately saving and quitting. This allows automated testing that saving/loading works for a variety of scenarios. I've started putting this into a script, and it has exposed some errors I'd like to fix before sharing it...
However, the big question is, how do we test campaign. In particular, we need to test every level transition in some automated manner. We already have the "let me win" cheat, which works most of the time. It does not work for transport missions, though. I'm thinking of adding a hack to load up the transport with whatever units are nearby and launch it, on "let me win", and then add another command line option to repeatedly issue "let me win" until the campaign is over. But there could be complications, and other situations where "let me win" will get stuck. Ideas on how to do this in the best way would be welcome.
We currently have some command-line facilities that allow us to run this semi-automated, from scripts. With --loadskirmish=savegame and --loadcampaign=savegame, we could jump straight into action for a while now. Recently, I added --autogame, which will cause the game to skip straight through hosting menues, set current player to AI, run at 500x speed, and exit once game is done. This is useful with savegames, or with the --game option, or with --host and --join. Yesterday, I added --saveandquit=savegame, which allows to start a game, in combination with loading a savegame or --game, then immediately saving and quitting. This allows automated testing that saving/loading works for a variety of scenarios. I've started putting this into a script, and it has exposed some errors I'd like to fix before sharing it...
However, the big question is, how do we test campaign. In particular, we need to test every level transition in some automated manner. We already have the "let me win" cheat, which works most of the time. It does not work for transport missions, though. I'm thinking of adding a hack to load up the transport with whatever units are nearby and launch it, on "let me win", and then add another command line option to repeatedly issue "let me win" until the campaign is over. But there could be complications, and other situations where "let me win" will get stuck. Ideas on how to do this in the best way would be welcome.