Unit testing

[aubergine]

I'm thinking of adding unit testing to my scripts. They are getting so numerous that I want to be able to run automated tests after making changes to make sure nothing's broken.

I've been looking at QUnit, jQuery's unit test suite of choice, and love it's syntax. Has anyone used it before? Is it as nice to work with as it looks (code-wise)? I'm thinking of porting it to WZ JS environment, possibly a trimmed down version as some of the stuff in there is HTML DOM related.

[aubergine]

Ok, I've now started coding a QUnit-esque unit testing API for WZ... I hope to have an initial release in about 1-2 weeks, complete with full documentation.

I plan on making my first set of unit tests put a bunch of JS API features under the microscope, for example checking that events are firing properly (and in what order) as a result of in-game actions. The actions will initially be scripted, but could quite easily be replaced by console messages that provide instructions for a human player to perform actions and thus test the user interface. I'll probably deploy the JS API unit tests as challenges to make them super simple to install and use.

But... before I do all that, is there any existing unit testing in WZ that I should be aware of?

[Per]

But... before I do all that, is there any existing unit testing in WZ that I should be aware of?

Yes. I wrote a unit testing framework for the qtscript API, which resides in tests/lint.cpp and friends. It also used for the 'lint' tool that can check scripts for basic correctness, attempting to call all events with fake data and verifying parameter types. However, this code has not been updated in master for a while. I am wondering whether these tests and the lint tool is worth the work. If not, maybe I should just move the parameter checking into the API itself.

[dak180]

But... before I do all that, is there any existing unit testing in WZ that I should be aware of?

Yes. I wrote a unit testing framework for the qtscript API, which resides in tests/lint.cpp and friends. It also used for the 'lint' tool that can check scripts for basic correctness, attempting to call all events with fake data and verifying parameter types. However, this code has not been updated in master for a while. I am wondering whether these tests and the lint tool is worth the work. If not, maybe I should just move the parameter checking into the API itself.

As a general note, the jslint tool is distributed as a hidden executable in the mac app bundle in order to facilitate testing by AI devs.

[aubergine]

I think it's worth the effort to update the unit tests. I noticed things like SS_BEING_DEMOLISHED are still in the current unit tests and things like ENEMIES are missing.

As for moving parameter checking in to the API, would that add extra overhead for every call to a JS API function? Also, what about scenarios where script developers use a constant that's not yet defined by JS API, such as DORDER_CIRCLE -- they seem to work fine, so long as the value used matches what WZ uses internally, and being able to do this allows us to do some things that are not yet officially supported by the API.

[Per]

It is extra overhead and rather ugly code, which is why I did not put it in the main scripting code in the first place. Ok, I'll see if I can update the unit tests some day.

[aubergine]

Is there any chance a dump() function, as described here, can be added to next release of WZ 3.1 branch? It will me massively useful for my unit testing stuff. Trying to output gazillions of test results to the console is somewhat painful, and very difficult to copy and paste into a wiki

[aubergine]

Almost completed my unit testing API, it's similar in many ways to QUnit.

However, for the assertion functions (eg. ok(), equal(), etc) I don't want to store them on global object/scope like QUnit does.

I want them only to be available within a unit test function.

So, let's say I have this unit test:

Code: Select all

Test("Can we chat()?", Test.ANY(), function(_global) {
  ok( _global.hasOwnProperty("chat"), "chat() is defined?" );
});

The function defined in the 3rd parameter of Test() is the unit test. I want stuff inside that function to have access to ok(), etc., functions that are defined separately in the Test API.

So far I've tried altering the prototype chain on that function, but that turned out to be troublesome in various scenarios.

I then tried calling the function, scoping it to my unit test instance (an object that has the ok(), etc., functions on it's prototype) and that works but only if I prefix the assertion methods with 'this', for example, "this.ok( ... )" -- which is ugly.

Anyone have any ideas how I can make externally defined (like in a separate .js file so no chance of creating a closure) assertion functions available within any function of my choosing without needing to prefix them with 'this." and without having to make them global?

[aubergine]

Uhm, so I found a way to do it. But I had to sell my soul to the devil. Two devils in fact...

Code: Select all

with (test) { eval("("+run+")()"); }

Yes, "with" and "eval" used together in the same sentence. Oh, the humanity! Cats and dogs living together!

There's something very important I forgot to tell you! Don't cross the streams… It would be bad… Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light.

And while on the subject, was that the primary buffer panel falling off the ship?

EDIT: On the bright side, I now have an operational unit testing API

EDIT #2: And my unit tests that test the Test API just unearthed a bug in the deepEqual( ) assertion!

[aubergine]

Ok, this is doing my nut in... Can anyone see why this would return false when passed two objects {a:5} and {a:5}?

EDIT: Ignore code comments, been trying all sorts of stuff so they are out of synch with code.

Code: Select all

var visted;

var deepEq = function(a, b, inside) {
	if (!inside) {
		// see if we can quick fail
		if (a != b) return false;
		// ok, looks like we're gonna have to do it hard way
		// init list of places we don't need to visit
		visited = [
			_global,
			Object.prototype,
			Array.prototype,
			Date.prototype,
			a,
			Boolean.prototype,
			String.prototype,
			Number.prototype,
			Function.prototype,
			Error.prototype,
			Test
		];
	} else if (inside > 10) {
		return true; // don't bother digging any deeper
	} else if (visited.some(function(place) { return place === a })) {
		return true; // already visited here
	} else {
		visited.unshift(a); // add to list of places not to visit again (avoid circular references)
	}
	
	var keysA = Object.getOwnPropertyNames(a);
			
	if ( keysA.some(function difference(key) {
		// bail if b doesn't have the key
		if (!b.hasOwnProperty(key)) return true;
		// bail if strict check fails
		if (a[key] != b[key]) return true;
		// compare a[key] to b[key]
		switch (typeOf(a[key])) {
			case "object":
			case "array": {
				return !deepEq(a[key], b[key], (inside || 0)+1);
			}
			default: {
				return a[key] != b[key];
			}
		}
	}) ) { // a != b
		return false;
	};

	if (!inside) {
		visited = null; // free up RAM
	}
	return true;
}

Note: I don't want something as brutal as some of the heavyweight implementations found out on the interwebs, just something basic.

It's the only thing preventing me releasing the Test API. :s

[aubergine]

Fixed it Just finishing the unit tests then will release the Test API.

Really need a dump() function in JS API to make it easier to export test results!

[aubergine]

Documentation for Test API almost complete: https://warzone.atlassian.net/wiki/display/EGG/Test+API

Unit tests that test the Test API unearthed some errors with asynchronous tests which I still need to resolve.

Desperately need a dump() function in JS API so I can output HTML reports.

[Duha]

Is there any chance a dump() function, as described here, can be added to next release of WZ 3.1 branch? It will me massively useful for my unit testing stuff. Trying to output gazillions of test results to the console is somewhat painful, and very difficult to copy and paste into a wiki

Then I back from vacations I can add test support for output reader.

[aubergine]

Once I get a dump() function I plan on adding multiple output modes for the Test API.

Currently I have a kludge that just outputs a few lines of results per few seconds to the console. It's grim but it works.

Using current debug() implementation on mac os x is grim as my console app is drowned with 5 years worth of un-read log files and I'm loathe to go poking around in a terminal window as I'm very much a GUI kinda guy.

For me the best solution would be a way of dump()-ing text to a file of custom name in the WZ logs folder - also a place that's easy for most end-users to find so they can send me such files if I ask them to run diagnostics or tests.

[aubergine]

Here is a quick demo of v1.0:

test api demo.zip

Test API demo

(13 KiB) Downloaded 288 times

To use it, move to mods/autoload and extract the zip (not sure if the folder structure is set-up right but you should be able to work it out) -- it's packaged as an AI.

Start a 2 player skirmish game (eg. on Sk-Startup) and select the "Test API Demo" AI.

When the game starts, the unit tests will automatically run - they provide near 100% code coverage of the Test API.

15 seconds in to the game, the results form the tests will start getting output to the AI player's console -- you'll need to activate cheat mode (Shift + Backspace) and then switch to Player 1 via the Debug Menu (aka Options Menu) to see the AI's console. The output is generated by a "Crufty results output function" at the bottom of the API script. If you prefer having output go to debug() instead of console() it's a simple change to make. Future plans are to provide multiple output modes and, if we get a dump() function (or whatever it might end up being called) in the JS API, the ability to export to HTML, Wiki Markup, etc.

Note: Some of the tests are designed to fail (to test that tests can fail) -- so you should expect to see some failed tests in the output.

If you want to take a look at the source code:
* The API code is in multiplay/skirmish/Test.js
* The unit tests for the Test API are in multiplay/skirmish/Test/Test.js

As mentioned in an earlier post, full documentation for the Test API is now available at https://warzone.atlassian.net/wiki/display/EGG/Test+API

If the Dependency Checker of Util API is installed (will be released soon, once I get unit tests done for it!) the Test API will fully integrate with it, enabling unit tests to load automatically if the Test API is loaded. I'll be creating some chat commands to load the Test API, run unit tests, etc., via the Chat API (again not released until I get some unit tests done for it and hopefully a Mac-compatible build of WZ 3.2 so I can actually test chat features!). Currently there's no diagnostics routines for the Test API, but they will follow soon (and yes, you guessed it, Diag API isn't getting released until it has unit tests...).

Anyway, it's 12noon, which is about 12 hours past my bedtime, so I need to get some sleep...