[Contribution] The Warzone2100 Updater!

[noccy]

Okay, here's something for all you Windows users who have been asking about an updater. Here's the solution, or well at least part of it. Introducing: The Warzone2100 updater!

It's plain c code, written and compiled on Linux but in theory it should work just as well with Mingw or Cygwin. It works, at least. And it's easy to administrate. There are a few things that need fixing that I've been too lazy to take care of so far, namely:

• The database is not implemented. Its purpose is to make sure that an unmodified file is not downloaded. Until it's fixed, each new release will cause a download of all the files.
• Hashchecking, to make sure that the files were downloaded Ok. After all, we do already have the hashes so they should just be compared.
• Run mode, simply add the code to load Warzone. This way you could use wzupdater to launch the game, always being up to date.

The update server should be set up with a file named "LATEST" containing the version number of the latest release. For this version number there should also be a matching folder having the name of the version (f.ex. "2.3.4") and a file named as "wzupdate-2.3.4.md5" being the output of the md5sums command for all the files in the release.

This is from a test run. As you can see all of the files i unzipped from the installer are being downloaded into the empty application directory.

Code: Select all

noccy@noccy-desktop:~/Development/c/wzupdater$ cd test
noccy@noccy-desktop:~/Development/c/wzupdater/test$ ls
wzupdater
noccy@noccy-desktop:~/Development/c/wzupdater/test$ ./wzupdater 
Looking for updates: Latest is 2.3.4
An update is available! Downloading manifest
Checking files: 0 in database, 54 new, 0 needs updating.
Downloading ./Readme.en.html [ 1/54] . 28.57kB
Downloading ./mp.wz [ 2/54] ... 715.03kB
Downloading ./COPYING [ 3/54] . 17.59kB
Downloading ./dbghelp.dll [ 4/54] ... 1020.63kB
Downloading ./Readme.de.html [ 5/54] .. 30.53kB
Downloading ./Readme.en [ 6/54] . 20.11kB
Downloading ./oalinst.exe [ 7/54] ... 790.52kB
Downloading ./locale/ru/LC_MESSAGES/warzone2100.mo [ 8/54] . 404.00b
Downloading ./locale/zh_CN/LC_MESSAGES/warzone2100.mo [ 9/54] . 407.00b
Downloading ./locale/es/LC_MESSAGES/warzone2100.mo [10/54] . 404.00b
Downloading ./locale/la/LC_MESSAGES/warzone2100.mo [11/54] . 404.00b
Downloading ./locale/da/LC_MESSAGES/warzone2100.mo [12/54] . 404.00b
Downloading ./locale/uk/LC_MESSAGES/warzone2100.mo [13/54] . 404.00b
Downloading ./locale/nl/LC_MESSAGES/warzone2100.mo [14/54] . 404.00b
Downloading ./locale/ko/LC_MESSAGES/warzone2100.mo [15/54] . 404.00b
Downloading ./locale/en_GB/LC_MESSAGES/warzone2100.mo [16/54] . 407.00b
Downloading ./locale/zh_TW/LC_MESSAGES/warzone2100.mo [17/54] . 407.00b
Downloading ./locale/sl/LC_MESSAGES/warzone2100.mo [18/54] . 404.00b
Downloading ./locale/cs/LC_MESSAGES/warzone2100.mo [19/54] . 404.00b
Downloading ./locale/pl/LC_MESSAGES/warzone2100.mo [20/54] . 404.00b
Downloading ./locale/et/LC_MESSAGES/warzone2100.mo [21/54] . 404.00b
Downloading ./locale/fy/LC_MESSAGES/warzone2100.mo [22/54] . 404.00b
Downloading ./locale/fr/LC_MESSAGES/warzone2100.mo [23/54] . 404.00b
Downloading ./locale/lt/LC_MESSAGES/warzone2100.mo [24/54] . 404.00b
Downloading ./locale/fi/LC_MESSAGES/warzone2100.mo [25/54] . 404.00b
Downloading ./locale/hr/LC_MESSAGES/warzone2100.mo [26/54] . 404.00b
Downloading ./locale/ro/LC_MESSAGES/warzone2100.mo [27/54] . 404.00b
Downloading ./locale/de/LC_MESSAGES/warzone2100.mo [28/54] . 404.00b
Downloading ./locale/sk/LC_MESSAGES/warzone2100.mo [29/54] . 404.00b
Downloading ./locale/it/LC_MESSAGES/warzone2100.mo [30/54] . 404.00b
Downloading ./locale/nb/LC_MESSAGES/warzone2100.mo [31/54] . 404.00b
Downloading ./locale/pt_BR/LC_MESSAGES/warzone2100.mo [32/54] . 407.00b
Downloading ./locale/ga/LC_MESSAGES/warzone2100.mo [33/54] . 404.00b
Downloading ./locale/pt/LC_MESSAGES/warzone2100.mo [34/54] . 404.00b
Downloading ./warzone2100.exe [35/54] ........................................................................ 20.61mB
Downloading ./styles/readme.screen.css [36/54] . 392.00b
Downloading ./styles/readme.print.css [37/54] . 391.00b
Downloading ./dbghelp.dll.license.txt [38/54] . 12.42kB
Downloading ./AUTHORS [39/54] . 1.86kB
Downloading ./base.wz [40/54] ......................................................................................................... 30.11mB
Downloading ./music/track3.ogg [41/54] . 384.00b
Downloading ./music/track2.ogg [42/54] . 384.00b
Downloading ./music/music.wpl [43/54] . 383.00b
Downloading ./music/track1.ogg [44/54] . 384.00b
Downloading ./music/menu.ogg [45/54] . 382.00b
Downloading ./Readme.de [46/54] . 22.10kB
Downloading ./ChangeLog [47/54] . 103.67kB
Downloading ./fonts/DejaVuSans-Bold.ttf [48/54] . 393.00b
Downloading ./fonts/$PLUGINSDIR/LangDLL.dll [49/54] . 397.00b
Downloading ./fonts/DejaVuSans.ttf [50/54] . 388.00b
Downloading ./fonts/fonts.conf [51/54] . 384.00b
Downloading ./mods/multiplay/ntw.wz [52/54] . 389.00b
Downloading ./mods/multiplay/dydo-ai.wz [53/54] . 393.00b
Downloading ./mods/multiplay/old-1.10-balance.wz [54/54] . 402.00b
noccy@noccy-desktop:~/Development/c/wzupdater/test$ ls
AUTHORS  ChangeLog  dbghelp.dll              fonts   mods   music        Readme.de       Readme.en       styles           wzupdate.latest  wzupdater
base.wz  COPYING    dbghelp.dll.license.txt  locale  mp.wz  oalinst.exe  Readme.de.html  Readme.en.html  warzone2100.exe  wzupdate.md5
noccy@noccy-desktop:~/Development/c/wzupdater/test$

The updater only depends on libcurl, and is intelligent enough to keep track of what version was updated to.

That's it Feel free to put this on the svn, that way people can help hack on it.

[MaNGusT]

such a good thing and no replies from devs

[Wolftrak]

Ow,snap.Haven't tried it,like the idea.
Keep up the good work,noccy.

[Buginator]

The only problem with this approach is, Kamaze doesn't want to be a file host.
That is why all releases are done from SourceForge-- and they don't allow this kind of access to their servers.

[noccy]

That is why all releases are done from SourceForge-- and they don't allow this kind of access to their servers.

Are you sure that isn't that just the load balancing script causing trouble? (lib)Curl downloads files just fine from Sourceforge if you tell it to follow redirects properly. I've seen SourceForge been used like this earlier too.

Would be great to see this piece of code come to use. All it really needs is a front-end (in win32) to act as a launcher/updater. I've been told about the package/update hell on Mac, so it's understandable if it's no use there Still, it would hopefully be able to strip a few old megabytes off any update which would make wonders for anybody with a poor connection.

Thanks for the kinds words MaNGusT and Wolftrak I'm glad you like the idea

[stiv]

such a good thing and no replies from devs

You did not see replies because we discussed the idea on IRC.

[Zarel]

You did not see replies because we discussed the idea on IRC.

What was the conclusion? You should inform us users about what the result of that discussion was.

[Safety0ff]

I wasn't aware of this thread before today, anyways, this is the approach I'd use:
viewtopic.php?p=66859#p66859

Note: There are SVN and Bsdiff python modules, so I think python might be a good choice.

Also, AFAIK you don't have to re-zip the data/base and data/mp directories, some people have said that uncompressed might lead to faster start times.

I have tried a BSDiff of two arbitrary Warzone 2.3 executables (for fun,) and it was 1.3 MB (compare to the 22.4 MB it would take to re-download the whole .exe.)

Caveat: I don't think there was as many differences between the two exe's as there would be between releases, so the diff would likely be bigger in practice.
Caveat2: I didn't test patching and seeing if everything worked.

[noccy]

Note: There are SVN and Bsdiff python modules, so I think python might be a good choice.

We are talking about normal users here, aren't we? Expecting people to install not only svn (or bsdiff) but also python on windows to be able to update sounds a bit over the top to me

Also, AFAIK you don't have to re-zip the data/base and data/mp directories, some people have said that uncompressed might lead to faster start times.

True. In that case just updating the hashes and copying the updated files to the server should be enough to cause the updated files to be updated. Good find, no point in updating monolithic bundles if only one or two files in the bundle have changed.

I have tried a BSDiff of two arbitrary Warzone 2.3 executables (for fun,) and it was 1.3 MB (compare to the 22.4 MB it would take to re-download the whole .exe.)

True. I considered the diff approach too, but the downside is that you are dependent of strict consistency on the client side. If i go all hacky-hacky on my local files, wouldn't that break the diff? If my local files differ with this pretty brute force approach, it would update since the hashes would differ which would indicate the server file is newer. With an added "no-update" magic file developers or hackers can still make modifications without having their hard work trashed.

Besides, any traffic win is a win. If the updates can be cut down to 50% of the full package, that's still a whole lot of bytes saved on slow connections

Also, making it work with Sourceforges mirror system is just a matter of pointing the updates to the main download page and setting the CURLOPT_FOLLOWLOCATION option to 1.

[Safety0ff]

We are talking about normal users here, aren't we? Expecting people to install not only svn (or bsdiff) but also python on windows to be able to update sounds a bit over the top to me

Me too .
In that case something there is svn C api, there is also a C++ wrapper to svn.
BSDiff & patch are written in C and shouldn't be too difficult to integrate as well.

If i go all hacky-hacky on my local files, wouldn't that break the diff?

It would need to have the option of downloading the whole exe even if diff'ing is used.
The output of "warzone2100.exe --version" could be used to complement the hash information.

Besides, any traffic win is a win. If the updates can be cut down to 50% of the full package, that's still a whole lot of bytes saved on slow connections

Yea definitely.
AFAIK (someone correct me if I'm wrong) a fair amount of the exe size comes from linked libraries (which don't change very often.)

Anyways, I'm just bouncing ideas around.

[noccy]

We are talking about normal users here, aren't we? Expecting people to install not only svn (or bsdiff) but also python on windows to be able to update sounds a bit over the top to me

Me too .
In that case something there is svn C api, there is also a C++ wrapper to svn.
BSDiff & patch are written in C and shouldn't be too difficult to integrate as well.

That's true, but unless you can statically link those to the binary it still feels a bit over the top

If i go all hacky-hacky on my local files, wouldn't that break the diff?

It would need to have the option of downloading the whole exe even if diff'ing is used.
The output of "warzone2100.exe --version" could be used to complement the hash information.

Hm, well you could take a look and see if you can implement some sort of diff/patch mechanism The code is pretty straight forward, you would just have to change the file to that of the diff.

Besides, any traffic win is a win. If the updates can be cut down to 50% of the full package, that's still a whole lot of bytes saved on slow connections

Yea definitely.
AFAIK (someone correct me if I'm wrong) a fair amount of the exe size comes from linked libraries (which don't change very often.)

[/quote]

That's correct. Statically linked libraries add to the size of the binary. Curl is pretty lightweight (the updater compiled under Linux is only 230KB when statically linked to it) but every library added adds to the size.

[Safety0ff]

That's correct. Statically linked libraries add to the size of the binary. Curl is pretty lightweight (the updater compiled under Linux is only 230KB when statically linked to it) but every library added adds to the size.

I was talking about the Warzone binary & its size.

[noccy]

That's correct. Statically linked libraries add to the size of the binary. Curl is pretty lightweight (the updater compiled under Linux is only 230KB when statically linked to it) but every library added adds to the size.

I was talking about the Warzone binary & its size.

Yeah, but it still applies to any binary That's just the price you have to pay to avoid the .DLL hell :p

In my opinion an updater should be as light as possible and do nothing but just update. If you are interested in hacking on the updater, let me know and I'll pitch in. We've come up with a number of improvements in this thread, and some of them could definitely be put to the test .

[Safety0ff]

Yeah, but it still applies to any binary That's just the price you have to pay to avoid the .DLL hell :p

Yea, I know, I was just stating it as a good source for further bandwidth savings.

If you are interested in hacking on the updater, let me know and I'll pitch in.

I haven't yet looked at how you have implemented it (none of the above was criticism,) but I will see if I can pitch in once my other WZ related project is closer to being complete.

[noccy]

If you are interested in hacking on the updater, let me know and I'll pitch in.

I haven't yet looked at how you have implemented it (none of the above was criticism,) but I will see if I can pitch in once my other WZ related project is closer to being complete.

Don't worry, none of it was taken as bad criticism There's always room for improvements and experimentation so let's just keep the discussion going

One such improvement would be adding a quick UI while updating. Nothing fancy, a progress bar, a label and a cancel button Would probably look much better in WindowsLand