Warzone 2100 Forums

I here about some mentions about changes in stats structure, here is my ideas.

1. Supporting many formats at once. This will allow you to add new functionality with out bracking old.

2. Best format.
There is many human readable formats for contains data: csv, json, xml, ...
But then I get data from outside (for example mods) I prefer xml. It has many disadvantiges but one thing is awesome:
Validation by scheme. It gives some bonus:
You forced to write self documented data. (I can convert to html to show in wiki.)
You validate in one string. Some value checks from code will be moved to schema.


Speed and memory usage:
SAX xml parsers propably will work with same memory usage and speed.

3.Convering data from old to new format
I can write python scripts what do it. (it is more easy when writing it on c(c++))

Some other Ideas:
Names should be included inside files. name.txt should not be edited manualy (some script should extract all names there, and after that you just run you gettext script). It make stats more human readable.

My veiw to armor:
One armor - one element:
hard: ap=10, antitank=20, antibunker=30, antifire=40, .....
soft: ap=10, antitank=20, antibunker=30, antifire=40, .....
bunker: ap=10, antitank=20, antibunker=30, antifire=40, .....

Research:
research components and research structures can be merged (add type STRUCTURE and make less files.)

Reusage of stats:
I dont know how works cam1, cam2, cam3 swithcing. But propably mod can be counstructed by some parts:
mod_descriptor.txt: and mode folder structure: This will allow to mode only part of game. And launching mode will be mode=mode_descriptor.txt
base and mp will be moved to mods folder.

We're switching from csv to ini right now, afaik ...

@Duha: +1 for XML and DTD!

It should be relatively easy to convert an ini file in to an XML file and then start working on a DTD.

My only worry with the DTD is that it might inhibit the hackability of the data files. For example, someone might be able to easily change the data in an XML file and the C++ code to handle it, but they might find a DTD a more unfamiliar thing and struggle updating it.

That being said, having a DTD will make it possible to create much better error messages should the main data XML contain errors - and we'd be able to create documentation from the DTD as well I presume?

For AI scripts to query the file, the game engine would likely need to load and parse the file (eg. the file itself might in a zipped .wz file which is something JS can't really access). Thus, the JS API could expose some new global objects such as researchTree (would contain js object representation of the research xml file) and so on.

Also, it would likely be possible to have XML for specific "challenge" games as well, simply by referencing the relevant XML file in the challenge's ini file (I can't remember name convention of file off top of my head, sorry).

aubergine wrote: - and we'd be able to create documentation from the DTD as well I presume?

If there is not libs for it I can easy write it.

For AI scripts to query the file, the game engine would likely need to load and parse the file (eg. the file itself might in a zipped .wz file which is something JS can't really access). Thus, the JS API could expose some new global objects such as researchTree (would contain js object representation of the research xml file) and so on.

JSON can be sent to javascript with out parsing.
Looks like it has some validation: http://davidwalsh.name/json-validation.

NoQ wrote:We're switching from csv to ini right now, afaik ...

No, we wont after all.

And I strongly believe we'll stick with csv since I talked to vexed.

The reason is : csv format separate data with only one character. Others stuff like xml or ini generate huge amount of unuseful character.

So a mod that could be easily passed to another player in 20ko of csv, might turn out to be a 1Mo beast in .ini files. The gain for human readability is not so great anyway, it's easy to use a reader or my php modding functions instead.

We already did to some little extent ... will body.ini get deprecated again?

Iluvalar wrote:And I strongly believe we'll stick with csv since I talked to vexed.

Try talking to Per then...

@Iluvalar +1
csv format is simple and this is main benefit

i see current problem in bad parser, in some cases wrong stats format may cause access violations or other weird errors
imo code is too complex for simple loading data from strings

I know that's per that asked _ME_ to look at that and convert the other files since I have the tool to do so with little effort.

But now I believe, as a modder, that it is simply worst than before.

Duha wrote:JSON can be sent to javascript with out parsing.
Looks like it has some validation: http://davidwalsh.name/json-validation.

Yes, I know JSON won't need parsing (it's exactly same format as javascript anyway so it just gets processed like all other sctipts).

The problem arises if there are, for example, multiple different research configs - the JS environment needs to be told which one to use. Also, because these configs will often find them in zipped .wz files, JS can't just read them -- the C++ code would need to grab them from within the .wz file and just make them available in the JS env (quite easy to do as there are already functions for all that kind of stuff in the C++ code).

In terms if ini files being easier to edit, it would be very easy to create a browser based editor for the XML or JSON files - simply you could have a table view (much like looking at a CSV in a spreadsheet app) and edit inline within the table or view a more detailed page for editing (whatever is preferred). So I wouldn't worry too much about how things are edited and file size is also not a valid reason not to switch to a better format IMHO.

For me, having a more accurate and testable transport for the data (such as XML with a DTD) will greatly improve consistency, testability and portability of the data.

Two very important features of XML, that you should keep in mind if you want to consider it are Hierarchy and Attributes.
While INI and CSV are generally linear, XML (and JSON) can contain a much more complex data representation.
Switching to one of these just for the sake of changing, is useless, but redesigning the data structures to represent better the game internals, using hierarchy and node attributes is a HUGE step forward. But of course this would require more efforts witch is something you should concern...

From my experience, a good data representation is priceless in the sense of making the entire program "stick" to a unified data model, and if it's good, it will prevent lots of bugs and glitches that would have present themselves otherwise.

Unfortunately, although I enjoy very much playing the game and reading all the discussions in the forum, I didn't have time to really examine the internals of the data (But I really do want to do my own mod... maybe in the year 2100... ).

Iluvalar wrote:The reason is : csv format separate data with only one character. Others stuff like xml or ini generate huge amount of unuseful character.

pffff. This topic not about modding. It is about codding. And code documentation. And code support.

Reg312 wrote:@Iluvalar +1
csv format is simple and this is main benefit

Simple format not means simple usage.
For me JSON is more simple (it is native to Python).
But if I need validation of data it is better DTD. It allow me to not write less code for validation. And make less chance for human_errors.

i see current problem in bad parser, in some cases wrong stats format may cause access violations or other weird errors
imo code is too complex for simple loading data from strings

IMHO parser should be changed. IMHO it should be one function for all files that get on file and schema, and return data structure.
My editor and you editor make it in that way. IMHO it is good time now to doit.

aubergine wrote:

Duha wrote: Yes, I know JSON won't need parsing (it's exactly same format as javascript anyway so it just gets processed like all other sctipts).

The problem arises if there are, for example, multiple different research configs - the JS environment needs to be told which one to use.

I don`t see wz js code yet.
I expect somthing like this:
var currentResearch = getFile(research_name)

Also, because these configs will often find them in zipped .wz files, JS can't just read them -- the C++ code would need to grab them from within the .wz file and just make them available in the JS env (quite easy to do as there are already functions for all that kind of stuff in the C++ code).

You can just unpack it in C and send to js.
How many time you win when using zip files? Is it speedup only level loading?

In terms if ini files being easier to edit, it would be very easy to create a browser based editor for the XML or JSON files - simply you could have a table view (much like looking at a CSV in a spreadsheet app) and edit inline within the table or view a more detailed page for editing (whatever is preferred). So I wouldn't worry too much about how things are edited and file size is also not a valid reason not to switch to a better format IMHO.

My editor parse stats to sql database. I can easy write dump/load functions for any format. Thing that realy slow down me is absence of current docks.

For me, having a more accurate and testable transport for the data (such as XML with a DTD) will greatly improve consistency, testability and portability of the data.

+1 This is main reason of this topic.

iap wrote:Two very important features of XML, that you should keep in mind if you want to consider it are Hierarchy and Attributes.

It can be solved by convention. Use only attrs or only herarchy.

While INI and CSV are generally linear, XML (and JSON) can contain a much more complex data representation.

We can not use this complexity. But some many to many relation can be done via it (structureweaporns.txt can be inside structure file. for example.)

Switching to one of these just for the sake of changing, is useless, but redesigning the data structures to represent better the game internals, using hierarchy and node attributes is a HUGE step forward. But of course this would require more efforts witch is something you should concern...
From my experience, a good data representation is priceless in the sense of making the entire program "stick" to a unified data model, and if it's good, it will prevent lots of bugs and glitches that would have present themselves otherwise.

+1

My RAM:

We're not talking about RAM. We're talking about download size. Mod size (limited by forum attachment size limit) or map-mod size (that is to be downloaded on joining the game). 235kb for a mod that changes all stats is considerably better than 11mb for the same mod, in this sense.

I hope we are talking about readability, not mere size. In which case, XML is a horrible choice. JSON is somewhat better. ini, I think, is the most easily readable format. We also have a parser already. Hammer, meet nail.

In any case, my reason for wanting to change format was merely because CSV is almost impossible to read changes in (since diffs touch so many values at once), and almost impossible to change (adding / removing columns breaks compatibility, which is bad). And I really want to make some changes.

Per wrote:I hope we are talking about readability, not mere size. In which case, XML is a horrible choice. JSON is somewhat better. ini, I think, is the most easily readable format. We also have a parser already. Hammer, meet nail.

weapons.txt has 53 entries. It is not human readable in csv. In that case well formatted csv will be better.
IMHO if make some tools, human readbility will not neccessary.

xml allow to skip some fields. (in weapons.txt some field used only under certain condition.)

and almost impossible to change (adding / removing columns breaks compatibility, which is bad). And I really want to make some changes.

It looks like everyone whant to change it

I dont know c++.
In python it easy write function that will read DTD and create parser for this schema.

My editor has ~650 lines of model schemas. And about 200 string of code that loads it to my database.

You have:
~3500 lines in stats.cpp
~7500 lines in structure.cpp

There is a lot of same functions. get_max..., set_max..., load...
Maybe using classes and inheritance will help. It is easy to do it in python but I dont know how to do it in c++
Looks like parsers is mixed in some other function. It is good idea to separate them.

NoQ wrote:

My RAM:

We're not talking about RAM. We're talking about download size. Mod size (limited by forum attachment size limit) or map-mod size (that is to be downloaded on joining the game). 235kb for a mod that changes all stats is considerably better than 11mb for the same mod, in this sense.

Forum attachment should not be the reason. For fast files I use free ftp on yandex.ru or dropbox and for long my github account.
Map-mod size that downloaded on joining game may be packed (text packs for 95-99%).