Stats data format I wish to have.

[Duha]

I here about some mentions about changes in stats structure, here is my ideas.

1. Supporting many formats at once.

Code: Select all

If version.txt exsists then use parser_{version};
else use old_parser;

This will allow you to add new functionality with out bracking old.

2. Best format.
There is many human readable formats for contains data: csv, json, xml, ...
But then I get data from outside (for example mods) I prefer xml. It has many disadvantiges but one thing is awesome:
Validation by scheme.

Spoiler:

http://xmlsoft.org/xmldtd.html

Well what is validation and what is a DTD ?

DTD is the acronym for Document Type Definition. This is a description of the content for a family of XML files. This is part of the XML 1.0 specification, and allows one to describe and verify that a given document instance conforms to the set of rules detailing its structure and content.

Validation is the process of checking a document against a DTD (more generally against a set of construction rules).

The validation process and building DTDs are the two most difficult parts of the XML life cycle. Briefly a DTD defines all the possible elements to be found within your document, what is the formal shape of your document tree (by defining the allowed content of an element; either text, a regular expression for the allowed list of children, or mixed content i.e. both text and children). The DTD also defines the valid attributes for all elements and the types of those attributes.

It gives some bonus:
You forced to write self documented data. (I can convert to html to show in wiki.)
You validate in one string. Some value checks from code will be moved to schema.


Speed and memory usage:
SAX xml parsers propably will work with same memory usage and speed.

3.Convering data from old to new format
I can write python scripts what do it. (it is more easy when writing it on c(c++))

Some other Ideas:
Names should be included inside files. name.txt should not be edited manualy (some script should extract all names there, and after that you just run you gettext script). It make stats more human readable.

My veiw to armor:
One armor - one element:
hard: ap=10, antitank=20, antibunker=30, antifire=40, .....
soft: ap=10, antitank=20, antibunker=30, antifire=40, .....
bunker: ap=10, antitank=20, antibunker=30, antifire=40, .....

Research:
research components and research structures can be merged (add type STRUCTURE and make less files.)

Reusage of stats:
I dont know how works cam1, cam2, cam3 swithcing. But propably mod can be counstructed by some parts:
mod_descriptor.txt:

Code: Select all

research = cam1
units  = base
strings = cam1
images = base
templates= multiplayer
....  //not in list use base

and mode folder structure:

Code: Select all

base/
base/research
base/units
...
cam1/
cam1/research
cam1/strings/  

This will allow to mode only part of game. And launching mode will be mode=mode_descriptor.txt
base and mp will be moved to mods folder.

[NoQ]

We're switching from csv to ini right now, afaik ...

[aubergine]

@Duha: +1 for XML and DTD!

It should be relatively easy to convert an ini file in to an XML file and then start working on a DTD.

My only worry with the DTD is that it might inhibit the hackability of the data files. For example, someone might be able to easily change the data in an XML file and the C++ code to handle it, but they might find a DTD a more unfamiliar thing and struggle updating it.

That being said, having a DTD will make it possible to create much better error messages should the main data XML contain errors - and we'd be able to create documentation from the DTD as well I presume?

For AI scripts to query the file, the game engine would likely need to load and parse the file (eg. the file itself might in a zipped .wz file which is something JS can't really access). Thus, the JS API could expose some new global objects such as researchTree (would contain js object representation of the research xml file) and so on.

Also, it would likely be possible to have XML for specific "challenge" games as well, simply by referencing the relevant XML file in the challenge's ini file (I can't remember name convention of file off top of my head, sorry).

[Duha]

- and we'd be able to create documentation from the DTD as well I presume?

If there is not libs for it I can easy write it.

For AI scripts to query the file, the game engine would likely need to load and parse the file (eg. the file itself might in a zipped .wz file which is something JS can't really access). Thus, the JS API could expose some new global objects such as researchTree (would contain js object representation of the research xml file) and so on.

JSON can be sent to javascript with out parsing.
Looks like it has some validation: http://davidwalsh.name/json-validation.

[Iluvalar]

We're switching from csv to ini right now, afaik ...

No, we wont after all.

And I strongly believe we'll stick with csv since I talked to vexed.

The reason is : csv format separate data with only one character. Others stuff like xml or ini generate huge amount of unuseful character.

So a mod that could be easily passed to another player in 20ko of csv, might turn out to be a 1Mo beast in .ini files. The gain for human readability is not so great anyway, it's easy to use a reader or my php modding functions instead.

[NoQ]

We already did to some little extent ... will body.ini get deprecated again?

[cybersphinx]

And I strongly believe we'll stick with csv since I talked to vexed.

Try talking to Per then...

[Reg312]

@Iluvalar +1
csv format is simple and this is main benefit

i see current problem in bad parser, in some cases wrong stats format may cause access violations or other weird errors
imo code is too complex for simple loading data from strings

[Iluvalar]

I know that's per that asked _ME_ to look at that and convert the other files since I have the tool to do so with little effort.

But now I believe, as a modder, that it is simply worst than before.

[aubergine]

JSON can be sent to javascript with out parsing.
Looks like it has some validation: http://davidwalsh.name/json-validation.

Yes, I know JSON won't need parsing (it's exactly same format as javascript anyway so it just gets processed like all other sctipts).

The problem arises if there are, for example, multiple different research configs - the JS environment needs to be told which one to use. Also, because these configs will often find them in zipped .wz files, JS can't just read them -- the C++ code would need to grab them from within the .wz file and just make them available in the JS env (quite easy to do as there are already functions for all that kind of stuff in the C++ code).

In terms if ini files being easier to edit, it would be very easy to create a browser based editor for the XML or JSON files - simply you could have a table view (much like looking at a CSV in a spreadsheet app) and edit inline within the table or view a more detailed page for editing (whatever is preferred). So I wouldn't worry too much about how things are edited and file size is also not a valid reason not to switch to a better format IMHO.

For me, having a more accurate and testable transport for the data (such as XML with a DTD) will greatly improve consistency, testability and portability of the data.

[iap]

Two very important features of XML, that you should keep in mind if you want to consider it are Hierarchy and Attributes.
While INI and CSV are generally linear, XML (and JSON) can contain a much more complex data representation.
Switching to one of these just for the sake of changing, is useless, but redesigning the data structures to represent better the game internals, using hierarchy and node attributes is a HUGE step forward. But of course this would require more efforts witch is something you should concern...

From my experience, a good data representation is priceless in the sense of making the entire program "stick" to a unified data model, and if it's good, it will prevent lots of bugs and glitches that would have present themselves otherwise.

Unfortunately, although I enjoy very much playing the game and reading all the discussions in the forum, I didn't have time to really examine the internals of the data (But I really do want to do my own mod... maybe in the year 2100... ).

[Duha]

The reason is : csv format separate data with only one character. Others stuff like xml or ini generate huge amount of unuseful character.

pffff.

Spoiler:

Txt:
stats 60files 235kbites.
XML (lets multiply to 50 for example) # json has multipler 10
235*50 = 11750 kbites

My RAM:
4057756 kbites
Memory usage less then persent:
text: 0.0058 % of RAM
xml: 0.2896 % of RAM

This topic not about modding. It is about codding. And code documentation. And code support.

@Iluvalar +1
csv format is simple and this is main benefit

Simple format not means simple usage.
For me JSON is more simple (it is native to Python).
But if I need validation of data it is better DTD. It allow me to not write less code for validation. And make less chance for human_errors.

i see current problem in bad parser, in some cases wrong stats format may cause access violations or other weird errors
imo code is too complex for simple loading data from strings

IMHO parser should be changed. IMHO it should be one function for all files that get on file and schema, and return data structure.
My editor and you editor make it in that way. IMHO it is good time now to doit.

Yes, I know JSON won't need parsing (it's exactly same format as javascript anyway so it just gets processed like all other sctipts).

The problem arises if there are, for example, multiple different research configs - the JS environment needs to be told which one to use.

I don`t see wz js code yet.
I expect somthing like this:
var currentResearch = getFile(research_name)

Also, because these configs will often find them in zipped .wz files, JS can't just read them -- the C++ code would need to grab them from within the .wz file and just make them available in the JS env (quite easy to do as there are already functions for all that kind of stuff in the C++ code).

You can just unpack it in C and send to js.
How many time you win when using zip files? Is it speedup only level loading?

In terms if ini files being easier to edit, it would be very easy to create a browser based editor for the XML or JSON files - simply you could have a table view (much like looking at a CSV in a spreadsheet app) and edit inline within the table or view a more detailed page for editing (whatever is preferred). So I wouldn't worry too much about how things are edited and file size is also not a valid reason not to switch to a better format IMHO.

My editor parse stats to sql database. I can easy write dump/load functions for any format. Thing that realy slow down me is absence of current docks.

For me, having a more accurate and testable transport for the data (such as XML with a DTD) will greatly improve consistency, testability and portability of the data.

+1 This is main reason of this topic.

Two very important features of XML, that you should keep in mind if you want to consider it are Hierarchy and Attributes.

It can be solved by convention. Use only attrs or only herarchy.

While INI and CSV are generally linear, XML (and JSON) can contain a much more complex data representation.

We can not use this complexity. But some many to many relation can be done via it (structureweaporns.txt can be inside structure file. for example.)

Switching to one of these just for the sake of changing, is useless, but redesigning the data structures to represent better the game internals, using hierarchy and node attributes is a HUGE step forward. But of course this would require more efforts witch is something you should concern...
From my experience, a good data representation is priceless in the sense of making the entire program "stick" to a unified data model, and if it's good, it will prevent lots of bugs and glitches that would have present themselves otherwise.

+1

[NoQ]

My RAM:

We're not talking about RAM. We're talking about download size. Mod size (limited by forum attachment size limit) or map-mod size (that is to be downloaded on joining the game). 235kb for a mod that changes all stats is considerably better than 11mb for the same mod, in this sense.

[Per]

I hope we are talking about readability, not mere size. In which case, XML is a horrible choice. JSON is somewhat better. ini, I think, is the most easily readable format. We also have a parser already. Hammer, meet nail.

In any case, my reason for wanting to change format was merely because CSV is almost impossible to read changes in (since diffs touch so many values at once), and almost impossible to change (adding / removing columns breaks compatibility, which is bad). And I really want to make some changes.

[Duha]

I hope we are talking about readability, not mere size. In which case, XML is a horrible choice. JSON is somewhat better. ini, I think, is the most easily readable format. We also have a parser already. Hammer, meet nail.

weapons.txt has 53 entries. It is not human readable in csv. In that case well formatted csv will be better.
IMHO if make some tools, human readbility will not neccessary.

xml allow to skip some fields. (in weapons.txt some field used only under certain condition.)

and almost impossible to change (adding / removing columns breaks compatibility, which is bad). And I really want to make some changes.

It looks like everyone whant to change it

I dont know c++.
In python it easy write function that will read DTD and create parser for this schema.

My editor has ~650 lines of model schemas. And about 200 string of code that loads it to my database.

You have:
~3500 lines in stats.cpp
~7500 lines in structure.cpp

There is a lot of same functions. get_max..., set_max..., load...
Maybe using classes and inheritance will help. It is easy to do it in python but I dont know how to do it in c++
Looks like parsers is mixed in some other function. It is good idea to separate them.

My RAM:

We're not talking about RAM. We're talking about download size. Mod size (limited by forum attachment size limit) or map-mod size (that is to be downloaded on joining the game). 235kb for a mod that changes all stats is considerably better than 11mb for the same mod, in this sense.

Forum attachment should not be the reason. For fast files I use free ftp on yandex.ru or dropbox and for long my github account.
Map-mod size that downloaded on joining game may be packed (text packs for 95-99%).