Stats data format I wish to have.

[Emdek]

sqlite is bad for git. It can`t be diffed.

True, for all binary files...

Sqlite does not have data validation. You can`t tell that this integer should be from -90 to 90. Or weporn type is one of defined words? DTD can.

Not true, you can have check constraints on fields.

EDIT:
We could complicate this even more, like create SQLite files from "sources" (CSV, INI, JSON, whatever) during installation (like currently data is compressed using ZIP format to create .wz files).

[Per]

We had people in the forums make a few tools to edit the stats (2 windows only, 1 cross-platform made recently), and going to ini without tool support was a hasty, premature decision.

Don't bitch at me for not supporting tools I can't run.

As for the diffs, I don't know on linux, but there are visual diff programs that show you exactly what was changed on said line, so I don't think this is a valid argument at all.

Give us a decent way to diff CSV files, and I'll drop that argument against it. But that still won't make it extensible, which is a blocker.

That is only part of the problem.
The way Qt works with ini blows chunks

We use ini in a ton of places now. If you have issues with the ini reader, then this is not an argument against using the ini format for stats, it is an argument against the reader we use, and that is a totally separate issue.

[Shadow Wolf TJC]

Personally, I've been using a simple .txt editor, like WordPad, to edit not only .txt files, but also .js, .pie, .slo, .vlo, and .wrf files to name a few. So long as these .ini files are somewhat readable using a .txt editor, then I'm cool with it.

[aubergine]

I also use a text editor for working with ini files (always have since back in the days of Windows v2.0!) and its fine so long as there are docs explaining the various properties.

If anyone has docs for any of the ini/txt/pie/etc files I'll happily format them up in to full documentation pages that explain each field/property and what it means/affects/etc.

@Duha - if I can get docs together for all the ini/txt/etc files maybe you could create something that takes the data from them and turns them in to an XML file, and then a tool that converts the XML in to ini/txt/etc files? That way those of us that want to work in XML can do, and simply import/export with the other file formats before/after making changes to the XML. We'd then be able to create editors for the XML file and maybe gain some additional support for it once people see what the editors are like?

[iap]

It is important to remember that changing to XML just for the change is useless. XML is a powerful format and has much more advantages over ini, but switching to it must come with a better data design. This will be a huge step, but will also require more refactoring.

No use for changing to XML if the developers are comfortable with less powerful ini.

[aubergine]

What I'm suggesting is that the XML thing can be a side-format not used by the game itself, but instead used for a more reliable way of combining and editing game config and then export the results to more basic ini/txt files.

[vexed]

We had people in the forums make a few tools to edit the stats (2 windows only, 1 cross-platform made recently), and going to ini without tool support was a hasty, premature decision.

Don't bitch at me for not supporting tools I can't run.

You can use wine...but that is besides the point. You changed it without looking at the available tools we have now.

As for the diffs, I don't know on linux, but there are visual diff programs that show you exactly what was changed on said line, so I don't think this is a valid argument at all.

Give us a decent way to diff CSV files, and I'll drop that argument against it. But that still won't make it extensible, which is a blocker.

Are you saying you don't see this when you look at commits ?

As far as extensibility goes, if you were to add data or remove data, then CSV is easy as pie.
With ini, you must manually add/remove that to every entry, and make sure you don't make any mistakes.
Yeah, some trivial code changes would be needed for CSV, but the same is true for the ini way of doing things, is it not ?
I know with ini you can ignore fields, but, it is still trivial to ignore fields with CSV as well.

That is only part of the problem.
The way Qt works with ini blows chunks

We use ini in a ton of places now. If you have issues with the ini reader, then this is not an argument against using the ini format for stats, it is an argument against the reader we use, and that is a totally separate issue.

Ok, I suppose that is true. I guess I can look around for another ini parser, but, I still am not happy with the way ini turned out.
It is fine for config stuff, but for everything else, not so much.

My # 1 priority/concern is ease in debugging. Everything else is secondary.

[Duha]

If old format is not supported it will take not a lot time to clear existing csv from unused fields.
I can write python scripts that convert one csv to another.

Any code with tons of old and unused staff is harder to support.

Ini issues:
There is no default value for flameModel.

Code: Select all

 if (ini.contains("flameModel"))

[Per]

As far as extensibility goes, if you were to add data or remove data, then CSV is easy as pie.
With ini, you must manually add/remove that to every entry, and make sure you don't make any mistakes.
Yeah, some trivial code changes would be needed for CSV, but the same is true for the ini way of doing things, is it not ?
I know with ini you can ignore fields, but, it is still trivial to ignore fields with CSV as well.

Adding and removing columns in a CSV format is easy as pie? *sigh*

I'm left guessing what you are actually suggesting here. Code to look up column names (that mostly do not exist yet), and ignore those we do not recognize? Add code to read only as many columns as is supported, and ignore any new ones we don't, then leave unused columns where they are forever? Or something else?

[vexed]

As far as extensibility goes, if you were to add data or remove data, then CSV is easy as pie.
With ini, you must manually add/remove that to every entry, and make sure you don't make any mistakes.
Yeah, some trivial code changes would be needed for CSV, but the same is true for the ini way of doing things, is it not ?
I know with ini you can ignore fields, but, it is still trivial to ignore fields with CSV as well.

Adding and removing columns in a CSV format is easy as pie? *sigh*

Double *sigh*. You must be joking. Load it into a spreadsheet, or something like notepad++, select column mode, highlight what you want to remove (or add), and then adjust the code accordingly.

With ini, you need to search & replace (if possible), and modify the code (unless you only remove things) and it will take you much longer...and is more error prone.

I'm left guessing what you are actually suggesting here. Code to look up column names (that mostly do not exist yet), and ignore those we do not recognize? Add code to read only as many columns as is supported, and ignore any new ones we don't, then leave unused columns where they are forever? Or something else?

I am suggesting that using ini for stats isn't/wasn't a good idea.
The codebase has become much harder to debug for little (if any) perceived gain.

Making a csv file with a bunch of dummy columns is one way to handle it, and would allow much easier debugging compared to what we have now.

With everyone else talking about XML or whatever else, I *still* have concerns over debugging all this crap, you know how inflexible & fragile wz's codebase is already...heck, debugging the savegame code was actually easier *before* the move to ini with a debugger, now, you are forced to write more temp debug code to get the info you want because of the way Qt obfuscates with template calls what you are looking for as shown above.

[Per]

Double *sigh*. You must be joking. Load it into a spreadsheet, or something like notepad++, select column mode, highlight what you want to remove (or add), and then adjust the code accordingly.

I am not talking about editing, when I talk about extensibility. I am talking about backwards and forwards compatibility. Adding or removing a column in a CSV file breaks such compatibility, unless you do something clever that you have yet to suggest.

I cannot understand how you can fail to misunderstand me that badly. I have been very clear on this the whole time. Go back and read my first post in this thread, for example. It is right there! I have never talked about it being hard to edit, and nobody else has as far as I can remember! To the contrary, I have admitted that CSV is easier to edit.

With everyone else talking about XML or whatever else, I *still* have concerns over debugging all this crap, you know how inflexible & fragile wz's codebase is already...heck, debugging the savegame code was actually easier *before* the move to ini with a debugger, now, you are forced to write more temp debug code to get the info you want because of the way Qt obfuscates with template calls what you are looking for as shown above.

I do not know what debugger you are using, but if it cannot debug anything other than binary data (not QSettings, not any XML reader, not anything else...?), I think you are really sending complaints to the wrong address here.

[Cyp]

A way to make CSV extensible is to have a first row, giving each column a name (like a .ini file does), and reading data based on column name rather than column index. Then columns could be added/removed/reordered.

(I think keeping data as .ini makes more sense (means more chance of successful automatic merges, at least), perhaps with a tool to convert to/from a .csv format to also allow spreadsheet usage.)

[Emdek]

perhaps with a tool to convert to/from a .csv format to also allow spreadsheet usage.

Easy to do (although it would be good to have some entry with list and order of all columns - like comment based header).

[Duha]

@Duha - if I can get docs together for all the ini/txt/etc files maybe you could create something that takes the data from them and turns them in to an XML file, and then a tool that converts the XML in to ini/txt/etc files? That way those of us that want to work in XML can do, and simply import/export with the other file formats before/after making changes to the XML. We'd then be able to create editors for the XML file and maybe gain some additional support for it once people see what the editors are like?

Yes.

My editor has 3 parts.

1. Load data to inner format (sqite database)
2. Editor for inner format
3. Dump to game format

It is easy to add loaders and dumpers to other formats.
After adding dump to ini I will release next version.

[vexed]

Double *sigh*. You must be joking. Load it into a spreadsheet, or something like notepad++, select column mode, highlight what you want to remove (or add), and then adjust the code accordingly.

I am not talking about editing, when I talk about extensibility. I am talking about backwards and forwards compatibility. Adding or removing a column in a CSV file breaks such compatibility, unless you do something clever that you have yet to suggest.

I cannot understand how you can fail to misunderstand me that badly .... To the contrary, I have admitted that CSV is easier to edit.

Ok then, Cyp's method could work, or using the dummy columns--sure not pretty, but works.
We just need proper tool support, and we (would) have none for master, and the current tools still work for 3.1

With everyone else talking about XML or whatever else, I *still* have concerns over debugging all this crap, you know how inflexible & fragile wz's codebase is already...heck, debugging the savegame code was actually easier *before* the move to ini with a debugger, now, you are forced to write more temp debug code to get the info you want because of the way Qt obfuscates with template calls what you are looking for as shown above.

I do not know what debugger you are using, but if it cannot debug anything other than binary data (not QSettings, not any XML reader, not anything else...?), I think you are really sending complaints to the wrong address here.

There is no good way to trace into what Qsettings is doing, if you know of a way, I would love to hear it.
As that example I posted, tell me how I can find what group/childgroup it is parsing from, from that data ?

I am not sure what library we would use if we go with XML or whatever, that is why I said we would have to look at them first, test them out, and see how it goes.