dMZX Forums: MZX 2.90 ZIP-based file formats -> Archived Requests -> Tracker

To squash issues like this and this (at least from 2.85 onward, sorry huntdrak), and also to let minor format updates be slipped in without structural changes, I intend to change the world format... once and for all... />/>

<xml etc>
<mzxml version="255">
  <world version="25A" save="false">
    ...
    <library>
      <source />
      <bytecode />
    </library>
    <programs>
      <program>
        <name />
        <source />
        <bytecode />
      </program>
    </programs>

    <boards>
      <board id="0" position="0">
        <name><![cdata[Super "MZX" Land]]></name>
        <rle2 id="underId">
          <![cdata[...]]>
        </rle2>
        ...
        <robots>
          <robot id="1" x="10" y="20">
            ...(see global)...
          </robot>
          ...
        </robots>
        <scrolls />
        <sensors />
        <savedata>
          <rle2 id="underId">...</rle2>
        </savedata>
      </board>
    </boards>

    <robot id="0" x="-1" y="-1">
      <name><![cdata[GLOBAL]]></name>
      <source><![cdata[...]]></source>
      <bytecode><![cdata[...]]></bytecode>
      <labels>
        <label name="touch" position="0" zapped="false" />
      </labels>
      ...
      <savedata>
        <var name="loop_count" v="-581" />
        ...
      </savedata>
    </robot>

    <savedata>
      <var name="bi_mesg_status" v="1" />
      <counters>
        <ctr name="particle13x" v="59" />
        ...
      </counters>
      <strings>
        <str name="$message"><![cdata["Hello world!"]]></str>
        ...
      </strings>
      <sprites y_order="0">
        <spr id="0" x="20" y="30" w="35" h="18" cw="35" ch="18" cx="0" cy="0" vlayer="1" overlaid="0" />
        ...
      </sprites>
      <vlayer size="32768" width="256" height="128">
        <rle2 id="vlayer">...</rle2>
        <rle2 id="vlayer_color">...</rle2>
      </vlayer>
    </savedata>
  </world>
</mzxml>

<xml>
<mzxml version="255">
  <board>
  </board>
</mzxml>

Obviously doesn't include everything, but gives a general idea. The programs/library stuff is WAAAYYY down the line. The world file size will inflate but the end result in terms of validation and flexibility will be desirable vs. the binary format. I chatted with ajs and essentially got approval on this but I want to know what everyone else thinks as well.

This is a good idea. BTW, we already link zlib for PNG and the updater, so you can deflate the XML to make it smaller (world.xml.gz for example). Another alternative would be to use binary XML, EBML possibly, to keep the size down. That said, I wouldn't be overly worried about size.

You should avoid using attributes, you're definitely using them too much in your example. Take a look at http://www.w3schools..._el_vs_attr.asp which explains why more elements are usually preferable to attributes. Long story short, use attributes for ID references, nothing else. Using attributes sparingly will also simplify your parser, and you may choose to write your own parser.

Don't use base 16 numbers. Either convert them to decimal, or just use a string.

I'd also recommend writing a DTD or XSL (XML schema) for the format. Even if we can't justify adding validation code to MZX proper, it'll still be useful for interop and also for debugging issues.

You should also consider factoring out all the binary data from the format. Modern XML formats like OpenDocument move all binary data into discrete files and then use elements to reference that data in the XML. This eliminates the need to base64 (or otherwise) encode the binary fields, further reducing bloat. I started adding a ZIP loader to MZX which was never completed, this could be extended to handle "world.zip" style files which contained an XML document plus additional binary files.

Of course, there are other options. EBML helps with the binary issue. I've also seen formats which lead with an XML document and then trail with back-to-back binary files, then reference the data by offset in the XML document. Strictly speaking, this breaks the XML encoding rules, but it might be a simple first step towards a final format.

--ajs.

This post has been edited by ajs: 31 May 2012 - 08:24 PM

Upon further review I've realized that XML would be incredibly unfit for something like this given how lenient MZX has always been about naming things (you can name a counter literally anything from char 1-255...). I'm interested in EBML but I need to read up on it more... if we can't find a binary format that fits we may have to construct our own.

I guess you're concerned about debytecode and the fact it will save programs as text scripts, rather than bytecode. I know that other things go in the SAV file, like counter state and strings, but I don't think the format selection should be based around what's optimal for SAV files. These files don't need to be cross-version compatible and could remain binary blobs for now.

I would say if you want to hand-roll the encoding, you should just abandon the idea and fix the existing binary parser. The core issue with the existing format is that the parsers never checks for errors from any of the standard C functions, and don't range check any of the values they read. If the parsers did those things the argument that a re-implementation of the world format improves robustness just evaporates.

--ajs.

I'm more concerned about having to <![cdata[]]> and base64 every field that isn't strictly numeric. A binary format sounds much more appropriate for world data that will almost entirely be binary. I went ahead and tried to figure out EBML last night:

declare header {
  DocType := "mzxebml100";
  EBMLVersion := 1;
}

define types {
  bool := uint [ range: 0..1; ]
  ubyte := uint [ range: 0..255; ]
  uword := uint [ range: 0..65535; ]
  short := int [ range:-2147483648..2147483647; ]
  ascii := binary [ range:32..126; ]
}

define elements {
  MZXEBML := 00abcdef container [ card:1; ] {
    FormatVersion := uword [ card:1; def:0x0100; ]
  }

  World := 00aaaa container [ card:?; parent:MZXEBML; ] {
    WorldVersion := aa00 uword [ card:1; def:0x0255; ]
    IsSaveFile := aa01 bool [ card:1; def:0; ]

    WorldBoards := aabb container [ card:1; ]
    WorldRobot := aacc container [ card:1; ]
    WorldVars := aaee container [ card:1; ]

    WorldSavedata := aaff container [ card:?; ] {
      WorldSaveVars := af00 container [ card:1; ]
      Counters := af01 container [ card:1; ]
      Strings := af02 container [ card:1; ]
      Sprites := af03 container [ card:1; ]

      RLEVlayerChar := af10 binary [ card:1; ]
      RLEVlayerColor := af11 binary [ card:1; ]
    }
  }

  Board := 00bbbb container [ card:*; parent:MZXEBML,Boards,OriginalBoard; ] {
    BoardID := bb00 uint [ card:1; range: 0..249; ]
    BoardOrder := bb01 uint [ card:1; range: 0..249; ]
    BoardName := bb02 binary [ card:1; ]
    BoardWidth := bb03 uword [ card:1; ]
    BoardHeight := bb04 uword [ card:1; ]
    ViewportX := bb05 uint [ card:1; range: 0..79; ]
    ViewportY := bb06 uint [ card:1; range: 0..24; ]
    ViewportWidth := bb07 uint [ card:1; range: 1..80; ]
    ViewportHeight := bb08 uint [ card:1; range: 1..25; ]
    OverlayMode := bb09 uint [ card:1; range 0..3; ]
    BoardModule := bb0f binary [ card:1; ]

    RLEBoardID := bb10 binary [ card:1; ]
    RLEBoardColor := bb11 binary [ card:1; ]
    RLEBoardParam := bb12 binary [ card:1; ]
    RLEBoardUnderID := bb13 binary [ card:1; ]
    RLEBoardUnderColor := bb14 binary [ card:1; ]
    RLEBoardUnderParam := bb15 binary [ card:1; ]
    RLEOverlayChar := bb16 binary [ card:?; ]
    RLEOverlayColor := bb17 binary [ card:?; ]

    BoardScrolls := bb20 container [ card:1; ] {
      Scroll := b2ff container [ card:*; ] {
        ScrollID := b200 ubyte [ card:1; ]
        ScrollText := b201 binary [ card:1; ]
        ScrollIsSign := b202 bool [ card:1; ]
      }
    }
    BoardSensors := bb30 container [ card:1; ] {
      Sensor := b3ff container [ card:*; ] {
        SensorID := b300 ubyte [ card:1; ]
        SensorName := b301 binary [ card:1; ]
        SensorTarget := b302 binary [ card:1; ]
        SensorChar := b303 ubyte [ card:1; ]
      }
    }
    BoardRobots := bbcc container [ card:1; ]
    BoardVars := bbee container [ card:1; ]

    BoardSavedata := bbff container [ card:?; ] {
      BoardSaveVars := bf00 container [ card:1; ]
      OriginalBoard := bf01 container [ card:?; ]
    }
  }

  Robot := 00cccc container [ card:*; parent:WorldRobot,BoardRobots,ImageRobots; ] {
    RobotID := cc00 ubyte [ card:1; ]
    RobotChar := cc12 ubyte [ card:1; ]
    Bytecode := cc80 binary [ card:?; ]
    SourceCode := cc90 binary [ card:?; ]

    RobotSavedata := ccff container [ card:?; ] {
      RobotVars := cf00 container [ card:1; ]
      ProgramData := cf01 containter [ card:1; ] {
        ProgramPosition := cf10 uword [ card:1; ]
      }
      Labels := cf02 container [ card:1; ]
    }
  }

  Image := 00dddd container [ card:?; parent:MZXEBML; ] {
    ImageVersion := dd00 uword [ card:1; ]
    IsLayerImage := dd01 bool [ card:1; ]
    IsSaveImage := dd02 bool [ card:1; ]
    Width := dd03 uword [ card:1; ]
    Height := dd04 uword [ card:1; ]
    RobotCount := dd05 ubyte [ card:?; ]

    ImageData := dd80 binary [ card:1; ]
    ImageRobots := dda0 binary [ card:?; ]
  }

  Var := ee container [ card:*; parent:WorldVars,WorldSaveVars,BoardVars,BoardSaveVars,RobotVars,SpriteVars; ] {
    VarName := e0 ascii [ card:1; ]
    VarByte := e1 ubyte [ card:?; ]
    VarWord := e2 uword [ card:?; ]
    VarShort := e3 short [ card:?; ]
  }

  Counter := 10 container [ card:*; parent:Counters; ] {
    CounterName := 11 binary [ card:1; ]
    CounterValue := 12 short [ card:1; ]
  }

  MZXString := 20 container [ card:*; parent:Strings; ] {
    StringName := 21 binary [ card:1; ]
    StringValue := 22 binary [ card:1; ]
  }

  Sprite := 30 container [ card:*; parent:Sprites; ] {
    SpriteID := 31 ubyte [ card:1; ]
    SpriteVars := 32 container [ card:1; ]
  }

  Label := 90 container [ card:*; parent:Labels; ] {
    LabelName := 91 binary [ card:1; ]
    LabelOffset := 92 uword [ card:1; ]
    LabelZapped := 93 bool [ card:1; def:0; ]
  }
}

That DTD isn't finished (and I haven't even started to learn libebml), but many values are intentionally missing since variables such as "divider" and "bi_mesg_status" would be put in WorldSaveVars, which would also leave room for extensibility. I'm not against simply improving the current loaders, which I feel is mandatory somewhere along the line even if we replaced the world format. The new format obviously would not improve validation of legacy worlds.

Updating status to: Flagged For Future Version
Updating version to: GIT (debytecode)

I think that if I do something like this, it'll be much further down the line than 2.85. I'd love to continue talking about it and refine the EDTD so there's a solid layout, but for now what MZX needs is strict validation on the old format and ideally the ability to recover partial games. I got ahead of myself here

I'm sure you'll be busy with other things, but you might be able to turn this into a slow-burner task by making MZX save out EBML files in parallel to regular files in development builds. This would allow you to slowly add features and validate it without affecting its impact on your release schedule.

IMO the EBML DTD looks very promising and I agree it seems like a better fit than XML. It's great that you've shared your work in progress here.

--ajs.

That's a good idea; I could make it a 0x025A-enabled feature for testing purposes or possibly its own config option until it's stabilized enough to include in a new release.

Now that I have a better idea of how old format validation is done, I think maybe we can thwart accidental false positives in old MZX's version checking by sticking a padding field in at the start of the MZXEBML and filling it with nothing but 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF, it depends though....... I don't think false positives are likely anyhow.

I'm also considering what Exophase said in the "robot parsing" or whatever thread about making saves purely differing data from the world format. This seems feasible and now's much better than later, although I doubt there will be any serious gains by doing this. Boards will have to be saved as a whole due to how essentially everything besides their dimensions and name can change, and therefore their robots must be saved as well. The save data is also titanic compared to the original game data when taking counters, strings, vlayer, compiled bytecode in DBC, and the upcoming "revert board to original state on justentered" feature (which will either require a second copy of the entire board to be stored in the save file, as spec'd above, or access to the original world, which can never be guaranteed is present or even the same as when the save file was created) into consideration.

I propose we use JSON or something smaller than XML. :|

We're using EBML.

If it matters, I support this decision. (Use of EBML)

While I was working on LayerEdit this evening I realized an issue there would be with a format switch like this -- there will be no easy way to tell how large the incoming MZM files will be. Before you could FREAD_OPEN it and get the words at 0x4 and 0x6, but this format change will make that impossible and break anything that uses it (hopefully not many games -- LayerEdit does but I can ensure I'll update it immediately). The fix here will pretty much have to be two part as the functionality I see as ideal will have to wait until Phase 4 of the Exotic roadmap.

2.85 Solution -- the problem here is that it relies on the file being loaded first, not a major issue but still a little sloppy.

put `@filename` image_file p00 whatever nobodycares

if MZM_WIDTH > 20 then `O NO`

Phase 4 Solution -- I see this as Ideal but it could probably use some improvement. The idea is that the file gets decoded into easily accessible global data on its first occurrence since parsing an EBML every single time would probably not be fun

if mzm_width(@filename) > 20
{
  * "O NO!"
  avalanche
  change c?? boulder p?? c?? explosion pf0
}
else
{
  // This doesn't have to parse the file again because we saved it to a
  // cache last time and we can at least tell that MZX hasn't modified it
  // since.  Potential flaw, what if some other program modifies it?
  put @filename image_file p00 (whatever * 5) (nobodycares + 1822)
}

If the former were implemented it would be converted to this function with no argument, and it'd default to whatever the last MZM was. We can't fill a value in at conversion time (because of string evaluation and expressions) or attempt to copy the same param field into it (because of changing variables and things like FREAD) so this is how it would have to be done as far as I can tell.

Updating status to: Awaiting Feedback
Updating version to: 2.85

I want to do a zip file attached to a MZX world header instead. World/board/robot variables could each become their own extensible binary formats (like light ebmls), layers/diffs would be separate files, and so would chrs/palettes/intensities/counters/strings/vlayer/sfx. This would give us the same thing with (hopefully) a fraction of the headache. It's been discussed pretty much a million times before.

The main thing though: to diff boards for save files without including a new copy of every single robot program or a convoluted attempt to map them back to their .mzx counterparts, we'll need to separate robots from programs earlier than planned. I don't see any reason why this can't happen before the debytecode switch. We'll suck it up and add the PROGRAM "" command early, and rewrite LOAD_ROBOT to search for a program, evaluate a checksum to make sure it's the same program (in case somebody changed the file), and if it's not found or the checksum doesn't match, a new program is loaded. INCLUDE "" would have to wait, since it's intended for functions/macros only.

Example filelist:
world
charset.chr
palette.pal2 //RGBI x16 or x256
vlayer_chars.rle2 //if it compresses better we'll just use raw though
vlayer_color.rle2
global_robot
board0
board0_robot1
board0_robot2
board0_scroll0 //lololo
board0_id.rle2
board0_param.rle2
board0_color.rle2
board0_under_id.rle2
board0_under_param.rle2
board0_under_color.rle2
board0_overlay_chars.rle2
board0_overlay_color.rle2
custom_sfx
program0
program1

world_format_v285 {
  ...
  WORLD_NUM_COUNTERS,
  WORLD_NUM_STRINGS,
  WORLD_VLAYER_SIZE,
  WORLD_VLAYER_WIDTH,
  WORLD_VLAYER_HEIGHT,
  WORLD_MULTIPLIER,
  ...
}

New boards and mzms would be saved the same way. Boards/robots/etc could be combined into the same file as the world since they'll basically be read the same way, leaving the bulky parts as the only separate files. We'd basically be dealing with our own proprietary EBML-lite at that point. Which isn't necessarily a bad thing. In fact, the whole thing could be one file and the zip could be completely optional if we decided to save the zip part of it for later

(This is just my 2c)

I think if you go down this route, you shouldn't concoct your own formats for the binary parts. Otherwise you just traded one proprietary format for a bunch of others -- arguably with some small interoperability concessions -- rather than embracing an open standard for formatting data.

Changing the PAL, CHR, or bytecode formats is of questionable use. But I really think the metadata (world/board/robot headers) should not remain as magical binary formats. As you just freed yourself of all the gnarly binary blobs the EBML idea came about to solve, maybe you could just go back to XML, json, or some simple plain text key=value system.

Also, don't RLE compress any data. You just put the files in a ZIP. DEFLATE will take care of making it small.

--ajs.

i dunno

/*
game.mzx
    (header)

    _info
    char
    counter
    gr
    grp
    grs
    pal
    pali
    sfx
    string
    vch
    vco

    b[00-ff]
    b[00-ff]bco
    b[00-ff]bid
    b[00-ff]bpr
    b[00-ff]fco
    b[00-ff]fid
    b[00-ff]fpr
    b[00-ff]och
    b[00-ff]oco
    b[00-ff]r[01-ff]
    b[00-ff]r[01-ff]p
    b[00-ff]r[01-ff]s
    b[00-ff]sc[01-ff]
    b[00-ff]se[01-ff]
*/

/*
board.mzb
    (header)
    _info
    bco
    bid
    bpr
    fco
    fid
    fpr
    och
    oco
    r[01-ff]
    r[01-ff]p
    r[01-ff]s
    sc[01-ff]
    se[01-ff]
*/

/*
image.mzm
    (header)
    (raw data)
    r[01-ff]
    r[01-ff]p
    r[01-ff]s
*/

where (header) and (raw data) preceed any zip data, and "_info" (base world info-always read first) and the base board/robot/scroll/sensor files are something like what ajs suggested.

/*
setting name=data
setting name=data
...
*/

This post has been edited by Lachesis: 23 May 2017 - 04:54 AM

Initial commit of zip handler in GIT 78e997d. Final specs of the zip world/board/MZM format TBD.

Metadata format (little endian):

/*
repeat k times:
{
  2 bytes -- identifier
  4 bytes -- value length (n)
  n bytes -- value
}

2 bytes -- 0 (end of metadata)
*/

Updating status to: Confirmed

As of GIT 530178a in the zipworld branch, worlds, saves, and counters saves can be saved and loaded with no (known) issues.

Updating status to: Implemented
Issue fixed in: GIT

Implemented in GIT 2d3690e.