dMZX Forums: Improved Validation for Legacy Formats -> Archived Requests -> Tracker

Merged from several different corrupt world reports. MZX currently barely does anything in terms of validation for worlds, boards, robots, and MZMs, leading to memory errors, falsely identified locked worlds, swap world crashes, MZM freezes...

Validation checks will be run on a particular piece of data before that data is loaded. Different validation results will result in different handling of aspects of the load. The term "blank out" means the following:
Robot - replace with default settings and an empty program.
Board - replace with a 100x100 blank default board.

Flow:
World Validation - Magic, encryption, value bounds verification, truncation, custom sfx table. If this fails, nothing happens -- the current world is allowed to continue running. If it succeeds, world loading is started, so all subsequent steps must result in the final world being able to run without error.

• Boards
  
  • Board Validation - Dimensions, RLE2s, truncation, robot/scroll/sensor counts.
    
    • Robot/Program Validation
      
    • Scroll Validation
      
    • Sensor Validation
    
    
  • Board Scan - Replace scrolls, robots, and signs that would try accessing uninitialized data
  
  
• Global Robot

Known problematic worlds:
d00d, visit my website (wiki page)!

I should note this has changed a lot as I've actually been writing the code for it and I'll need to update the first post tomorrow. As far as it is, it covers the contents of try_load_world with added truncation checks for both world and 2.84 save files. I went through and counted the size of both by hand, since it's good to know the entirety of load_world.

Partial implementation as of GIT master/e485362. So far I've only written basic validation (no boards or robots) for worlds, saves, and mzms, and the validation call for the later hasn't actually been inserted or tested yet. The world and save validation seems to at least be good enough to kill maybe 99% of files that aren't really MZX worlds before it tries to load them. It could easily replace the majority of try_load_world.

While I was at it, I made world.c's fgetw and fgetd functions (semi) safe. When either detects an EOF in one of the bytes it reads, it simply passes EOF through (instead of trying to treat it like a byte). For the latter in validation cases this will be fine since if it can be a negative number it will not be an offset or length, so we'll just fseek over it during validation anyway.

Boards are now validated as thoroughly as I care to for now, as of GIT master/047b9966. All that's left is Robots, Scrolls, Sensors, and MZMs, from hardest to easiest. You can now actually load 9.mzx in MegaZeux (you end up with a blank 80x25 board though and the global robot is still very corrupt). I should probably change that so it puts a player somewhere on the blank board.

Trace for accessing 9.mzx's global, posted because I was asked. Windows XP 32-bit on GIT master/8b90f7decf.

Program received signal SIGSEGV, Segmentation fault.
0x70419ce4 in move_line_down (rstate=0x23eab8, count=1)
    at src/editor/robo_ed.c:1961
1961        if(rstate->current_rline->next == NULL)
(gdb) bt
#0  0x70419ce4 in move_line_down (rstate=0x23eab8, count=1)
    at src/editor/robo_ed.c:1961
#1  0x7041c1f2 in robot_editor (mzx_world=0x40c320, cur_robot=0x40c83c)
    at src/editor/robo_ed.c:3343
#2  0x70411eb2 in global_robot (mzx_world=0x40c320)
    at src/editor/edit_di.c:1262
#3  0x70411d63 in global_info (mzx_world=0x40c320)
    at src/editor/edit_di.c:1237
#4  0x7040c2a0 in __edit_world (mzx_world=0x40c320, reload_curr_file=1)
    at src/editor/edit.c:2212
#5  0x618d675f in title_screen (mzx_world=0x40c320) at src/game.c:2478
#6  0x00403442 in SDL_main (argc=1, argv=0x9b3f90) at src/main.c:163
#7  0x00404312 in console_main (argc=1, argv=0x9b3f90)
    at ./src/main/win32/SDL_win32_main.c:315
#8  0x004043c8 in WinMain@16 (hInst=0x400000, hPrev=0x0,
    szCmdLine=0x251ef9 "", sw=10) at ./src/main/win32/SDL_win32_main.c:398
#9  0x00403aba in main (
    argc=<error reading variable: Cannot access memory at address 0x0>,
    argv=<error reading variable: Cannot access memory at address 0x4>,
    __p__environ=<error reading variable: Cannot access memory at address 0x8>)
at ../mingw/main.c:73

HUNTDRAK.MZX will load as of master/3609504. All that's left is MZMs and maybe malformed program checks (if we keep having errors like the one you pasted). For what it's worth I never have issues accessing the global in my build, maybe it's an arch inconsistency :X

I forced MZX files with bad CRCs out of the infamous broken FF:DP project dump archive, and MZX segfaulted when it was listing the problems with Robots. master/360950424b:

Program received signal SIGSEGV, Segmentation fault.
0x7c970441 in ntdll!RtlpNtMakeTemporaryKey ()
   from C:\WINDOWS\system32\ntdll.dll
(gdb) bt
#0  0x7c970441 in ntdll!RtlpNtMakeTemporaryKey ()
   from C:\WINDOWS\system32\ntdll.dll
#1  0x7c94bafc in ntdll!LdrFindEntryForAddress ()
   from C:\WINDOWS\system32\ntdll.dll
#2  0x009b0000 in ?? ()
#3  0x7c91a1ba in ntdll!RtlpUnWaitCriticalSection ()
   from C:\WINDOWS\system32\ntdll.dll
#4  0x77c2c2de in msvcrt!free () from C:\WINDOWS\system32\msvcrt.dll
#5  0x009b0000 in ?? ()
#6  0x618e26c9 in clear_robot_contents (cur_robot=0x110fb20)
    at src/robot.c:666
#7  0x618e2734 in clear_robot (cur_robot=0x110fb20) at src/robot.c:694
#8  0x618c2399 in load_board_direct (cur_board=0x10f7098, fp=0x77c5fd00,
    data_size=20274, savegame=0, version=521) at src/board.c:418
#9  0x618c27c3 in load_board_allocate (fp=0x77c5fd00, savegame=0, version=521)
    at src/board.c:552
#10 0x61900ce1 in load_world (mzx_world=0x40c320, fp=0x77c5fd00,
    file=0x6199a000 "FFTOWNO.MZX", savegame=false, version=521,
    name=0x23f983 "Town1", faded=0x23f9c8) at src/world.c:1510
#11 0x61901560 in reload_world (mzx_world=0x40c320,
    file=0x6199a000 "FFTOWNO.MZX", faded=0x23f9c8) at src/world.c:1691
#12 0x618d1b9f in load_world_file (mzx_world=0x40c320,
    name=0x6199a000 "FFTOWNO.MZX") at src/game.c:172
#13 0x618d1ca6 in load_world_selection (mzx_world=0x40c320) at src/game.c:193
#14 0x618d61be in title_screen (mzx_world=0x40c320) at src/game.c:2249
#15 0x00403442 in SDL_main (argc=1, argv=0x9b3f90) at src/main.c:163
#16 0x00404312 in console_main (argc=1, argv=0x9b3f90)
    at ./src/main/win32/SDL_win32_main.c:315
#17 0x004043c8 in WinMain@16 (hInst=0x400000, hPrev=0x0,
    szCmdLine=0x251ef9 "", sw=10) at ./src/main/win32/SDL_win32_main.c:398
#18 0x00403aba in main (argc=2356584, argv=0x240000, __p__environ=0x204000)
    at ../mingw/main.c:73

A bad world file that causes megazeux to seize up on loading. I think it might actually be a board file or an mzm. http://www.mediafire...p7snvkn75ly53y3

This post has been edited by Old-Sckool: 11 June 2012 - 02:46 AM

It's definitely a world file. I'm still trying to figure out what's going wrong in it.

As of master/c1a9d7d, instacrash.mzx loads. The problem there was the end of the board data was corrupt (despite both robots being intact), causing it to return the wrong number of robots on the board. MZX tried to run a robot that was not initialized, and... well, you know what happens. MZX now does a board scan for instances of this and replaces robots, scrolls, signs, and sensors if it becomes aware of this condition, and displays error summaries. This is not a "fix", this is simply the minimum required to get the world to successfully load.

I also added bytecode validation, although it doesn't go through command by command making sure it's correct. z1111000.mzx's robots are now correctly identified as being corrupt, as is 9.mzx's global robot.

I'm hoping somewhere along the line one of us can write a program that scans world files, finds valid legacy bytecode programs, and dumps it to legacy source code, if something like this doesn't exist already. It shouldn't be difficult (but maybe processor intensive).

GIT master/5dbf407 properly ignores empty programs in Catacombs now, and does not count or replace the bad sensors since they can safely be left unloaded, they're all over old games, and nobody really cares about them.

instacrash.mzx may load, but it appears that the robot still executes it's code?, as you cannot move the player south and there's no code in the global. If you save the world as a different world, megazeux thinks it's still valid but the player is still locked NS until you move it.

This post has been edited by Old-Sckool: 13 June 2012 - 04:38 PM

That's an artifact of the corrupt board data. The locked_player_ns flag is set to 63 (it's the first corrupt byte). It's not unreasonable to improve validation for certain things here (board exits strike me as another thing).

Made a change in master/44f1e9e. Worlds where the robot program length is longer than the actual program, like Aura and Chronos Stasis, were giving error dialogs. Now it just leaves a debug message and wipes the rest of the allocated program.

DoZ 2008 Q3, Entry 76722 generates corrupt saves. Save_Game by itself is not corrupting things, you must watch the stupid intro first for save_game to corrupt the save. GIT only.

This post has been edited by Old-Sckool: 13 June 2012 - 07:55 PM

Fixed in master/4291032. I miscounted the sizes of two sections of the save format and somehow things were loading in spite of it.

As of master/aea7c4a, lptws2in.mzx loads. It was segfaulting due to several board values having never been initialized, but board validation should have been catching it as well so I updated it.

ca08848... is reporting that Robot @ 44FBh is truncated or corrupt in Loco.mzx

That's because it is.

From the hex:

   Len CMD Str "a" Ter ??? Len
...04  1D  02  61  00  7E  04...

There seems to be further corruption in the bytecode following that.

List of things in the archive with listed corruption so far, and aren't caused by people having saved the world in a freaking 2.83 SVN (lol, Pretentia v1):

• CR2-6
  
• EP + ARP World MZX
  
• SZZT7_ori.mzx (in George Great Adventure)
  
• Slave Pit
  
• Civil War 2040 (requires decrypting first)

I'm nowhere near done looking.