dMZX Forums: Improved Validation for Legacy Formats -> Archived Requests -> Tracker

Merged from several different corrupt world reports. MZX currently barely does anything in terms of validation for worlds, boards, robots, and MZMs, leading to memory errors, falsely identified locked worlds, swap world crashes, MZM freezes...

Validation checks will be run on a particular piece of data before that data is loaded. Different validation results will result in different handling of aspects of the load. The term "blank out" means the following:
Robot - replace with default settings and an empty program.
Board - replace with a 100x100 blank default board.

Flow:
World Validation - Magic, encryption, value bounds verification, truncation, custom sfx table. If this fails, nothing happens -- the current world is allowed to continue running. If it succeeds, world loading is started, so all subsequent steps must result in the final world being able to run without error.

• Boards
  
  • Board Validation - Dimensions, RLE2s, truncation, robot/scroll/sensor counts.
    
    • Robot/Program Validation
      
    • Scroll Validation
      
    • Sensor Validation
    
    
  • Board Scan - Replace scrolls, robots, and signs that would try accessing uninitialized data
  
  
• Global Robot

Known problematic worlds:
d00d, visit my website (wiki page)!

Latest commit now shows the broken bytecode in the debug window so I don't need to use hexedit

HELLSTAR (from my Shame Pack) has an extra robot on a board.

Note: i is the offset of the current command, the bytecode displayed includes the byte before it (the length)

CR2-6: We should change the validator so this passes through. The problem is the validator has no way to determine how many params it's actually supposed to be looking for and thus no way to skip that extra 00 after the param.

Prog len: 636   i: 413
Bytecode: 0C 1D 09 66 69 72 65 66 69 72 65 00 00 0C

Ep+Arp World MZX; SZZT7_ori.mzx (George Great Adventure): These were created with mikawo's converter, and validation fails on the global robot. This will affect most games made with that converter, probably (hard to tell since these have different global program sizes, probably because one's a ZZT convert and the other's a SZZT convert). I doubt this matters a whole lot, as the global robot is just the cheating code.

Prog len: 1305   i: 1263
Prog len: 1334    i 1328
Bytecode: 04 30 03 00 01 6E

Slave Pit: Yeah I don't even know what's up here, but it's probably for the best that this was caught.

Prog len: 26992    i: 2
Bytecode: FF FF FF 00 00 00 FF FF FF 88 01 00 00 00 00 00 (continued to the end)

Civil War 2040: Truncated board. This game crashes 2.84.

Board IDs >=128 are converted to CustomBlocks or CustomFloors as of master/cd22a9f. These IDs could result in segfaults in the board editor. There is no solution yet for lines with these IDs in programs yet. Such a thing would require exposing or creating a function to access legacy_rasm.c's command list, but that might be necessary to validate some bytecode lines that aren't "strictly valid" (like the CR2-6 example above).

This makes me glad zzo didn't change the magic for PZX since I wouldn't have thought to fix this otherwise.

GIT d5bde54 has two bugs fixed. curr_world doesn't get overwritten by failed worlds anymore and failed world loads restart the current world's music.

Implemented MZM validation in d0b6d0e. It still needs a lot of work:

• There's no level id or under id validation Fixed in 53db5df
  
• There's no check against the number of robots found on the MZM vs. the number of robots specified in the header (unless there's already something like that in the loading code) They get turned into customblocks if it finds more than the header says there should be in 53db5df
  
• It's probably still possible to spoof MZX worlds as MZMs given the proper dedication (in 2.84 it's idiot-proof but not even close to impossible to cause memory error quits by spoofing the MZM header in the title board name).

Other things that still need to be done:

• clean validation checks out of try_load_world since they all got outsourced to validation.c Done in 97f3b24
  
• Improved program parameter validation Done in 9f7feb8. Program validation was moved to legacy_rasm so it'd have full access to the command list. CR2-6 is passed and games with THING IDs >127 in their code have the affected robots blocked.
  
• Board validation needs to not attempt to load every robot once it's hit the EOF Done in 8f33671. This actually changes the robot count of the board and reallocs the robot pointer lists to reflect it, so the "expected N" message will trigger too.

Case for needing stronger MZM validation: rename this world to a .MZM and try importing it in the current GIT: herpes.mzx

More fun with FFTOWNO.MZX! GIT 8f336711b3.

Program received signal SIGSEGV, Segmentation fault.
0x618e7065 in optimize_null_objects (src_board=0x10eefd8) at src/robot.c:3299
3299              cur_robot = robot_list[d_new_param];
(gdb) bt full
#0  0x618e7065 in optimize_null_objects (src_board=0x10eefd8)
    at src/robot.c:3299
        num_robots = 22
        num_scrolls = 1
        num_sensors = 0
        robot_list = 0x41abc20
        scroll_list = 0x41aa9a0
        sensor_list = 0x0
        optimized_robot_list = 0x41abc20
        optimized_scroll_list = 0x41abc98
        optimized_sensor_list = 0x41abcb8
        robot_id_translation_list = 0x110f8e8
        scroll_id_translation_list = 0x41abcd8
        sensor_id_translation_list = 0x41abcf8
        cur_robot = 0x1108ab8
        cur_scroll = 0x1108bb0
        cur_sensor = 0x19
        board_width = 200
        board_height = 50
        i = 1
        i2 = 1
        x = 39
        y = 0
        offset = 39
        level_id = 0x10f6ff0 ""
        level_param = 0x10fbe40 ""
        d_id = ROBOT
        d_param = 23
        d_new_param = -1414812757
        do_modify = 1
#1  0x61901c2a in load_world (mzx_world=0x40c320, fp=0x77c5fd00,
    file=0x23fb7c "e:\\megazeux\\Invalids\\ffdp\\FFTOWNO.MZX",
    savegame=false, version=521, name=0x23faf3 "Town1", faded=0x23fb38)
    at src/world.c:1471
        i = 1
        num_boards = 4
        gl_rob = 4367
        last_pos = 4235
        charset_mem = 0x10db940 "\377\aj\005main"
        r = 63 '?'
        g = 63 '?'
        b = 63 '?'
        cur_board = 0x10eefd8
        config_file_name = 0x41abe60 ""
        file_name_len = 33
        file_info = {st_dev = 2357352, st_ino = -5856, st_mode = 31888,
          st_nlink = 96, st_uid = 31889, st_gid = -1, st_rdev = 2089877597,
          st_size = 2009252574, st_atime = 10158080, st_mtime = 0,
          st_ctime = 2009252579}
        file_path = 0x41abc20 "<?@"
        current_dir = 0x41abd40 "\037"
        meter_target = 6
        meter_curr = 0
#2  0x6190235d in reload_world (mzx_world=0x40c320,
    file=0x23fb7c "e:\\megazeux\\Invalids\\ffdp\\FFTOWNO.MZX", faded=0x23fb38)
    at src/world.c:1627
        name = "Town1\000~cF ~e Bat~ftle\000ux"
        version = 521
        fp = 0x77c5fd00
#3  0x618d205b in load_world_file (mzx_world=0x40c320,
    name=0x23fb7c "e:\\megazeux\\Invalids\\ffdp\\FFTOWNO.MZX")
    at src/game.c:172
        src_board = 0x23fb04
        fade = 0
#4  0x618d2174 in load_world_selection (mzx_world=0x40c320) at src/game.c:199
        world_name = "e:\\megazeux\\Invalids\\ffdp\\FFTOWNO.MZX", '\000' <repeat
s 222 times>
#5  0x618d668c in title_screen (mzx_world=0x40c320) at src/game.c:2255
        reload_curr_world_in_editor = 1
        fadein = 0
        key = 108
        fade = 1636837756
        file_info = {st_dev = 2358760, st_ino = -15314, st_mode = 30658,
          st_nlink = 0, st_uid = 1, st_gid = 0, st_rdev = 2358808,
          st_size = 1636792077, st_atime = 17611048, st_mtime = 1637534535,
          st_ctime = 41}
        src_board = 0x10db940
        current_dir = 0x9baff8 "e:\\megazeux"
#6  0x00403442 in SDL_main (argc=1, argv=0x9b3f90) at src/main.c:163
        err = 1
        mzx_world = {active = 0, name = "Town1\000n Space Pirates!\000\000",
          version = 521, status_counters_shown = {'\000' <repeats 14 times>,
            '\000' <repeats 14 times>, '\000' <repeats 14 times>,
            '\000' <repeats 14 times>, '\000' <repeats 14 times>,
            '\000' <repeats 14 times>}, keys = '\177' <repeats 16 times>,
          blind_dur = 0, firewalker_dur = 0, freeze_time_dur = 0,
          slow_time_dur = 0, wind_dur = 0, pl_saved_x = {0, 0, 0, 0, 0, 0, 0,
            0}, pl_saved_y = {0, 0, 0, 0, 0, 0, 0, 0}, pl_saved_board = {0,
            0, 0, 0, 0, 0, 0, 0}, saved_pl_color = 27, was_zapped = 0,
          under_player_id = 0, under_player_color = 7,
          under_player_param = 0, mesg_edges = 1, scroll_base_color = 143,
          scroll_corner_color = 135, scroll_pointer_color = 128,
          scroll_title_color = 143, scroll_arrow_color = 142,
          real_mod_playing = "\000V_TITLE.MOD", '\000' <repeats 247 times>,
          edge_color = 0, first_board = 0, endgame_board = 255,
          death_board = 255, endgame_x = 0, endgame_y = 0, game_over_sfx = 1,
          death_x = 0, death_y = 0, starting_lives = 7, lives_limit = 99,
          starting_health = 100, health_limit = 200, enemy_hurt_enemy = 0,
          clear_on_exit = 0, only_from_swap = 0, player_restart_x = 0,
          player_restart_y = 0, num_counters = 0, num_counters_allocated = 0,
          counter_list = 0x0, num_strings = 0, num_strings_allocated = 0,
          string_list = 0x0, num_sprites = 0, num_sprites_allocated = 0,
          sprite_num = 0, sprite_list = 0x0, active_sprites = 0,
          sprite_y_order = 0, collision_count = 0, collision_list = 0x0,
          multiplier = 10000, divider = 10000, c_divisions = 360,
          fread_delimiter = 42, fwrite_delimiter = 42, bi_shoot_status = 1,
          bi_mesg_status = 1, output_file_name = '\000' <repeats 259 times>,
          output_file = 0x0, input_file_name = '\000' <repeats 259 times>,
          input_file = 0x0, input_is_dir = false, input_directory = {d = 0x0,
            entries = 0, pos = 0}, commands = 40, vlayer_size = 0,
          vlayer_width = 0, vlayer_height = 0, vlayer_chars = 0x0,
          vlayer_colors = 0x0, num_boards = 4, num_boards_allocated = 4,
          board_list = 0x10cb7e8, current_board = 0x10db940,
          current_board_id = 0, global_robot = {world_version = 521,
            program_bytecode_length = 2,
            program_bytecode = 0x110f7c8  <incomplete sequence \377>,
            robot_name = '\000' <repeats 14 times>, robot_char = 2 '\002',
            cur_prog_line = 1, pos_within_line = 0, robot_cycle = 1,
            cycle_count = 0, bullet_type = 1 '\001', is_locked = 0 '\000',
            can_lavawalk = 0 '\000', walk_dir = IDLE, last_touch_dir = IDLE,
            last_shot_dir = IDLE, xpos = 37, ypos = 0, status = 0 '\000',
            used = 1 '\001', loop_count = 0, num_labels = 0,
            label_list = 0x0, stack_size = 0, stack_pointer = 0, stack = 0x0,
            local = {0 <repeats 32 times>}}, custom_sfx_on = 0,
          custom_sfx = '\000' <repeats 3449 times>, player_x = 0,
          player_y = 0, target_where = TARGET_NONE, target_board = 0,
          target_x = 0, target_y = 0, target_id = SPACE, target_color = 0,
          target_d_id = SPACE, target_d_color = 0, dead = 0, bomb_type = 1,
          slow_down = 1, key_up_delay = 0, key_down_delay = 0,
          key_right_delay = 0, key_left_delay = 0, first_prefix = 0,
          mid_prefix = 0, last_prefix = 0,
          special_counter_return = FOPEN_NONE, swapped = 0, mzx_speed = 4,
          default_speed = 4, lock_speed = 0, conf = {fullscreen = false,
            resolution_width = 1600, resolution_height = 875,
            window_width = 1280, window_height = 700, allow_resize = false,
            video_output = "glsl", '\000' <repeats 11 times>, force_bpp = 32,
            video_ratio = RATIO_STRETCH,
            gl_filter_method = "nearest\000\000\000\000\000\000\000\000",
            gl_vsync = 0, output_frequency = 96000, buffer_size = 512,
            oversampling_on = 1, resample_mode = 2,
            modplug_resample_mode = 3, music_volume = 8, sam_volume = 8,
            pc_speaker_volume = 8, music_on = 1, pc_speaker_on = 1,
            startup_path = '\000' <repeats 255 times>,
            startup_file = "\000averns.mzx", '\000' <repeats 244 times>,
            default_save_name = "saved.sav", '\000' <repeats 246 times>,
            mzx_speed = 4, disassemble_extras = 0, disassemble_base = 10,
            startup_editor = 0, mask_midchars = 1, system_mouse = true,
            network_enabled = false, socks_host = '\000' <repeats 255 times>,
            socks_port = 1080, update_host_count = 2,
            update_hosts = 0x9b70f0,
            update_branch_pin = "Stable", '\000' <repeats 249 times>},
          editor_conf = {editor_space_toggles = 0, bedit_hhelp = 1,
            editor_enter_splits = true,
            color_codes = "\v\n\n\016\377\003\v\002\016\000\017\v\a\017\001\002\
003", '\000' <repeats 14 times>, color_coding_on = 1,
            default_invalid_status = 1, redit_hhelp = 1, backup_count = 0,
            backup_interval = 30,
            backup_name = "backups/arch-", '\000' <repeats 242 times>,
            backup_ext = ".mzx", '\000' <repeats 251 times>,
            default_macros = {"char ", '\000' <repeats 58 times>,
              "color ", '\000' <repeats 57 times>,
              "goto ", '\000' <repeats 58 times>,
              "send ", '\000' <repeats 58 times>,
              ": playershot^", '\000' <repeats 50 times>},
            num_extended_macros = 0, num_macros_allocated = 0,
            extended_macros = 0x0}, editing = false, help_file = 0x77c5fce0,
          update_done = 0x41a74a8 "", update_done_size = 10000}
#7  0x00404312 in console_main (argc=1, argv=0x9b3f90)
    at ./src/main/win32/SDL_win32_main.c:315
        n = <optimized out>
        bufp = <optimized out>
        appname = <optimized out>
        status = <optimized out>
#8  0x004043c8 in WinMain@16 (hInst=0x400000, hPrev=0x0,
    szCmdLine=0x251ef9 "", sw=10) at ./src/main/win32/SDL_win32_main.c:398
        handle = <optimized out>
        argv = 0x9b3f90
        argc = 1
        cmdline = 0x9b2688 "e:\\megazeux\\megazeux.exe"
        env_str = <optimized out>
        bufp = <optimized out>
        nLen = <optimized out>
#9  0x00403aba in main (argc=-858993469, argv=0x106acccc,
    __p__environ=0xc1208868) at ../mingw/main.c:73
        szCmd = 0x251ef9 ""
        startinfo = {cb = 68, lpReserved = 0x253800 "",
          lpDesktop = 0x253820 "WinSta0\\Default",
          lpTitle = 0x253848 "e:\\megazeux\\megazeux.exe", dwX = 0, dwY = 0,
          dwXSize = 0, dwYSize = 0, dwXCountChars = 0, dwYCountChars = 0,
          dwFillAttribute = 0, dwFlags = 0, wShowWindow = 0, cbReserved2 = 0,
          lpReserved2 = 0x0, hStdInput = 0xffffffff, hStdOutput = 0xffffffff,
          hStdError = 0xffffffff}
        nRet = <optimized out>

Yeah, that's pretty awful. I changed it back and just have it suppress the extra error messages now as of GIT a4c54c7.

Lachesis, on 15 June 2012 - 12:21 PM, said:

Well, the best I could handle this is to wipe out any IDs >= SENSOR (besides robots which are handled separately), which has prevented crashes entirely as far as I can tell. The invalid types <122 seem harmless enough in the editor that I won't worry too much about them.

Program validation is causing problems (especially with that "MZM" file above) due to its behavior of simply memsetting the extra program to 0. It seems like a LOT of MZX relies on program_length far more than that silly mandatory condition that the program ends when the length byte is 0, so I need to let the program validator return an updated program size.

GIT ae08fc2. Big surprise, letting validate_legacy_program return an updated program size and reallocing the program fixed All The Crashes!

Updating status to: Implemented
Issue fixed in: GIT (debytecode)

Overall, this seems to have been successfully implemented, aside from the gaping hole that was LOAD_BC. Please continue to add failed validation worlds to the wiki as you see them.

Issue fixed in: GIT

Moving to: Archived Requests