[ajs]

paul is correct that you can associate MZX files. For the moment you have to use /path/to/megazeux.exe startup_file=file.mzx. In the soon-to-be-released 2.82b this will no longer be necessary.

--ajs.

[Exophase]

ajs, on Nov 20 2008, 07:25 PM, said:

paul is correct that you can associate MZX files. For the moment you have to use /path/to/megazeux.exe startup_file=file.mzx. In the soon-to-be-released 2.82b this will no longer be necessary.

--ajs.

You mean MZX 2.83, since Mr_Alert added new counters to the SVN.

:B

[Terryn]

If you're referring to recognising marker points in WAVs and OGGs, that's intrinsic to the actual WAV and OGG files themselves and introduce no new counters in MZX. Otherwise, I don't know what you're talking about =/

[ajs]

This change poses a problem though because it allows games using the loop feature (passively) to be played in a version of MZX that doesn't have it. It's not as bad as other similar compatibility issues we've had, but I'll probably just disable the change temporarily for 2.82b. We've got some other changes that have been earmarked as requiring a world version bump, and I'll make them after this maintenance release.

--ajs.

[Scarmiglione Malebolge]

a suggestion for implementing in next versions of MZX:
MIDI support!
just my suggestion.

[Terryn]

MIDI's a perennial request, but there are some issues with it.

1) MIDI, with its reliance on hardware, sounds different on different machines.
2) Unifying how MIDI sounds will probably rely on a softsynth like TiMidity and introduce bloat to the program.
3) We have a panoply of module formats and WAV/OGG supported already.

MIDI is nice for its tiny filesize, but even the biggest, most music-laden MZX games don't even top 75MB yet.

[Frobozz]

Besides you can either convert MIDI to MOD (using ModPlug Tracker) or you could get a MIDI program and soundfonts and make an WAV/OGG file. Also Timidity is going to need a soundfont anyways. And any decent soundfont will probably be bigger than an OGG.

[Wervyn]

Here's a minor thing that's come up enough to bug me, given how much I play with interesting applications for sprites: can we change the cx and cy values for sprites to be signed ints instead of signed bytes? It should be a five second change, and cost at most 1.5K to the save game size. As far as I know, no game relies on the fact that they can only hold values between -128 and 127, but I've definitely had to work around that fact a few times, and just discovered another bug with an old engine today when trying to do a collision offset into a large sprite layer. The cy value in particular is also currently the only good way to dynamically order large sprite layers, when combined with the y-order option.
In addition, it may be worth looking at the sprite width and height (and cwidth and cheight, to correspond), and whether there's a reason to keep them under 255 or if those could be 32- or 16-bit as well. It may be there's some issue with the display code that would make attempting to keep track of something larger than that (particularly for collision types 1 and 2 where spaces don't count) expensive. But it also seems like it should be possible to optimize that to an extent, especially in the case of just display, if it hasn't been already.

"Why is it always sprites with you?"

[djtiesto]

Quick suggestion I thought of, I apologize if people have talked about this in the past...

You have something like:

goto #do
goto #do2
* "~fhey!"

:Do
*will do first
goto #return

:do2
*blah
goto #return


well, how about have something like...

send bot2 "#do"

which will prevent code following that line from executing, until the second bot executes the "do" label... make it even more like OOP, and easier to organize your program in different 'bots, instead of putting everything in one big bot, or breaking things up and then worrying about timing issues/race conditions.

[Terryn]

Wervyn, on Dec 17 2008, 10:29 PM, said:

As far as I know, no game relies on the fact that they can only hold values between -128 and 127, but I've definitely had to work around that fact a few times, and just discovered another bug with an old engine today when trying to do a collision offset into a large sprite layer.

I haven't taken a good look at it, but I do remember the changelog addressing a snag like that:

MZX 2.80b changelog said:

+ Wrapped around sprite values so LogiCow's bad code would work (HTMCIAB).

[Risu2112]

Request a config option to fun MZX at say current desktop resolution, but then actually display it at the largest multiple of 640x350 while in full screen mode (probably enabled by default I would think) For use on LCD displays that _Cannot_ support full screen resolutions in any multiple of 640x350 (most LCD's)

Problem is I have a 1920x1200 display, and I cannot set mzx to run at 640x350, 1280x700 or 1920x1050 because these are not supported resolutions. :<

This would effectively let me run mzx at 1920x1200 with 75 pixel letterboxes on the top and bottom so the scaling is the same for X and Y

thankyouthankyouthankyou

[Aussie Evil]

Would like an option to disable the default fade transitions in the editor when switching to test mode.

[ajs]

Ken, I've implemented your aspect ratio preservation thing, when I release 2.82b you need to set fullscreen_resolution to 1920,1200 and video_ratio to "modern".

--ajs.

[Terryn]

Risu desires the ability to load compressed MZMs natively in MZX. He says that the ones developed for his pet project are around 1.1GB in total and that they compress to about 15MB. This is obviously an extreme example, but still.

[Risu2112]

Hey I've got hard drive space to burn this is a request to spare every one else my MZM wrath.
But yes, thanks for the request terryn, seriously that's how many mzm's I'm using ;_;

[mzxgiant]

I'll tack on here that if we're doing compressible files, it wouldn't be too much trouble to throw a command to ZLib while saving/loading those, I don't think. Especially since we're already integrating it into the builds for other purposes...

[Risu2112]

It should go without saying, if we allow mzx to read compressed mzm's directly we should not prevent people from using uncompressed mzm's in the event they are working on something where the extra cpu cycles would start causing problems, by default mzx should not put anything under compression unless it's specifically asked to do so.

Also also, I'm talking about putting many mzm's into one compressed file, not one each ^.^

This post has been edited by Risu2112: 27 December 2008 - 12:59 AM

[ajs]

There might have been a good reason for this at the time, but MZMs really could have been more similar to boards in that they are saved as planes which are RLE compressed, that probably would have mitigated this issue. As it is, they are saved interleaved, which I'm sure is handier for the programmer, but is much less likely to benefit from RLE. Maybe time for an MZM3?

That said, what on earth do you need so much MZM data for?

--ajs.

[Exophase]

May as well use something better than RLE anyway. And change the board format to no longer tie in the RLE stuff. I'd much rather not have compression code sitting in the middle of the board loader. We could go ahead and unify boards and MZMs, I guess, and just let all loaded files be compressed. Entire world files, MZMs, even things like mods and charsets. Also, allowing them to be pulled from archives. Some unified way to do this, even by filename, could be good. IE:

load charset "charset_a.chr"
load charset "charset_a.chr.bz2"
load charset "charset_a#chars.tar"
load charset "charset_a#chars.bz2.tar"

(tars compressed in the other direction not allowed for obvious reasons, although I suppose we could cache decompressions)

Put this in a unified layer and it'll just apply to everything (with the file loaders modified to accept variants with compression extensions at the end)

[ajs]

As MZXGiant pointed out, we'd probably be best off re-using zlib (gzip) for this, even if it is slightly inferior to bzip2, we already pull it in for libpng and (soon to be) the network layer. As for adding compression globally, I'm not too keen about having to explicitly refer to compressed content.

Perhaps it would be better if the compression mode could be inferred from the file name, for example if somebody loads "popcorn.mod" and there's only a "popcorn.mod.gz", it'll decompress and load that one instead. I'm not sure what the point of TAR is if we're not going to support gzipped TAR as opposed to tarred GZIP? (As you mentioned, I'm definitely not in support of "tar.gz" due to the awkwardness of finding/seeking and write behaviour).

As for procedurally generating compressed files, this might be better as a flag (i.e. another counter) that is set before you start the file I/O? The advantage of all this is that it doesn't encode details of the compression being used into the interface, and it allows us to replace the file I/O with sandboxing, archives, and alternative compression methods later..

--ajs.

[Exophase]

ajs, on Dec 27 2008, 10:42 AM, said:

As MZXGiant pointed out, we'd probably be best off re-using zlib (gzip) for this, even if it is slightly inferior to bzip2, we already pull it in for libpng and (soon to be) the network layer. As for adding compression globally, I'm not too keen about having to explicitly refer to compressed content.

Perhaps it would be better if the compression mode could be inferred from the file name, for example if somebody loads "popcorn.mod" and there's only a "popcorn.mod.gz", it'll decompress and load that one instead. I'm not sure what the point of TAR is if we're not going to support gzipped TAR as opposed to tarred GZIP? (As you mentioned, I'm definitely not in support of "tar.gz" due to the awkwardness of finding/seeking and write behaviour).

As for procedurally generating compressed files, this might be better as a flag (i.e. another counter) that is set before you start the file I/O? The advantage of all this is that it doesn't encode details of the compression being used into the interface, and it allows us to replace the file I/O with sandboxing, archives, and alternative compression methods later..

--ajs.

I don't see any real problem with having to refer to the compressed files explicitly - that is their file name, right? It doesn't really hurt to have it implicitly searched for too, though. The point of tar (sans compression) is two-fold - organization/less file clutter, and more importantly, avoids file system fragmentation. A charset might only be a few bytes but it uses an entire filesystem segment, which could be several KB. When you have thousands of these this amounts to a ton of waste. Oh yeah, it'd also be good to default to a tar search path to fall back on if something's not found, if the .mzx was opened from a tar. This would make it easy to distribute single files; although people would probably be compressing them again, it'd really help to keep files together.

I don't think there's any serious demand for Robotic generated files to be compressed. Let's worry about that later.

By the way, there could be an option in the save file dialog to compress any files (tick box) and I guess some means of adding them to an existing TAR. Maybe the tars wouldn't be .gz in extension, instead the accessing something like:

blah.gz#archive.tar

How large is bz2 anyway? In case we decide to use both. Bear in mind, the size sensitive platforms (like Nintendo DS, NOT PCs) may not be bringing in PNG or network stuff anyway, which are both not core mandatory features. This, on the other hand, would be.

[Terryn]

Sorry to distract from the main discussion, but I have another request. I'd like wildcards for the purpose of string comparison (and these symbols are arbitrary and only in to clarify examples):

-A wildcard * for any/all characters before/after given characters. EX: *enclosed*
-A wildcard ^ for any alphanumeric characters (I.E. the char numbers typically taken up by those characters). EX: Captain_^^^^
-A wildcard % for any character. EX: Worker_No._10%

Wildcards for only numbers, only letters and maybe only upper/lowercase letters could be nice.

[ajs]

AFAIK zlib and bzip2 are around the same size; they would add 30-50K each to the executable. I'm not too concerned about space but I don't think it's probably worth the configuration burden adding multiple archive formats for the sake of some % at this time. Another pro-zlib argument is its superior speed and significantly reduced memory requirement. Standardising on zlib is also useful because the DEFLATE algorithm in zlib is the same as the one used in most ZIP files, which allows us to add automatic game extraction and other features later on (which would make zlib mandatory too). We'll be using this in the archive browser, for example, which may well make it to portable platforms in the future.

As for your TAR idea, I appreciate the attempt to reduce fragmentation (which I must add isn't really a big deal for modern filesystems) but this comes at a cost. TAR itself still has a block/segment size, so it's not size-optimal (default is 10KB, see tar -b for more information) and seeking in a tar with many thousands of files is still going to be more expensive than making an operating system stat() call on most platforms (which may use btrees or other FS optimisations to locate one file in many thousands). I've never really been convinced by the "file clutter" argument; you can use a sub-directory if you care about this.

A more compelling argument for TAR, in my view, is that on some embedded platforms there is a wide gulf between memory performance and file storage speed. For example, Mr_Alert pointed out that SD is painfully slow on the Wii; worlds can take many seconds to load. This is a constant problem during runtime, too, for example loading large mods or vorbis files. Having things in a TAR would, for modern games, allow us to pre-cache things more easily if we have the memory available (which might be a hard thing to determine, I guess).

Though, back to my original point. What ADVANTAGES do you see to having the user explicitly note the compression extension? Is there any inherent disadvantage to using an implicit basename "search" procedure instead?

By the way, I would be very interested in working on this in a more general way, for example playing a game from within a ZIP and extracting resources only if they are procedurally edited/created. ZIP can be efficiently appended too, since it has an index, so this might be a superior solution in the short term (though inferior in several other ways, such as compression ratio).

Basically, I see this discussion as implicit VFS (minimal changes to MZX, no changes to robot programs, pretty good compression/archive options) to an explicit VFS (significant changes required to the editor, changes to robot programs that aren't backwards compatible, and superior compression/archive options). I don't buy the "explicit" option yet, because I don't understand what the benefits are.

--ajs.

[Wervyn]

@Terryn: If you're going to ask for variable string comparison you might as well go all the way and request full regex support. Sure it might not be as strictly intuitive as a more vanilla wildcard match, but it's a hell of a lot more powerful. You could probably hack substitution and translation into the set command, too. "set "$string" to "s/^(.*)foo(.*)$/\2bar\1/"" or "set "$toupper" to "tr/[a-z]/[A-Z]/"".

This is probably a terrible idea.

@Everyone else: Support for file compression sounds like a good idea. Though I can't help but think that if you have a file 1.1 GB large that compresses nearly 100 fold, you're probably doing something wrong. But on the other hand MZMs are a pain right now because they don't take advantage of any optimization for size.

"Didn't need that kidney anyway."

[Exophase]

Regular expressions are cool, but we'd probably need to bring in a library to do it, which is another amount of bloat.

Settling for zlib is fine by me - I don't think it's a huge deal.

ajs; I was referring to "internal fragmentation", which is still very much a problem, at least on NTFS (and on FAT which is still used on some of the platforms we support.. and will probably continue to be common on embedded media). For instance, a 56 byte char set here has a "size on disk" of 4KB. Three of them take up 12KB. This is a huge amount of waste, and hurts both physical storage and disk caching. Thousands of char sets later, which really is something that is realistic for an MZX game now, you're wasting several MB. TAR does have internal segmentation, but at 512 bytes it's better than it could be at least. And yeah, it'd be easier to cache. Having to create sub-directories is an extra step and it'd be nice not to have to worry about it, both in having to create them and having to reference them when loading your files. The seek time can be mitigated by maintaining our own file tables for them.

Like you said, I'd like to see an implicit VFS in MZX; ie, I think we should abstract file objects and operations around our own structs and functions (if they aren't already, I haven't looked yet). We can treat TAR files the same as directories, and even list them in the right hand column. Saving new things to TAR could be an extra burden, but MZX doesn't really need to overwrite different files a lot; one constantly modified one would bubble up to the top of the TAR. I don't really know what you meant by "explicit." I don't see anything wrong with allowing the search path for loading, but you should still be allowed to do it explicitly, in case you have a compressed and uncompressed file for example (I don't know why you would).

[ajs]

The reason why I wouldn't recommend changing it in that way is that it makes it more difficult to change the design later. Right now we could solve, say, Risu's problem, and probably the whole fragmentation issue by just providing implicit ZIP VFS access. It's quick, dirty, and not particularly space optimal, but we have the libraries required linked in today (zlib and minizip) and it requires far fewer modifications.

What it would mean is that people working on games on their regular PC would still have the "thousands of charsets" or "1.1GB of MZMs" but as soon as they zipped the game up, which we could of course provide in-game helpers for, the ZIP could be used to play the game on other computers without ever extracting it. That way, there's no user-level difference between accessing a file in a VFS container or on a literal filesystem, and that means on systems where the VFS is costly to use, you can omit it.

Admittedly I'm probably not thinking of all the cool things this "explicit" (perhaps not the best term) VFS provides, but I think if we tackled this use case first, it would be easier to see what additional design was required. Basically, I just don't like the load charset "blah.tar:asset" thing very much. This project has moved pretty much to the top of my list for 2.83, I hope we can collaborate on the finer points before that release.

--ajs.

[Exophase]

Do we want a utility that explicitly zips things up when we could just build it into the file load/save dialog box? With "explicit VFS" it probably makes as much sense to allow TARs or ZIPs or whatever to be treated as directories. I just don't see a good reason why we can't allow it both explicitly and also an implicit environmental archive that's used as a last search. And it doesn't require changing several things, just the file loading/saving infrastructure, same as what you'd need to do regardless.

So loading a file IE:

load charset "blah.zip/whatever.chr"

The file path is parsed by the VFS layer which returns a file object instance that it uses. If, when searching for a file, it doesn't find one then it would search an archive if the game was opened under one. ZIP or TAR, doesn't really matter that much which to me.

This also means that to save something within an archive you open it as a directory and do it there. It also means you can navigate the achives in MZX, and perform the other file dialog options (rename, delete, etc).

[ajs]

The only fundamental difference between our proposals, as I see it (and please correct me if I'm wrong), is that you see something like:

Game/
  World.MZX
  Charsets.ZIP/ <-- VFS directory
	A..
	B..
  Mzms.ZIP/ <-- VFS directory
	C..
	D..

And I see:

Game/ <-- VFS directory
  World.MZX
  Charsets/
	A..
	B..
  Mzms/
	C..
	D..

The method for creating the initial world file or ZIP with my strategy is debatable. Initially, I'd propose having no additional "helper" in MZX for doing this, since it's not really necessary to save the world from fragmentation and vast sizes on my own local development PC. It's much more important to do that on other people's machines/consoles/etc. However, if I did for some reason care about this, it would be possible to create an "empty" ZIP that contained only the world and add some UI for poking files into it, etc. It's a step towards "saving a world" becoming "creating a project" instead.

The advantage of my method is that file addressing is entirely identical to the traditional, non-VFS strategy. There's no hacky ZIP directory traversal. It's definitely less flexible though, because the end user doesn't have much of a choice what is and isn't compressed. But in my view, that's a good thing, in case we decide to change compression format, or people want to play games on a platform without zlib support or where a VFS is simply too much overhead.

Another advantage is that all of the archive games look pretty much exactly like my VFS already. They don't need to be altered to benefit from compression (arguably with your strategy they wouldn't either, since we'd likely support both VFS strategies if we supported your "explicit" method.).

I'm starting to see some of the advantages of your approach (mostly control) but I think both approaches largely solve the same problem, and it is this that I (crudely) consider to be the "implicit" versus "explicit" VFS design.

(MZX 3.0?? Project and Resource oriented;

Game/ <-- VFS directory
  README.txt
  world.xml
  Boards/
	Blah.raw
  Screenshot.png
  Thumb.png
  Charsets/
	A..
	B..
  Mzms/
	C..
	D..

) btw Exo, this is all sounding eerily familiar..

--ajs.

[KirbySSB2]

make it so if i delete an object that is above something (ie a robot standing on a customfloor) that the customfloor below it won't delete? so I would have to hit delete twice to remove it this would be extremely helpful for me when moving robots that are on certain type of floor.

[LogiCow]

SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...
SAVING...

I hope the message will be made optional in the next version. And set the rate lower, too.