[Nightwatch]

I thought I'd share some interesting (to me, anyway) design problems with the MZX Lua library and get some feedback. Here's one that's been giving me grief.

I'm trying to eliminate the special cases in the core basis library. All of the Robotic commands should be available, but ideally as many of them as possible should be implemented in terms of the core commands in Lua rather than C, where this makes sense in terms of performance.

I ran into a snag with directions. Intuitively, you think of a direction as an object that combines with a point to produce another point. So, there's a function (and in Lua it's mzx.rel()) with the type "point * direction -> point". For instance, mzx.rel({10, 10}, 'east'} produces {11, 10}. Actually, in MegaZeux there's a hidden robot argument, for the directions like "flow" and "seek", so the type of the function is really "point * robot * direction -> point".

But in MZX there's one exception: BENEATH. BENEATH of (10, 10) doesn't refer to a point but to the hidden "underneath" layer that you can't really access in Robotic, with a few exceptions. The need to handle BENEATH correctly is revealed with the difference between "if c03 floor to beneath of player" and "if c03 floor at &playerx& &playery&". This is code that you find all the time in MZX games. And without BENEATH, there's no difference between "become litbomb" and "put litbomb to BENEATH" (a.k.a. LAYBOMB).

So you have to handle BENEATH. But this makes the idea of a direction as a function completely meaningless: its interpretation now depends on the command it's in. For example, "GO BENEATH 1" is not legal.

The idea I had to solve this is to make points 3-dimensional in MZX. Don't worry, this is normally completely invisible, because the Z coordinate defaults to 0 (the top layer) if unspecified. The only time it would matter is in the primitive operations, mzx.fetch() (returns the color/thing/param at the given location) and mzx.put() (places a color/thing/param at a given location).

With this scheme, there's no need to make "IF color/thing/param direction PLAYER" and "PUT object DIRECTION" separate commands in C. They'll simply be implemented as mzx.match(mzx.rel('player', dir), color/thing/param) and mzx.put(mzx.rel('self', direction), object), and the mzx.rel() command does the job of setting the Z coordinate properly. The only weird thing is that it makes points 3-dimensional at least internally, which is a bit strange.

Thoughts/ideas?

[mzxgiant]

I'm wondering how this was done originally without the Lua aspect. I know that at least several versions ago, there were scenarios where the layer beneath a robot or player wouldn't render properly and you'd end up with a gaping hole in the "floor" sometimes. (I don't remember how I reproduced this, and it may have been deliberately or inadvertently fixed since)... I feel like making 3D points may be a bad idea if only because it goes back to what ajs was saying about making Lua an optional library of sorts to MZX... setting the entire vector system to a different standard may be too intrusive on the existing base.

Of course, I've never looked at the source, so if I'm talking out my ass here, I'm terribly sorry

[Nightwatch]

mzxgiant, on May 12 2008, 09:27 PM, said:

I'm wondering how this was done originally without the Lua aspect.

Every Robotic command that takes a direction is coded differently. Usually BENEATH is hard-coded to do something special.

Quote

I feel like making 3D points may be a bad idea if only because it goes back to what ajs was saying about making Lua an optional library of sorts to MZX... setting the entire vector system to a different standard may be too intrusive on the existing base.

Actually, MZX doesn't even have the concept of a vector or a point internally. It just saves array offsets (and X and Y coordinates for some functions, but they just get multiplied to produce an array offset at some point). All the conversions between points and array offsets is done within the language-specific front ends (Robotic and Lua), so it wouldn't affect the existing code.

[ajs]

This might be too much to ask, but I'd recommend enhancing the concept of a point internally to 3D (as you've suggested), adding some helpers for deciding when to flatten/go under, and fixing the existing robotic commands to use it. Ideally, this patch would be separate from the rest of your Lua work so I could see how it was going to affect the internals; if it looks okay I'll merge it.

MZX has an explicit approach to layers, and as you pointed out BENEATH is quite an exception to this. Pretty much the first thing people might be inclined to change in a new GCS was to make layering implicit, which requires such a 3D point anyway, so I don't think it's conceptually TOO bad an idea. There's all sorts of ordering hacks with sprites, for example.

--ajs.

[Nightwatch]

ajs, on May 12 2008, 11:44 PM, said:

This might be too much to ask, but I'd recommend enhancing the concept of a point internally to 3D (as you've suggested), adding some helpers for deciding when to flatten/go under, and fixing the existing robotic commands to use it. Ideally, this patch would be separate from the rest of your Lua work so I could see how it was going to affect the internals; if it looks okay I'll merge it.

MZX has an explicit approach to layers, and as you pointed out BENEATH is quite an exception to this. Pretty much the first thing people might be inclined to change in a new GCS was to make layering implicit, which requires such a 3D point anyway, so I don't think it's conceptually TOO bad an idea. There's all sorts of ordering hacks with sprites, for example.

--ajs.

I do plan on introducing the primitive fetch() and put() operations in robo_ops.c that operate on (3-D) Point and CTP structures (color/thing/param really needs to be a structure at this point). I'll try to optimize them heavily and convert as much code as possible to use them instead of digging around inside the board arrays directly. This should add orthogonality to Robotic as well as Lua.

[Frobozz]

ajs said:

Lua can't be considerably slower, and this should be benchmarked. At least, it should not be cripplingly slower

I was thinking it might actually be faster in a few situations. If I'm not mistaken, aren't expressions in Robotic stored and processed as strings (iow not converted to bytecode before processing)? If so then I would think expressions in Lua would be faster than Robotic and possibly entire scripts would be faster if they are expression heavy (think lancer's scripts).

This post has been edited by Frobozz: 13 May 2008 - 03:10 AM

[Nightwatch]

Ok, this is actually far more complex than I imagined.

Assuming that (x1, y1) contains a SpittingTiger, there is an important difference between "COPY BLOCK at x1 y1 for 1 1 to x2 y2" and "PUT c?? SpittingTiger p?? at x2 y2" The difference is that COPY BLOCK copies both the under and over layers, while PUT places an object, pushing down whatever is underneath (unless it's placing something that can be walked on, like a floor; then it just squashes what was there first).

Also, there are three layers, not two. The third can only ever be filled underneath the player. Try going onto a floor, laying a bomb, and moving away, and then make a robot on the a floor that uses LAYBOMB and moves away. The floor stays if you lay a bomb, but it disappears when the robot lays the bomb! This is because layer 3 only ever exists under the player, so the robot squashes the floor on layer 2 when it lays the bomb.

I think the behavior of COPY BLOCK should stay the same: people expect it to copy all layers, I think, and it's much faster to do so because when you PUT, you have to treat the layers as a stack, and the rules for layering are complex given what I just mentioned. Also, in these days of being able to copy freely among the board, overlay, and vlayer, it's time to give the overlay and vlayer first-class status in a sense; you should be able to PUT on the vlayer, for instance, and you should be able to WRITE on the board. So I'm going to design the new API with this orthogonality in mind. Hopefully this can eliminate a lot of confusion and bugs.

[Nightwatch]

Edit: Never mind, this is wrong

This post has been edited by Nightwatch: 13 May 2008 - 08:33 AM

[Exophase]

Ah, now Nightwatch comes upon the lovely problems of MZX's layering designs.. yay! IMO you're best off going with "board" layers and "overlay" layers, where the former have IDs and under parts and the latter just have graphics. That's the format MZMs follow too. That is, if you wanted to expand things into multiple layers. Trying to convert the over/under parts into distinct layers isn't going to work that well.

[Nightwatch]

Exophase, on May 13 2008, 06:38 PM, said:

Ah, now Nightwatch comes upon the lovely problems of MZX's layering designs.. yay! IMO you're best off going with "board" layers and "overlay" layers, where the former have IDs and under parts and the latter just have graphics. That's the format MZMs follow too. That is, if you wanted to expand things into multiple layers. Trying to convert the over/under parts into distinct layers isn't going to work that well.

I'm treating over/under/under player as three different spaces only as far as put() and fetch() are concerned. The important thing to realize about put() is that it won't always do what you request: if you try to place an object on layer 3 that isn't underneath the player, it won't do it, and if you try to place an "under" thing on top of another "under" thing, it'll do it, but the second under thing will be squashed.

Basically, the idea is to make the Z axis completely invisible to robots and higher-level new MZX code, except when BENEATH is used. There has to be some way to refer to the thing under something else, or BENEATH won't work.

I'm writing a more efficient copyblock() (which subsumes COPY BLOCK and COPY OVERLAY BLOCK into one unified command) right now. Instead of copying to an external buffer and back again, it uses optimized block copy code. copyblock() doesn't deal with over/under layers separately: it copies everything, just as you suggested. It also copies forwards or backwards as necessary to prevent blocks from overwriting themselves.

Interestingly, copying blocks on the board is going to be significantly slower than copying blocks on the overlay, because of all the integrity constraints that have to be enforced on the board. Both should be faster than the current MZX implementation, though, because the board buffer will be bypassed.

[ajs]

Since I just finished merging Kev's Nintendo DS port (which involved several changes to the codebase that conflicted with your core changes) I updated the port to the latest SVN revision (currently, 594). The files have been updated.

(I'm sure this code is obsolete from your perspective but for other people it might be handy.)

--ajs.

[Exophase]

Nightwatch, on May 13 2008, 05:07 PM, said:

I'm writing a more efficient copyblock() (which subsumes COPY BLOCK and COPY OVERLAY BLOCK into one unified command) right now. Instead of copying to an external buffer and back again, it uses optimized block copy code. copyblock() doesn't deal with over/under layers separately: it copies everything, just as you suggested. It also copies forwards or backwards as necessary to prevent blocks from overwriting themselves.

Good man, I was meaning to do that ages ago but forgot to (if I was a big macro freak back then like I am now I probably would have, but I suppose I would have done a lot of things differently). Although, memmove might do it for you.

[Nightwatch]

Exophase, on May 15 2008, 03:00 PM, said:

Good man, I was meaning to do that ages ago but forgot to (if I was a big macro freak back then like I am now I probably would have, but I suppose I would have done a lot of things differently). Although, memmove might do it for you.

I've written my new COPY BLOCK routine, as well as a new set of board primitives. memmove() is fine for the overlay and the vlayer, but the board is horrendously complicated, because of the necessity of duplicating, removing, and moving robots/sensors/scrolls as necessary. There's no way to get around looping over the board tile by tile. I'm using a complex system of reference counting to handle robots/sensors/scroll duplication or deletion logic, and they seem to be working now. (The Triple Parallax Starfield in Xenogenesis is a good way to test this: it copies and deletes robots like crazy in COPY BLOCK commands.)

Copying to or from the board is much slower than copying to and from the overlay or vlayer, in case somebody is writing a game and is concerned about speed. Copying to and from the board is exceptionally slow.

By the way, the MZX COPY BLOCK command has a strange quirk in that if you specify a width or height of 0, it's treated as 1 instead. Booshkies demo 3 relies on this.

Unfortunately, I'm running into a nasty bug which is making some of the right walls look wrong in the 3D maze in Bernard the Bard. BtB uses width and height values that overflow the bounds of the board and expects the copy to go through anyway. My COPY BLOCK routine checks for this and clips as necessary, but there's still an issue with some right hand walls that may or may not be related.

To make a long story short, Robotic accepts some crazy-ass malformed input, and there are games out there that rely on every quirk of the system. This is why Lua will be stricter than Robotic was. I'm not convinced that the Bernard the Bard problem isn't just a wacky bug on my part though.

Here's the code for the new set of board primitives, for anyone who might want to take a look. It won't compile as is, though, because it depends on the hash table functions which I wrote and I'll include in the next release.
http://pastie.caboo.se/198075

This post has been edited by Nightwatch: 16 May 2008 - 06:18 AM

[ajs]

It's a shame we don't have a regression test for these features, but I guess you'll nail all the bugs eventually.

BTW, it would be good if you could drop the mzx_ prefix on your function names/macros/structures. Nothing else in MZX uses this kind of namespacing, and it's not really necessary.

It's interesting that you've introduced hashtable primitives; I received a patch from inmate the other day that switched counters to be hashed, so when you post your code I'll try to consolidate that.

--ajs.

[mzxgiant]

Just putting in a quick thought; if we're integrating some of these strict input validation routines, would it be worthwhile to add a toggled Quirks/Strict mode a la W3C standards? I mean just to avoid breaking some of the historical games (BtB, for example).

[ajs]

Nightwatch won't alter the behaviour when the game uses Robotic, just when it uses Lua. This will preserve backwards compatibility. This has to do with the way the commands are interpreted by the various backends.

It should be obvious that no changes to MZX that break backwards compatibility are acceptable in any way. Even if, historically, a few have slipped through (mostly Akwende's fault).

--ajs.

[Wervyn]

I believe a general scheme for that is already in place, which keys certain features and additions to the MZX version advertised by the game. So for example, Doom Keep (I think, this is anecdotal so I could be wrong about the specifics) used a counter called "riddles", which broke when the rid* counters were introduced. The use of version keys fixed this backwards compatibility bug.

"I miss it already."

[ajs]

Slightly OT for the thread, but I thought I'd clarify..

Currently these backwards compatibility checks only apply to counters (not in Nightwatch's case where he cares about the parameters to commands), and in 2.82 (which will be released shortly) I've back-versioned every counter, just to be doubly sure of no conflicts. Unfortunately, there are some places where backwards compatibility was broken:

• Some early 2.6 versions of MegaZeux were released without bumping the world or SAV magics. This is a severe bug and basically prevents us from being fully backwards compatible with 2.51s3.1/3.2, which is a real shame;
  
• Sprites and strings changed semantics slightly as they evolved;
  
• The ABS_VALUE, R_PLAYERDIST, SQRT_VALUE, WRAP, VALUE were removed in 2.68 and no compatibility layer was written. Writing one is retroactively possible, but I'm not aware of any broken games, so it hasn't been done;
  
• FREAD_PAGE, FWRITE_PAGE were removed in 2.80 and no compatibility layer was written. These don't really make sense any more and probably weren't used by any released game anyway. Writing a compatibility layer would again be possible, but hasn't been attempted.

Unfortunately this list only protects counters from name collisions, if commands were to change, or the semantics of a counter was to change, this wouldn't be protected. Additionally, there's been plenty of syntactic sugar with strings and expressions which could break backwards compatibility (expressions are disabled before a certain version, however, so this is mostly theoretical).

However, in some cases such as fread/fwrite counter (which has been upgraded in 2.82 to support the full 32bit signed limit), an explicit version check is used to retain backwards compatibility.

Summary of the above: we're not bad at backwards compatibility, and the fact that so few games broke in Exo's tremendously complex port is a testament to this. 2.82 will eliminate more of the "theoretical" issues and fixes some real game-breaking bugs too. As long as Nightwatch continues to diligently examine the behaviour of commands in Robotic, we should be able to retain backwards compatibility without "modes", even if large routines get re-written, or dun dun dun a new language gets added.

--ajs.

[zzo38]

What I would like to see is not Lua support, what I would like to see is Forth instead of Lua.

[asgromo]

zzo38 has finally stopped by to do some heavy lifting for you guys.

[zzo38]

asgromo, on May 16 2008, 07:04 AM, said:

zzo38 has finally stopped by to do some heavy lifting for you guys.

Do you need me to lift something? It's hard to lift something over the internet, you know.

[asiekierka]

What I'D like to see is a BF interpreter built-in to MZX---
---wait, I will add this stuff to my fork!

zzo38: If I did knew forth and think it is good, i'd do it right now, hehe.

[Nightwatch]

ajs, on May 16 2008, 11:51 AM, said:

It's a shame we don't have a regression test for these features, but I guess you'll nail all the bugs eventually.

BTW, it would be good if you could drop the mzx_ prefix on your function names/macros/structures. Nothing else in MZX uses this kind of namespacing, and it's not really necessary.

Well, if I don't use mzx_ I'll stomp on "move". The mzx_ prefix is there as a hint to indicate "this is the new API, use this for future code". Eventually it can be eliminated.

Quote

What I would like to see is not Lua support, what I would like to see is Forth instead of Lua.

You could always write a Forth interpreter in Lua

This post has been edited by Nightwatch: 16 May 2008 - 05:50 PM

[ajs]

Nightwatch, on May 16 2008, 06:49 PM, said:

Well, if I don't use mzx_ I'll stomp on "move". The mzx_ prefix is there as a hint to indicate "this is the new API, use this for future code". Eventually it can be eliminated.

But surely this "new API" would replace the old one completely? Having two ways of doing the same thing with one having some technical advantages is simply not a good idea. There should be one, unified, optimal way of doing it, and this should have a decent name.

--ajs.

[Nightwatch]

ajs, on May 16 2008, 05:58 PM, said:

But surely this "new API" would replace the old one completely? Having two ways of doing the same thing with one having some technical advantages is simply not a good idea. There should be one, unified, optimal way of doing it, and this should have a decent name.

--ajs.

It'll take a long time to get every single MZX function that accesses the board to use the new API, unfortunately. If someone wants to go through runrobo2.c, game.c, game2.c, and so on and convert everything, that'd be great... but I have no time to do that, sorry. I will remove the mzx_ namespacing on everything that doesn't stomp on other things though; that's relatively easy.

By the way, I fixed the remaining bugs I could find.

[ajs]

Nobody expects you to do all the work here, I'd like this to be a cooperative process. However, you've got to see it from my perspective: I can't just be given a code dump consisting of thousands of lines of code, some mergeable, some not and made to work through it, at a time you decide.

We should both be able to agree that much of your stuff can be applied to MZX as-is -- if you fix the bugs in it, of course:

• Bug fixes can go straight in. I'd prefer to not have to find these in a multiple thousand line diff, so if you could break them out I would *much* appreciate that;
  
• Infrastructural changes like adding the HT data structure and using it with the new copy/move/etc. block primitives should be separate from your Lua stuff. I'd also like to see the robo_ops stuff here, if you're still using that;
  
• Necessary core changes to support Lua should be separate again, so we can see exactly what has to change to support it;
  
• Finally, the Lua support should be added, broken up in whatever way you see fit.

This way I can continue to be applying the "known good" conservative changes from you, things that require minimal discussion and should be added to SVN immediately, and people can start testing them and we can start shrinking the code quantity you have left to merge. Let me put it another way: in a worst case scenario, you suddenly get taken away from development here and the Lua editor and Lua/Robotic separation in the robot editor never gets completed -- would you rather all of your changes bit-rotted while MZX dev continued, or would you rather I took as much of it as possible into the main trunk so you can still have usefully contributed to something?

MZX isn't really a very good example of an open source project -- it has few developers and no formal process -- but from my experience with open source I'm trying to change things so that it's easier for people to contribute improvements and features to MZX without them being ignored or lost forever. The only way this works is if things are broken up into a "series" of patches which can be picked or dropped by me, and if we start some more continual dialogue over the code, rather than what has mostly just been hand-waving on my part.

--ajs.

[Nightwatch]

Namespacing is removed.

I'll try to break it up into several branches, lower branches being supersets of all higher branches:
- master (svn)
- bug-fixes
- robo-ops (this is really intimately tied with the new board manipulation code, so it should go together in one branch)
- lua

A lot of it is that I didn't really expect to be making many core changes to MZX when I started, so I wasn't very diligent with git branching. From my perspective originally I was planning to touch Lua and nothing else, but that isn't how things turned out. (I always use git when working on something, it's just too valuable to be able to go through the history if nothing else.)

What I should probably do is archive the old repo and just start anew, using git-svn as a base so that I can stay up to date with svn automatically. Then I can create those branches.

[ajs]

If you can make your git repo available online I can also track it from here. I'm used to both git and SVN, and I recently merged Kev's stuff via git. I recommend you do periodically merge SVN, though I suspect things will cool off for now (I've been doing a lot of work on it this week, but some weeks go by where nothing is committed).

Thanks for the response.

--ajs.

[Nightwatch]

ajs, on May 16 2008, 08:17 PM, said:

If you can make your git repo available online I can also track it from here. I'm used to both git and SVN, and I recently merged Kev's stuff via git. I recommend you do periodically merge SVN, though I suspect things will cool off for now (I've been doing a lot of work on it this week, but some weeks go by where nothing is committed).

Thanks for the response.

--ajs.

I put it online and cleaned things up significantly. Here it is:

git://github.com/pcwalton/megazeux.git

You'll notice there are four branches for now. In order of increasing supersets, they are: "master" is just a mirror of SVN, "hash-tables" adds hash table support, "robo-ops" adds the new board manipulation primitives and Robotic operations, and "lua" adds Lua support via a new CONFIG_LUA option, off by default. Lua support is being significantly cleaned up: you'll notice many fewer World * additions to functions, since robots now have an up-reference to the world in their structures. I think this probably should have been in there since the beginning... it seems cleaner than a global variable anyhow, and it avoids the diff noise.

Pluto patches are now in contrib/ in the lua branch, to be more consistent.

Editor support is not in this branch yet, so don't consider this fit for usage yet. When it goes back in, it will not be possible to mix Lua and Robotic in the same robot. The Alt+L switch will switch between Lua and Robotic for the entire current robot; if your program consists of more than whitespace, it will ask for confirmation and convert your current code to comments in the other language.

Also, I don't guarantee that the lua branch will compile when --enable-lua isn't specified to config.sh; obviously that needs to be fixed if it is indeed a problem.

[Nightwatch]

Ok, feedback time.

After fighting some more with the editor, I'm not convinced that the robotic editor is that good of a match for Lua. The Robotic editor is fundamentally tied to the concept of a line as bytecode, and Lua doesn't work that way. It's very difficult to do even simple things like commenting out code properly, and the number of bugs related to the robotic editor is bad enough without hacking a clumsy generic line editor into it. Not saying that Exo did a bad job, of course... I think the "Robotic is bytecode and its text representation doesn't really exist" design makes it difficult no matter what you do.

I'm thinking about moving to a modified version of the scroll/sign editor for Lua. It shouldn't theoretically be that hard to add a "Check Syntax" shortcut and some simple syntax highlighting for friendliness: Lua is very easy to parse. The only problem is that the scroll editor doesn't have copy/paste and other nice things... but one of the advantages of Lua is that you can read scripts from files on disk, so that isn't a hugely pressing matter right now, as far as I see it. (If you really want to use vi/Notepad/UltraEdit/whatever to edit your Lua, go for it!)

Thoughts?

This post has been edited by Nightwatch: 18 May 2008 - 06:25 AM