[Exophase]

As I tend to do sometimes, I've been thinking about, looking over, and this time even actually implementing some potential changes/improvements to MZX. Same kind of story as for years, but nothing terribly ambitious this time around (or at the very least, nothing that I would ever intend to do in too large steps). Not that I don't have better things to do than this, but I think the longer I stay out of Robotic and program in the real world the less likely it is that I'll be able to tolerate actually programming in it ever again. And I actually would like to return to it, but myself aside I think that this specific problem could affect a lot of people in the community. For those who aren't, they'd probably still benefit from some tweaks to the language and development environment.

Anyway, as most of you are probably aware, there have been some pretty bad design decisions behind MZX. I would say that just about every person who was contributed to MZX has implemented something in a strange kind of way, and unfortunately these things and other technical issues have compounded resulting in a mess. I in particular have done some things in a way that I'm not happy with. Some of these things could eventually be covered up internally without disallowing anything to be done the way it is now, but that is far from elegant. Other things are flat out locking MZX back.

What I'm basically asking is if people would be okay if some established syntax/convention in MZX was outright removed. Just so this is clear, I'm not talking about breaking compatibility - the current way of doing things can be frozen and converted to the new way (possibly with some edge cases). The thing is, with the way I'd like to see progression go, I don't think that this is something that we can keep going. Robotic would have to turn into a cleaner language in one step.

Here are some of the things I'm referring to:

- rel counters is pretty much useless; supporting it in the current 1 to 1 source to bytecode format is costing us more as well, although it wouldn't get in the way if we moved away from it. I still think it's a clumsy way to do things that shouldn't be supported anymore.

- A lot of the new features have been implemented in strange ways since we couldn't do a lot to add to or change Robotic. Some of it is weird but sort of natural (like copying to/from MZMs, inbetween layers, sprite stuff) - others were done in a way that's totally backwards (file access). Again, this can be hidden by the language compilation later but it'd be best if we changed it and didn't support the old stuff anymore.

- The symbols for AND, OR, and XOR in expressions.. I really dropped the ball on this one. Because they use such common letters they're basically preventing us from having unquoted expression names, which is a huge pain. I'd like to change them to something else.

- Expressions don't follow standard order of operations either. Quite frankly, I don't really mind it like this, but it's pretty improper. I think this would probably take actual adjusting and I'm okay with it staying as it is.

- The reason why I used the letters for those in the first place is because & was reserved for counter name interpolation. Now that expressions can do the same thing (and are more general) I think we should stick with that only. Of course, something will need to be done to make expressions capable of evaluating strings to really replace it...

- Message boxes are implemented as multiple commands - internally they could be merged into one with proper language compilation, but it'd prevent the ability to interrupt them. I don't think this is something that's really used a whole lot, so it should be okay if it isn't supported anymore. Making backwards compatibility for it would be kinda a kludge though.

I'm sure there are lots more. Let me know how you feel about this sort of thing.

[KKairos]

Backwards compatibility is my one condition for a "tolerate changes."

(I voted Yes.)

[paulguy]

I'd fully support a better language.

[Risu2112]

That dumpy old syntax can go fuck off as long as my backwards compatibility is preserved since I almost certainly would not be successful porting my work over to the new one.

[commodorejohn]

Backwards compatibility would be nice, but not absolutely necessary. As long as there's a fully stable older version I can revert to to run older games, go ahead and revamp it as much as you want; I can adapt.

[Pyro1588]

are there any other commands you'd consider stripping out? and is it safe to assume that you'd be using the spot opened up when you strip rel counters for something a little more valuable, like file access?

as far as order of operations goes, i've always just used massive numbers of parentheses to ensure everything goes in proper order. the only problem with this is maximum line length. would there be any way to increase it?

and on a similar note, if you can't increase line length, would you still allow multiple commands for message boxes? i'm certain that i have message boxes somewhere in my game that exceed standard mzx line length.

one more question: set chars was used to adjust some global values and whatnot. are any of these still open and unused? could they be useful for anything programming-wise or would it just make more sense to use counters?

[LogiCow]

I don't think I would mind.

Then again the new spacebar toggle behavior on robot pisses me off terribly.

[Exophase]

Pyro1588, on Dec 26 2008, 12:57 PM, said:

are there any other commands you'd consider stripping out? and is it safe to assume that you'd be using the spot opened up when you strip rel counters for something a little more valuable, like file access?

as far as order of operations goes, i've always just used massive numbers of parentheses to ensure everything goes in proper order. the only problem with this is maximum line length. would there be any way to increase it?

and on a similar note, if you can't increase line length, would you still allow multiple commands for message boxes? i'm certain that i have message boxes somewhere in my game that exceed standard mzx line length.

one more question: set chars was used to adjust some global values and whatnot. are any of these still open and unused? could they be useful for anything programming-wise or would it just make more sense to use counters?

Good questions, pyro.

If rel counters is kept it'd probably still be removed as a command in the bytecode (instead replaced by expression expansion). This is just kind of a painful thing to have to code in to the compiler, IMO. However, I wouldn't really worry about optimizing the number of command slots. It could probably be moved to 2 bytes for the command number instead of 1. There are various other places where bytes could be removed anyway (like forward and backward length), and it isn't that critical anyway.

The line length shouldn't end up being an issue because it should be possible to break up commands over multiple lines. Furthermore, in the "revised" basic language, expressions wouldn't have to be enclosed in quotes (but would have to be in parens still). If it's necessary to break up a string over multiple lines, what I was thinking of doing was adding a delimiting character to merge two strings, ie:

set $str to "This is one"+ "complete sentence"

Then you could break it up over one line. Since all the commands use a fixed number of arguments, and so long as there are concrete rules for what an argument can be (what we have now is pretty restricted, which is that arguments have to be separated by spaces - the exception is if they're surrounded by quotes or, in the future, by parens). Actually, they could take a variable number so long as the first argument says how many can be expected to come.

I'm not really decided on how to do message boxes yet. I'd like for them to be one command per box, but that requires encoding something into them to say how the different lines work. I guess we could keep it like it is and have it internally be represented as pushing a bunch of text into a box then having some kind of ending command for it, I don't know.

Just so everyone's clear, when I say backwards compatibility I mean that the robots will be converted to whatever is the compatible format in the newer version. If you load it and save it then you'll end up with a different world file. The old format will then not work in anything newly created.

[Risu2112]

Does that suggest that playing an old world file that has never been edited in the newer version would undergo a similar conversion before actually being played? Or would old world files still be supported to run in their native state?

[Wervyn]

Definitely support this, with the same caveat as everyone else that backwards compatibility is paramount. People can learn to do things better, but we want to make absolutely sure that all of the old games, or as much of them as possible, still work. Pay extra special attention to things like load_robot and load_bc, because people DO use these, and if you don't figure out how to handle them properly under the new system (and I'm at something of a loss as to how that should work), games will break. Notable example: Eternal Eclipse Taoyarin.

But as for the language changes, I'm in complete agreement:

• Anyone who still uses the rel commands should be hit in the head with like a sword but a hammer. These are seriously even more useless and redundant now than the avalanche command.
  
• The use of 'a', 'o', and 'x' in expressions was always an eyebrow raiser for me, even considering the issues with & which I understood. Of course, keep in mind that if you do switch these to something more conventional like &, |, and ^, that ^ is currently the exponentiation operator. You might consider, as long as you're changing things up like this, turning ** into an exponent, since I know I'd prefer to use conventional bitwise operators instead of something custom.
  
• On operator precedence, someone recently commented to me that the concept of precedence is a loan from mathematics, where in a programming setting what you're really concerned with is the computer doing what you thought you told it to do. The only reason to use precedence is to avoid confusing people who are used to things being that way, from math class. And when combining expressions with comparison operators and bitwise operators there isn't a well known convention on exactly what should happen, so people use parentheses anyway. My advice is to leave everything the way it is now, with strictly left-to-right order of operations except for unary operators like - and ~, which take overriding precedence. It's deterministic in a way that's simpler to understand than trying to remember if multiplication trumps a bitwise AND or not.
  
• On strings and incorporating them into expressions, I think the best way to do this is to evaluate them as integers only when the operator calls for it. For instance, ($astring) should return the value as a string, while ($astring+50) would attempt to numerically evaluate the string and add it to the literal integer. ($astring+$bstring) would attempt to concatenate the strings and return the result. This is particularly important when it comes to interpolation of strings, as can be done now with ampersands. In these cases it's probably necessary to use single quotes as an explicit "evaluate this counter or string" operator, so that ('$anobject'.'$aproperty') could be understood as a makeshift object reference, and then implanted as a normal value (which, who knows, could be a string itself) in a larger expression.
  
• Message boxes are a giant hack and could certainly benefit from some refactoring. Keep in mind all of the features they incorporate though, and that as much as we advanced coders may recommend the code-your-own option, people still use them and probably still will. This includes things like the & command for centering the line, and the ? command for providing a link. You want to keep the ability to do all of these things, and yet I'm not sure HOW to do them outside of the multi-command format currently employed.

Also, for the road, things I would like. Eventually.

• Code blocks.
  
• Control flow (if, while, for)
  
• Functions
  
• Unified layer to layer, layer to string to layer, layer to MZM to layer handling.
  
• The support for importing extended character set information onto the vlayer, like we discussed before, Exo.
  
• This is too much like a general MZX wishlist now.

"And now I'm going to escape on my jetpack."

[ajs]

I have no problem with the improvements or the methodology but I am concerned that the changes would be released incrementally or changed and then changed again. Retaining backwards compatibility by "converting" source is already a huge pain in the ass, it'll really suck if we have to support some version in between $NOW and $THE_FUTURE which has a slightly different, say, expression syntax, or similar.

So, perhaps contrary to common software engineering principles, I'd suggest that these changes all be made at once. They're done, merged in, and not touched again. Alternatively, it might be a good idea to support both language variants side by side. People don't want to have their projects converted by upgrading to 2.85 and then find that they can't go back, or the variant of the language they were using is revised in such a way that the source converter doesn't support it. (Just to clarify, I'm talking about source editing here. People might start making a game in 2.82 and then continue making it in 2.85 and then finally people would play it on 2.90.)

I think if we're going to revise the language it'd be better to not make it the same progressive garbage the previous (admittedly different) revisions were.

--ajs.

[hayashi]

I suppose you could make like Python, find someone to maintain MZX 2.x, and then make a MZX 3 that breaks compatibility along with a translator?

[Exophase]

Wervyn, on Dec 26 2008, 05:26 PM, said:

Definitely support this, with the same caveat as everyone else that backwards compatibility is paramount. People can learn to do things better, but we want to make absolutely sure that all of the old games, or as much of them as possible, still work. Pay extra special attention to things like load_robot and load_bc, because people DO use these, and if you don't figure out how to handle them properly under the new system (and I'm at something of a loss as to how that should work), games will break. Notable example: Eternal Eclipse Taoyarin.

Should work fine. Without going into too much detail at this point, the basic change is that robots in world files (not save files) and loaded externally will be expected to be source code. load_bc would still be supported for legacy purposes, but it'll no longer be possible to export bytecode from a robot. This means that all bytecode will be frozen at the 2.82 compatible level, so the disassembler will also be frozen to this. The conversions will be done at the bytecode level, not at the source level, and will also be part of the disassembler. This is why it's important to get these things all done in one version so they can be kept here and not placed in the source code. So we need to figure out everything that should be changed.

The big reason why it's important to get this done in one stage is that it keeps the conversions to bytecode, not source code. What this means is that it's less legacy syntax that we have to keep checking.

Wervyn, on Dec 26 2008, 05:26 PM, said:

But as for the language changes, I'm in complete agreement:

• Anyone who still uses the rel commands should be hit in the head with like a sword but a hammer. These are seriously even more useless and redundant now than the avalanche command.

Yes.

Wervyn, on Dec 26 2008, 05:26 PM, said:

The use of 'a', 'o', and 'x' in expressions was always an eyebrow raiser for me, even considering the issues with & which I understood. Of course, keep in mind that if you do switch these to something more conventional like &, |, and ^, that ^ is currently the exponentiation operator. You might consider, as long as you're changing things up like this, turning ** into an exponent, since I know I'd prefer to use conventional bitwise operators instead of something custom.

At the time, I basically never thought it'd make sense to not quote counter names, and these felt like something natural to use (I wish I had used OR, XOR, and AND - I probably didn't want to swallow the extra parsing load). I did intend to change exponentiation to something else like ** as well, like you said.

Wervyn, on Dec 26 2008, 05:26 PM, said:

On operator precedence, someone recently commented to me that the concept of precedence is a loan from mathematics, where in a programming setting what you're really concerned with is the computer doing what you thought you told it to do. The only reason to use precedence is to avoid confusing people who are used to things being that way, from math class. And when combining expressions with comparison operators and bitwise operators there isn't a well known convention on exactly what should happen, so people use parentheses anyway. My advice is to leave everything the way it is now, with strictly left-to-right order of operations except for unary operators like - and ~, which take overriding precedence. It's deterministic in a way that's simpler to understand than trying to remember if multiplication trumps a bitwise AND or not.

Bad precedence assumptions are actually a serious problem in programming languages like C that have a lot of operators beyond the basic four arithmetic ones. Simple left to right evaluation is just easier to remember. I tend to coat parens on just about everything anyway (although I sometimes let things like == slip), so my apathy might be because of this. But yeah, I think we're in agreement, and I don't think anyone else has really cared so far either.

Wervyn, on Dec 26 2008, 05:26 PM, said:

On strings and incorporating them into expressions, I think the best way to do this is to evaluate them as integers only when the operator calls for it. For instance, ($astring) should return the value as a string, while ($astring+50) would attempt to numerically evaluate the string and add it to the literal integer. ($astring+$bstring) would attempt to concatenate the strings and return the result. This is particularly important when it comes to interpolation of strings, as can be done now with ampersands. In these cases it's probably necessary to use single quotes as an explicit "evaluate this counter or string" operator, so that ('$anobject'.'$aproperty') could be understood as a makeshift object reference, and then implanted as a normal value (which, who knows, could be a string itself) in a larger expression.

We'll need to have a casting procedure if anyone does use them in expressions to convert to ints, though. Of course, this would probably still be okay:

set counter $string

But we'd have to have temporaries created for this if they're used in strings. I doubt that this has been done a lot so it isn't a huge deal. I'd prefer not to do it with function counters because I'm nervous about implementing those with string args (for now)

Wervyn, on Dec 26 2008, 05:26 PM, said:

Message boxes are a giant hack and could certainly benefit from some refactoring. Keep in mind all of the features they incorporate though, and that as much as we advanced coders may recommend the code-your-own option, people still use them and probably still will. This includes things like the & command for centering the line, and the ? command for providing a link. You want to keep the ability to do all of these things, and yet I'm not sure HOW to do them outside of the multi-command format currently employed.

Yeah, I'm trying to think of a good way too. I thought about encoding some kind of control structures for it, but that would be less intuitive to program. There's also the problem that all 256 chars can and are used in the boxes, so they'd have to be escaped too - and for graphical things this will make them unpleasant to look at.

Actually, variadic commands CAN be used, so long as it's clear when the next one's starting (command names can't be used as args right now anyway, although we might want to change that and just have priority - it's really annoying not being able to use an unquoted "loop" counter name for instance). So basically, the command would stay pretty similar, maybe something like this:

[ "This is the first line of my message box."
"This is the second line."
center "This line is centered."
link "Click this" it_goes_here
option "Maybe this will be here" if_this_is_true and_it_goes_here

(actually, & was overloaded to be variadic anyway. I wonder if that ever gave me problems...)

Wervyn, on Dec 26 2008, 05:26 PM, said:

Also, for the road, things I would like. Eventually.

• Code blocks.
  
• Control flow (if, while, for)
  
• Functions
  
• Unified layer to layer, layer to string to layer, layer to MZM to layer handling.
  
• The support for importing extended character set information onto the vlayer, like we discussed before, Exo.
  
• This is too much like a general MZX wishlist now.

Let me address these a little bit.

Code blocks and control flow - consider this model for macros: a macro starts with #, may appear anywhere a term can appear in Robotic, and can consist of multiple arguments. A macro argument is parsed like a command argument (so there are limited syntax rules, and it eats up a bunch of them), plus perhaps some more, such as code blocks. So you could really easily write loop macros that look like this:

#for i 1 to 10 in
{
* "~f(i)"
}

That matches a macro something like this - take the syntax tentatively because I made it up on the spot and I don't even remember what I used for extended macros:

macro for
loop_var:var start:var_or_constant _to end:var_or_constant _in code:code_block
set \var start
: _\random_sym(loop_name)
body
inc \loop_var 1
if \loop_var < end \loop_name

It'd take a few minutes to write all of these. What we'd need is a good way to incorporate these into the world files, though. I think the "include" directive would be a really good start - the robot compiler could even be smart enough to cache compilations of these, or something like that. It's not quite real OOP inheritance, but we can go from here.

Functions - once compiled expressions are in this won't be that big of a deal, at least to do in some fashion (might not be the most efficient). But I wouldn't want to compile expressions until the robot VM was made more efficient.

Layer to layer etc copying - they pretty much already are. Just need some tweaking to the syntax, I think.

Ext char set stuff - pretty much as planned before, but first I'd want to introduce a good plan for extending the world format while still making it easy to retain compatibility with the old stuff. I want to make more things that we already have editable in the worlds and not just creatable at run time.

One other thing that I'd like to change, that has bothered me for a long time now: I don't like how quotes (") are used to delimit counter names, because they're already used for strings. Expressions use single quotes ('), but unfortunately this can't be used for counters in general because they're already used for characters, so there'd be an ambiguity. What I'd like to do is use the backtick (`) character for both of these and no longer allow either. This isn't as big of a change as it sounds because I think that most counters won't need to be delimited, and this will encourage people to stop using ones that need to be if it means less overall typing work.

'Risu2112' said:

Does that suggest that playing an old world file that has never been edited in the newer version would undergo a similar conversion before actually being played? Or would old world files still be supported to run in their native state?

Yes and no. The old games store robots in bytecode, which has to be converted to source code (via the disassembler we already have, but will be extended to handle conversions) and then this will be compiled once the robot is actually executed (or copied; I won't explain all the methodology here). But this conversion will only happen internally; the world file itself won't ever change behind your back. However, it will happen if you load it in the editor then save it. In fact, I strongly suggest we add a warning before it lets you save up to a newer version because I think people won't always realize this. It's already a minor problem for compatibility purposes. While on that note, I think it'd be great if the converter also changes some of these compatibility traps so that they can work naturally while being newer version, so that this is less of a problem. These things we actually can add incrementally since they're no longer supported for new games anyway, and would just be restoring something that is being lost right now.

[Pyro1588]

first off, i highly endorse the warning before saving in a new version. i've run into problems with that before and it'd be a nice simple way to avoid them.

second, i have a request as long as we're discussing compression options. after having ajs explain what exactly tar, gz, and bzip2 are and how they work together, i'd like it if we had a way to download and play games as single-file archives. i'm not sure exactly how we'd do it and what format we'd use (zip was also discussed,) but i think it would add another level of user-friendliness if newbies didn't have to download and then extract the archive.

one problem with this would be games that modify files. ajs mentioned the difficulties and resource cost of extracting, modifying and recompressing whatever file is modified. ajs also mentioned the issue of when the game decides to update files in the archive. again, i'm not exactly sure how one would handle this, but ajs brought it up and i'm sure it's something that will need to be planned out.

another lesser issue would be the old method of splitting music and other unnecessary files into alternate archives. if we decide to implement single-archive games, old ones (a la cans 3) need to be extracted along with music and recompressed into a single archive.

sorry if i covered any points already addressed. overall, i really like the idea of having an entire game in one archive.

[ajs]

FWIW, I think compression is being discussed more closely on the feature request thread. However I do agree that "playing a ZIP" and "working with compressed games" are both worthy and conceivable goals. Together with an inline archive browser (slated for 2.83; we'll see how the updater and network code pans out, but it should be good) we should be able to get a good way towards user friendliness. If Exo wants the VFS features to be more explicit for whatever reason, that would probably be another breaking change, though far less technically significant than the language changes discussed here.

--ajs.

[Maxim]

I find Robotic code to be more than sufficiently readable and simple to use as it is. Personally I wouldn't fix what isn't broken, but if changes are actually made by the devs, I imagine they'll be good.

Everyone else is right; we need to protect backwards compatibility for our large, elaborate games that would be difficult to convert. Who knows when we'll get another Drelick or Lancer? These games are like golden eggs here.

Though I haven't given up on the idea of adding to the pile myself; someday we'll see Maxim MZX epic game..

[CJA]

You can change ANYTHING as long as you make a new extension. Yeah, new extension. .mzx2, .mz2, .mz3, whatever. I just want to be able to discern between new and old, dudes!

[Exophase]

That's not going to be happening.

[CJA]

Seriously though don't take rel counters away

[Dr Lancer-X]

get out cja

[Sai'ke]

Redundant as they may be... the rel counters are kind of newbie friendly.

[Dr Lancer-X]

yeah, but come on. it's been nearly 7 months since the last post in this thread! nobody cares

[Terryn]

It's been three months since Exo's touched debytecode anyway, so...yeah

[Old-Sckool]

Is anyone working on this still?

[Frobozz]

Nope. MegaZeux is dead in the sense that its in a maintenance cycle until someone decides they want to try implementing new features. Which may or may not happen.

[asiekierka]

Frobozz, on Dec 2 2009, 01:15 AM, said:

Nope. MegaZeux is dead in the sense that its in a maintenance cycle until someone except asiekierka or zzo or drippin' whyfi decides they want to try implementing new features. Which may or may not happen.