DOS ain't dead - MSA assembler released under MIT

DosWorld

10.11.2019, 13:44

MSA assembler released under MIT (Announce)

Hi!

Robert Östling release MSA assembler under MIT-license.
https://github.com/robertostling/msa

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

tom Germany (West), 10.11.2019, 19:00 @ DosWorld	MSA assembler released under MIT Post reply
	> Hi! > > Robert Östling release MSA assembler under MIT-license. > https://github.com/robertostling/msa what's the purpose of yet another assembler? (btw: YASM is already taken) Tom

DosWorld

12.11.2019, 14:42

@ tom

MSA assembler released under MIT

Post reply

> > Robert Östling release MSA assembler under MIT-license.
> > https://github.com/robertostling/msa
>
> what's the purpose of yet another assembler? (btw: YASM is already taken)

This is not new assembler - 2000-2001.
Robert make (re)release with modern license.

Purpose - wide. This is very small compiler and easy for modifications. For example - i modify MSA to have possibility compile overlays (something like dll: exe + symbol table) for DOS.

Also, for me it is good target assembler for my toy-compilers.

If you need macros, full x86 support etc - MSA is not your choice.

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

Rugxulo

Usono,
15.11.2019, 03:24

@ DosWorld

MSA assembler released under MIT

Post reply

> > > Robert Östling release MSA assembler under MIT-license.
> > > https://github.com/robertostling/msa
>
> This is not new assembler - 2000-2001.
> Robert make (re)release with modern license.

I see you've changed build methods (makefile) and compiler (wcl), and this is admittedly old code, so my remarks may be obvious or redundant.

Just FYI:

> bcc -Z -C -1 -O -mt -Id:\lang\bc\include -Ld:\lang\bc\lib msa.c
> exe2bin msa.exe msa.com
> del msa.exe

1). The "-I" and "-L" options should always only be in TURBOC.CFG (as created by the installer).

2). "exe2bin" isn't needed, use "-mt -lt" (since TLINK supports .COM directly).

I did try using it a bit, but it lacked some output size optimizations and had other quirks. I haven't looked much beyond that. (Most code doesn't demand it, but it's harder to verify precisely, compared to others.)

DosWorld

15.11.2019, 16:38

@ Rugxulo

MSA assembler released under MIT

Post reply

> I see you've changed build methods (makefile) and compiler (wcl), and this
> is admittedly old code, so my remarks may be obvious or redundant.

Yes, now, i am remove bcc builds stuff and, now, support Open Watcom C (real-mode DOS target), gcc (Linux), gcc+mingw (Win32).

msa2 (msa) does not support 100% full subset of x86, but i try to extend to full support when find a gaps (something could be missed). For example, today i find and fix uncompilable "retf 2".

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

Rugxulo

Usono,
02.12.2019, 20:47

@ DosWorld

MSA2 and TP7 overlays

Post reply

> i modify MSA to have possibility compile overlays (something like
> dll: exe + symbol table) for DOS.

Apparently you use TP7 for that purpose, which I don't have. So I can't (easily) compile with "freeware" TP 5.5 because it lacks "pchar" (and other things). However, it does support overlays (but I'm not familiar with them at all!).

Anyways, actually I only wanted to mention this (N.B. the forum is getting confused with italics, so I renamed the var byte from "i" to "n"):

> CALLER.PAS
> procedure puts
> for n:=1 to ord(s[0]) do putc(s[n]);

I did put that one function in a separate file, just to prove a point. You can change "ord(s[0])" into "length(s)" (at least with TP 5.5), and the output .EXE has the exact same CRC32. My point is that it might be cleaner, clearer, and more portable to not assume the length is only a byte at offset 0. Compile-time calculations and constant expressions are useful indeed. Just FYI.

marcov 03.12.2019, 16:37 @ Rugxulo	MSA2 and TP7 overlays Post reply
	> > i modify MSA to have possibility compile overlays (something like > > dll: exe + symbol table) for DOS. > > Apparently you use TP7 for that purpose, ^^^^ You rang mi'lord? > overlays (but I'm not familiar with them > at all!). For a summary and link to details see https://stackoverflow.com/questions/10155003/how-did-turbo-pascal-overlays-work

Rugxulo

Usono,
04.12.2019, 03:59

@ marcov

MSA2 and TP7 overlays

Post reply

(Sorry in advance for the ramble. I just want it all in one place so I don't forget later.)

> > [TP7] overlays (but I'm not familiar with them at all!).
>
> For a summary and link to details see
> https://stackoverflow.com/questions/10155003/how-did-turbo-pascal-overlays-work

Don't personally need it, just mildly curious.

You seem to mostly just point to The Slithy Tove, which I haven't read yet. A quick check of Bob Swart's Borland Pascal Efficiency book doesn't show any overlay usage. Probably old SWAG archives have something, but I haven't checked yet either. Maybe TP55 included some example code for it, I think? But I haven't checked those lately, don't remember. No mention of overlays on FPC's DOS wiki page, probably not supported yet.

Freeware TP 5.5 didn't really come with online books or manuals (except a chapter about their OOP). I feel (unsurprisingly) timid about places like BitSavers, even though they also have German mirrors, so maybe it's legal? Ugh, old copyright is annoying.

I know Go32v2 supports DXE1, but I've never used it much there. In fact, I never used it much (except Matt Mahoney's old C++/ASM code, paq8f, ten years ago!). DXE3 is better but rarely used (e.g. Hexen2, whose code I never studied closely).

I'm definitely curious, but I don't need it. Then again, I did recently recompile NASM 0.98.39 for 8086 with OpenWatcom ("large") and TurboC++ ("huge"), and the latter runs out of memory more easily (FD's kernel.asm)! TC++ 1.01 does support overlays, and I do have the actual physical books on that, so I'll probably read up on that soon, out of mild curiosity.

marcov

04.12.2019, 10:41

@ Rugxulo

MSA2 and TP7 overlays

Post reply

> Maybe TP55
> included some example code for it, I think? But I haven't checked those
> lately, don't remember. No mention of overlays on FPC's
> DOS wiki page, probably not
> supported yet.

There is an overlay chapter in SWAG. Possibly overlay handling differs between older versions, TP7 real and TP7 protected mode. (if protected mode exist at all)

> I know Go32v2 supports DXE1, but I've never used it much there. In fact, I
> never used it much (except Matt Mahoney's old C++/ASM code,
> paq8f,
> ten years ago!). DXE3 is better but rarely used (e.g. Hexen2, whose code I
> never studied closely).

DXE is slightly different since afaik DXE can be created using separate generation. IOW it can be used to create plugins for already existing (compiled) programs.

Afaik the 16-bit dos maintainers mainly has demo compos as usecase, and overlays overall increase binary size :-P And of course with FPC there is always the upgrade to 32-bit in case it gets too bad.

Rugxulo

Usono,
16.02.2020, 03:48

@ marcov

Turbo Pascal overlays (overleden? ha!)

Post reply

> > Maybe TP55 included some example code for it, I think?

Yes, it has OVRDEMO*.PAS, but that's fairly minimal.

I gleaned some useful info from Franz Glaser's old website.

> There is an overlay chapter in SWAG.

There might be a unit that loads overlays to XMS in there. (Default is conventional or EMS only.) Haven't checked closely. Most people in DOS these days already have XMS loaded (286+). EMS isn't super rare (especially EMM386), but I never load it by default anymore.

> DXE is slightly different since afaik DXE can be created using separate
> generation. IOW it can be used to create plugins for already existing
> (compiled) programs.

DXEGEN just uses an *.O file. TPC.EXE generates the .OVR from specified units (*.TPU) and main program (main .PAS). It also needs/forces far calls (intra-unit? across units they were always far, AFAIK).

> Afaik the 16-bit dos maintainers mainly has demo compos as usecase, and
> overlays overall increase binary size :-P

TP 5.5 is much smaller output than FPC (but also weaker, of course). It's probably a bad idea to use proprietary, compiler-specific (overlay) code, if it can be avoided. Smartlinking helps, of course.

Okay, so here's the deal:

Just to play around, I have a small project that is nine files (8 units and one main .PAS), not even 600 lines total. It doesn't use the heap at all (so uses roughly 33 kb of memory). The generated .EXE is about 15 kb (uncompressed). I'm using TP 5.5's OOP, so I can easily avoid using four of five units if I don't need that extra functionality (just don't use or declare those objects in main). So that part won't get linked in, and that can save me (up to) roughly 10 kb on the .EXE. That's the best, most portable way, I think (rather than relying on old TP-only overlays). That much is confirmed to work fine on various TP-compatible compilers.

However, with TP 5.5, I can make four of those units into an overlay. The .OVR is about 6 kb and the main .EXE will be about 9 kb. No, it's not mandatory, two of the five objects' functionality won't need the functions in the overlay. So I can still partially run that .EXE without the .OVR found, if I'm careful. BTW, using an overlay literally only saves me 2 kb of memory here! Also, UPX won't compress the .OVR (not sure about PKlite or whatever else). You could almost pretend .OVR is a .DLL here, but it's not worth it. The overlaid (subset) main .EXE (two units' functionality), if compressed by UPX, is slightly larger than the (full) .EXE from before (five units' functionality, when using no overlays)!

Probably not a great example use for overlays, but I tried anyways. (I don't know OOP "design patterns" at all.) It could definitely be a useful feature, but not so much here. (Remember I rebuilt old NASM 0.98.39 with Turbo C++ 1.01? I assume NASM's preprocessor would make a great overlay since you only need it at the beginning. But I have no idea how to fix that yet. I still need to read the physical manuals there.)

P.S. By coincidence, it's Niklaus Wirth's birthday. Happy 86th!

Rugxulo

Usono,
18.02.2020, 03:03

@ Rugxulo

Turbo Pascal overlays

Post reply

> Just to play around, I have a small project that is nine files (8 units and
> one main .PAS), not even 600 lines total. It doesn't use the heap at all
> (so uses roughly 33 kb of memory).

Because overlays use the heap, I wasn't sure, so I just removed the {$M ...} pragma from my source. But I should've just used {$M 4096,0,655360} to minimize the stack (which is a generous 16k by default, quite large for 16-bit). So that saves me an additional 12k. So the program really only needs about 20k of RAM. Obviously it doesn't matter than much on "modern" (full 640k) machines, but since we're splitting hairs anyways ....

ecm

Düsseldorf, Germany,
05.03.2020, 16:42

@ Rugxulo

Turbo Pascal overlays & NASM Preprocessor

Post reply

> Probably not a great example use for overlays, but I tried anyways. (I
> don't know OOP "design patterns" at all.) It could definitely be a useful
> feature, but not so much here. (Remember I rebuilt old NASM 0.98.39 with
> Turbo C++ 1.01? I assume NASM's preprocessor would make a great overlay
> since you only need it at the beginning. But I have no idea how to fix that
> yet. I still need to read the physical manuals there.)

Actually, the NASM preprocessor is run on each pass of the assembler. That's why my sources won't work with nasm -E (preprocess-only mode). For example, I often use constructs like %assign foo $ - $$. Using equ and *then* %assign allows even to learn of numbers that cannot be evaluated (as a critical expression) at that point in the source yet. I barely use the latter as yet, though.

For an example of the former, consider the _386 patch section macro in debug.mac. It uses %assign and then after the to-be-patched part %rep, each with $ (current assembly address).

During the work to split the data/entry section from the code section (and address them using different segments), I had to enable the _WPT_LABELS option, which avoids the %assign trickery in the patch macro. As noted there, this made the assembly take more than 2 minutes. Therefore I disabled this option as I finished the split.

---
l

Berlin, Germany,
03.12.2019, 21:34

@ Rugxulo

MSA2 and TP7 overlays

Post reply

> Anyways, actually I only wanted to mention this (N.B. the forum is getting
> confused with italics, so I renamed the var byte from "i" to "n"):

Nice find! This is caused by using italics for the word "exact" later. So, if you change "n" to "b" and use bold on "exact", that {b} will also disappear like the {i} before.

You can't even prevent it by enclosing the Pascal statement in {code}{/code}. I will report this behaviour to Heiko.

> > CALLER.PAS
> > procedure puts
> > for n:=1 to ord(s[0]) do putc(s[n]);
>
> I did put that one function in a separate file, just to prove a point. You
> can change "ord(s[0])" into "length(s)" (at least with TP 5.5), and the
> output .EXE has the exact same CRC32. My point is that it might be
> cleaner, clearer, and more portable to not assume the length is only a byte
> at offset 0. Compile-time calculations and constant expressions are useful
> indeed. Just FYI.

(Just cited to reproduce the bug.)

---
Forum admin

Rugxulo Usono, 14.12.2019, 13:28 @ Rugxulo	MSA2 and TP7 overlays Post reply
	> (N.B. the forum is getting confused with italics, so I renamed the > var byte from "i" to "n"): > > > CALLER.PAS > > procedure puts > > for n:=1 to ord(s[0]) do putc(s[n]); I forgot that most Pascal compilers (including TP) also support "s(.i.)" syntax. So that's a good workaround here.

Berlin, Germany,
14.12.2019, 23:42

@ Rugxulo

MSA2 and TP7 overlays

Post reply

> > (N.B. the forum is getting confused with italics, so I renamed the
> > var byte from "i" to "n"):
> >
> > > CALLER.PAS
> > > procedure puts
> > > for n:=1 to ord(s[0]) do putc(s[n]);
>
> I forgot that most Pascal compilers (including TP) also support "s(.i.)"
> syntax. So that's a good workaround here.

I never saw this kind of syntax before.

---
Forum admin

marcov 15.12.2019, 21:07 @ rr	MSA2 and TP7 overlays Post reply
	> I never saw this kind of syntax before. Pascal originates on a system (CDC 6600) with 6-bit characters. There are some shortcuts for characters not in the default (source) character set. Other workarounds are e.g. ^A..^Z for #1..#26

Rugxulo

Usono,
16.12.2019, 19:15

@ marcov

MSA2 and TP7 overlays

Post reply

> > I never saw this kind of syntax before.
>
> Pascal originates on a system (CDC 6600) with 6-bit characters. There are
> some shortcuts for characters not in the default (source) character set.

Some monitors (charsets?) or keyboards were more limited than others.

> Other workarounds are e.g. ^A..^Z for #1..#26

That's TP specific, no?

Anyways, quoting original Pascal (ISO 7185):

> @ = ^
> (. = [
> .) = ]
> (* = {
> *) = }

Just for comparison, Modula-2 has the following:

> [ = (!
> ] = !)
> { = (:
> } = :)
> | = !
> ^ = @

DosWorld

15.12.2019, 21:42

@ Rugxulo

MSA2 and TP7 overlays

Post reply

> Apparently you use TP7 for that purpose, which I don't have. So I can't
> (easily) compile with "freeware" TP 5.5 because it lacks "pchar" (and other
> things). However, it does support overlays (but I'm not familiar with them
> at all!).
>
> I did put that one function in a separate file, just to prove a point. You
> can change "ord(s[0])" into "length(s)" (at least with TP 5.5), and the
> output .EXE has the exact same CRC32. My point is that it might be
> cleaner, clearer, and more portable to not assume the length is only a byte
> at offset 0. Compile-time calculations and constant expressions are useful
> indeed. Just FYI.

Code for CALLER.PAS was wroten very fast - so i am ask for leniency.
You are right, ut is good point - fix it for TP5.5.

PS: I am little be stuck in code generation process (problem when label change offset betweet compiler pass) - i am try convert msa2 into single pass assembler. When i solve this trouble in my way - i will be very near to obj generation. This is the reason, why i have no new commits now (nothing ready).

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

Rugxulo

Usono,
16.12.2019, 19:32

@ DosWorld

MSA2 and one-pass assemblers

Post reply

> Code for CALLER.PAS was wroten very fast - so i am ask for leniency.
> You are right, ut is good point - fix it for TP5.5.

No worries, I don't demand anyone fix it. Just saying, I can't properly build it. But I still wanted to pass along the tip about the length function.

> PS: I am little be stuck in code generation process (problem when label
> change offset betweet compiler pass) - i am try convert msa2 into single
> pass assembler. When i solve this trouble in my way - i will be very near
> to obj generation. This is the reason, why i have no new commits now
> (nothing ready).

I'm not fully sure what "one pass" means. It must mean different things to different people.

A86 is one pass and fast. TASM was originally one-pass only (until version 2.0). PSR Invaders comes with a pre-assembled .COM, but he forgot to use multi-pass (/m or /m3), so it's inefficient output (9356 vs. 9194 bytes). Even A86 won't automatically shorten forward jumps (except local labels), only backwards jumps, but otherwise it optimizes for size where it can.

NASM 0.97 was two-pass only but didn't optimize for size. That was added later (-O3 or -Ox), eventually becoming default. Some assemblers are better about size than others. I don't blame those that don't, simplicity is a virtue, and slow hardware of the time hampered everyone. Still, verbose source annotations just to save a few bytes are redundant. But usually it doesn't practically matter at all anyways.

Just saying, adapting non-standard x86 asm syntax across various assemblers is a bit of a chore, but it's worse when instruction opcode sizes don't match up (changing all offsets and displacements). The varying instruction size (unlike RISC) makes it much harder to verify that the program is accurate to the original. In other words, unless INVADERS.COM is exactly 9194 bytes, I don't know how to "prove" it's correct (even with disassembly). Sometimes I've had to do some weird kludges just to get it to match 100% exactly (with instructions, not encodings).

Rugxulo

Usono,
12.02.2020, 08:18

@ DosWorld

MSA2 compilation with GCC

Post reply

I haven't (yet again) tried MSA2 recently. In fact, I only once barely tried to convert PSR Invaders and failed (and gave up). But, out of boredom, I did check your Github today again (thinking of TP overlays, barely):

> To build Linux version with GCC:
>
> gcc -O3 MSA.C TABLES.C ASSEMBLR.C MISC.C LEX.C -o MSA2
> strip MSA2

Just FYI:

.C (capital C) means compile as C++ with GCC. I guess you didn't notice yet? (They're like 95% compatible.) You can use "-x c" to always override filename detection.

Also, strip can be done with "gcc -s" (which I think actually just refers to "ld -s" instead). So you don't need two separate steps to compile and strip.