DOS ain't dead

MAWK version 1.3.4-20240622 for DOS built with DJGPP.

mawk is an interpreter for the AWK Programming Language. It uses a bytecode interpreter, and is known for speedy execution.

Requires an 80386 processor or better and CWSDPMI.EXE

https://archive.org/details/mawk134-20240622-for-dos

gopher://tilde.pink/1/~bencollver/files/dos386/devel/mawk/

> MAWK version 1.3.4-20240622 for DOS built with DJGPP.

I rebuilt the old 16-bit DOS version 1.2.2 (circa 1996) with TC++ in 2020 (thus no longer bound DOS+OS/2, thus now UPX compressible). Yes, this was faster, more compatible, and had more free memory than other 16-bit AWKs.

Re: newer versions, for a while I was double-checking scripts with a modern 32-bit MinGW build under HX.

> (circa 1996) with TC++ in 2020 (thus no longer bound DOS+OS/2, thus now UPX
> compressible). Yes, this was faster, more compatible, and had more free
> memory than other 16-bit AWKs.
>
> Re: newer versions, for a while I was double-checking scripts with a modern
> 32-bit MinGW
> build under HX.

That's fun. I briefly tinkered with HX and had trouble locating a toolchain that i could build win32 programs with to run on HX. It doesn't work with recent MinGW, and i couldn't find a mirror with MinGW versions old enough to be compatible with HX. Do you know where to find it?

> That's fun. I briefly tinkered with HX and had trouble locating a
> toolchain that i could build win32 programs with to run on HX. It doesn't
> work with recent MinGW, and i couldn't find a mirror with MinGW versions
> old enough to be compatible with HX. Do you know where to find it?

I never really used MinGW much. But maybe you need ReactOS's MSVCRT.DLL. (I think mine is an old one from 2012, possibly from 0.3.14, can't remember exactly.) You also may have to set DPMILDR=128 or 136 and/or set HDPMI=32.

I do know that OpenWatcom's Win32/PE output seems to run fine under HX. So I'd suggest to prefer that.

> I do know that OpenWatcom's Win32/PE output seems to run fine under HX. So
> I'd suggest to prefer that.

Just out of curiosity, in which ways would a win32 build be preferable to a DJGPP build?

> > I do know that OpenWatcom's Win32/PE output seems to run fine under HX.
> So
> > I'd suggest to prefer that.
>
> Just out of curiosity, in which ways would a win32 build be preferable to a
> DJGPP build?

I don't think it is better. I would almost always prefer DJGPP. I meant "if you must use HX, stick to OpenWatcom".

In theory, having only one binary that runs on two (incompatible) OSes would be nice (even if relying on external API emulation).

In 2008, Japheth recompiled the FTE text editor with HX. There he implied a "true flat memory model" was better somehow. (For DJGPP, I think for stability they restricted access to low DOS memory.) He also (rightly) said DJGPP was a bit bloated, having never been optimized for size, so his way gave a slimmer binary. (UPX helps but DJGPP UPX'd stuff runs much slower under NTVDM. Not sure whether specific DPMI hosts would be better overall with demand paging for speed or lower memory usage.)

HX also supports .DLLs. (DJGPP has DXE3, which has had a lot of work done, but I'm unsure if it's truly equal in robustness.)

Thanks for the explanation! My takes are that HX has:

* a cleaner memory model
* a lean runtime
* DLL support

I plan to experiment with OpenWatcom + HX when i find the time.

> In theory, having only one binary that runs on two (incompatible) OSes
> would be nice (even if relying on external API emulation).


You mean like APE and Cosmopolitan?

https://justine.lol/ape.html
https://justine.lol/cosmopolitan/

> > MAWK version 1.3.4-20240622 for DOS built with DJGPP.
>
> Re: newer versions, for a while I was double-checking scripts with a modern
> 32-bit MinGW
> build under HX.

I tried to run the latest win32 mawk build under HX.

https://invisible-island.net/datafiles/release/mawk-mingw32.zip

It gave me the following error.

C:\mawk\bin>mawk
DPMILDR32: Module=msvcrt.dll
DPMILDR32: File not found error
DPMILDR32: msvcrt.dll: cannot load PE file

I copied the latest reactos msvcrt.dll into the same directory as mawk.exe. I tried again and it gave me the following error.

C:\mawk\bin>mawk

dkrnl32: exception C0000005, flags=0 occurred at BF:330B9C
        ax=8000000 bx=7FFFC8A6 cx=3 dx=33C5A8
        si=33E000 di=326D00 bp=326D40 sp=326CF0
        ip = Module 'kernel32.dll'+9B9C
        [eip] = AC 22 C0 75 FB 4B 23 DB 75 F6 AC AA
        [esp] = 00330CEA 00000168 80000037 00000000 00326D20 003436DC
dkrnl32: fatal exit!

Any guidance? Thanks!

> I copied the latest reactos msvcrt.dll into the same directory as mawk.exe.
> ...
> Any guidance? Thanks!

The msvcrt.dll supplied with Windows XP seems to work - and I guess the ones supplied with Win9x are also ok.

---
MS-DOS forever!

> The msvcrt.dll supplied with Windows XP seems to work - and I guess the
> ones supplied with Win9x are also ok.

Thanks! I tried the following MSVCRT.DLL and at first glance it appeared to work:

https://archive.org/download/microsoftwindowsprein...%29%20%28EN%29.iso/I386%2FSYSTEM32%2FMSVCRT.DLL

C:\BEN\mawk\bin>mawk "BEGIN{print \"hi\"}"
hi

But on closer inspection, it appears to have trouble running itself.

C:\BEN\mawk\bin>type test.awk
function sys(str) {
    printf "test.awk: %s\n", str
    retval = system(str)
    print ""
    return retval
}

BEGIN {
    sys("cd")

    cmd = ARGV[0] " \"BEGIN{print \\\"hi\\\"}\""
    sys(cmd)
    exit 0
}
C:\BEN\mawk\bin>mawk -f test.awk
test.awk: cd
C:\BEN\MAWK\BIN

test.awk: C:\ben\mawk\bin\mawk.exe "BEGIN{print \"hi\"}"
C:\FreeDOS\BIN\COMMAND.COM: line 1: runaway regular expression /c ...

I'm not sure what to blame: this win32 build of mawk.exe, HX, or something else.

For comparison, here's the output from a DJGPP build of mawk.exe.

C:\BEN\BIN>mawk -f test.awk
test.awk: cd
C:\BEN\BIN

test.awk: mawk.exe "BEGIN{print \"hi\"}"
hi

> > The msvcrt.dll supplied with Windows XP seems to work - and I guess the
> > ones supplied with Win9x are also ok.
>
> Thanks! I tried the following MSVCRT.DLL and at first glance it appeared
> to work:

I didn't try them all, but I literally meant it when I said use old ReactOS's 0.3.14 version from 2012. I'm not sure others work as well.

I really should try some simple MAWK/Win32 tests under HX myself and report back. (Well, I don't use system() or popen() a lot there.)

BTW, you mentioned elsewhere some flaws (?) in DJGPP's build when you didn't explicitly close some files (i.e. FAT corruption). I believe the built-in GAWK lint (and/or debugger??) will hint at such things in advance.

N.B. I really don't see the point of using or preferring proprietary .DLLs from Win9x at all. In that case, just use DJGPP. However, most people already have several Windows licenses.

> C:\BEN\mawk\bin>mawk "BEGIN{print \"hi\"}"
> hi
>
> But on closer inspection, it appears to have trouble running itself.
>
> C:\BEN\mawk\bin>type test.awk
> function sys(str) {
> printf "test.awk: %s\n", str
> retval = system(str)
> print ""
> return retval
> }
>
> BEGIN {
> sys("cd")
>
> cmd = ARGV[0] " \"BEGIN{print \\\"hi\\\"}\""
> sys(cmd)
> exit 0
> }
> C:\BEN\mawk\bin>mawk -f test.awk
> test.awk: cd
> C:\BEN\MAWK\BIN
>
> test.awk: C:\ben\mawk\bin\mawk.exe "BEGIN{print \"hi\"}"
> C:\FreeDOS\BIN\COMMAND.COM: line 1: runaway regular expression /c
> ...
>
> I'm not sure what to blame: this win32 build of mawk.exe, HX, or something
> else.

It could be a RE library quirk. Realistically, I'd suspect HX, but it could be a Windows (which? XP? 10?) problem. Last I checked, MinGW stated an (untested??) minimum of Windows 10.

Specifically, DOS with all its different memory types and managers and environments makes things a bit confusing. There is less confusion on a modern OS where everything is all unified (a single memory API and default shell). You know, system() isn't really "portable". Alexei's SmallerC had some limitations and quirks trying to use system() in his compiler. By this I mean system() is maybe too simplistic and incomplete. Adding anything else (spawn? exec? fork?) makes a big mess for true portability.

If anything, I'd suggest to stick to simple file input and output with AWK, then use external tools with the shell to manipulate that data. Don't try to do it all mixed together.

Actually, REXX was specifically designed to do exactly what you want, pass data to and from environments without (much, if at all) use of the system shell. However, again, you have to be careful in DOS (enough free conventional memory? CWSDPMI support? needs different "extender" or DPMI host?). CWSDPMI (pure 32-bit DPMI) won't run DOS4GW binaries, for instance, and can clash with some OpenWatcom "extenders".

Seriously, I've played a lot with REXX lately, even in DOS, and it's easily as good as AWK. Howard Fosdick gives away his 2005 REXX book for free, too. I can point you to several builds of Regina (DJGPP, DOS4GW) and BRexx (BorlandC 386+ real mode, TurboC 8086, DJGPP / DPMI). Regina prefers ANSI REXX "ADDRESS SYSTEM 'dir' WITH INPUT STREAM 'in.txt' OUTPUT STEM dat." while BRexx (32-bit only, 16-bit's stack depth is too small) prefers "dir (stack" and "stack> sort". (Anything that isn't an internal REXX function or instruction is treated as an external command and shelled out.)

I just think some things on non-POSIX systems (like Windows) are never emulated properly, e.g. popen (hence the Win95 patches of that for AWK). In other words, dark corners exist, especially for Windows (non-*nix) builds. If some features don't work portably, don't rely on them.

> I'm not sure what to blame: this win32 build of mawk.exe, HX, or something
> else.

Here's the output that I got in MS-DOS 7.1 when I run your test program ( with msvcrt.dll from XP SP3 ):


test.awk: cd
C:\TEMP\bin

test.awk: c:\temp\bin\mawk.exe "BEGIN{print \"hi\"}"
hi


haven't tried the dll of your link yet...

EDIT: tried "your" dll, same result as above.

---
MS-DOS forever!

> Here's the output that I got in MS-DOS 7.1 when I run your test program (
> with msvcrt.dll from XP SP3 ):
>
>
> test.awk: cd
> C:\TEMP\bin
>
> test.awk: c:\temp\bin\mawk.exe "BEGIN{print \"hi\"}"
> hi
>
>
> haven't tried the dll of your link yet...
>
> EDIT: tried "your" dll, same result as above.

Curiouser and curiouser...

I reproduced the problem on MS-DOS 6.22. That eliminates FreeDOS.

I reproduced the problem in a Watcom win32 build of mawk. That eliminates mingw and MSVCRT.DLL.

I reproduced the problem using short sample C code built with Watcom. That eliminates mawk.

Note how the win32 version of test2.exe inserts extraneous arguments when it uses the system() C library call, but the DOS version doesn't.

https://bencollver.bitbucket.io/files/hxtest.zip

C:\hxtest\dos>test.exe foo bar
COMSPEC=C:\FreeDOS\BIN\COMMAND.COM
argc=3
argv[0]=C:\HXTEST\DOS\TEST.EXE
argv[1]=foo
argv[2]=bar

C:\hxtest\dos>test2.exe test.exe foo bar
COMSPEC=C:\FreeDOS\BIN\COMMAND.COM
argc=3
argv[0]=C:\HXTEST\DOS\TEST.EXE
argv[1]=foo
argv[2]=bar

C:\hxtest\win32>test.exe foo bar
DPMILD32: Module=hxguihlp.dll
DPMILD32: File not found error
COMSPEC=C:\FreeDOS\BIN\COMMAND.COM
argc=3
argv[0]=c:\hxtest\win32\test.exe
argv[1]=foo
argv[2]=bar

C:\hxtest\win32>test2.exe test.exe foo bar
DPMILD32: Module=hxguihlp.dll
DPMILD32: File not found error
DPMILD32: Module=hxguihlp.dll
DPMILD32: File not found error
COMSPEC=C:\FreeDOS\BIN\COMMAND.COM
argc=5
argv[0]=c:\hxtest\win32\test.exe
argv[1]=/c
argv[2]=test.exe
argv[3]=foo
argv[4]=bar

C:\hxtest>type test.c
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
    int i;
    char *str;
    str = getenv("COMSPEC");
    printf("COMSPEC=%s\n", str);
    printf("argc=%d\n", argc);
    for (i = 0; i < argc; i++) {
        printf("argv[%d]=%s\n", i, argv[i]);
    }
    exit(0);
}

C:\hxtest>type test2.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[]) {
    int i, result;
    char buf[256];
    if (argc < 1) {
        printf("Usage: test2.exe args...\n");
        exit(1);
    }
    strcpy(buf, argv[1]);
    for (i = 2; i < argc; i++) {
        strcat(buf, " ");
        strcat(buf, argv[i]);
    }
    result = system(buf);
    exit(result);
}

> I reproduced the problem in a Watcom win32 build of mawk. That eliminates
> mingw and MSVCRT.DLL.

Yes, BTW, that is a nice advantage of OpenWatcom. (MSVCRT is a "known DLL", cannot be overridden, and frankly shouldn't be relied upon by user programs. The fact that there are multiple newer version MSVCRT runtime .DLLs is also an annoyance. You can usually link it statically, but most people don't.)

> Note how the win32 version of test2.exe inserts extraneous arguments when
> it uses the system() C library call, but the DOS version doesn't.

What, the extra "/c"? Although I may be misunderstanding, as you know "%COMSPEC% /c my.bat" is often how most programs shell out. But you don't need to explicitly call the shell if not using a .BAT file.

In other words, some compilers treat system("runme") as system("command /c runme"). They don't try to be clever and only do it for .BATs, they do it for everything. (DJGPP is a little smarter than most here.) In particular, Turbo Pascal's Exec function (in most, but not all, TP-compatible compilers) explicitly needs something like Exec(getenv('COMSPEC'),' /c ' + myEXEname);

> What, the extra "/c"? Although I may be misunderstanding, as you know
> "%COMSPEC% /c my.bat" is often how most programs shell out. But you don't
> need to explicitly call the shell if not using a .BAT file.
>
> In other words, some compilers treat system("runme") as system("command /c
> runme"). They don't try to be clever and only do it for .BATs, they do it
> for everything. (DJGPP is a little smarter than most here.) In particular,
> Turbo Pascal's Exec function (in most, but not all, TP-compatible
> compilers) explicitly needs something like Exec(getenv('COMSPEC'),' /c ' +
> myEXEname);

I ran this again, this time in the WinPE command prompt, with the following results.

C:\hxtest\win32>test.exe foo bar >test.txt

C:\hxtest\win32>type test.txt
COMSPEC=X:\i386\system32\cmd.exe
argc=3
argv[0]=C:\hxtest\win32\test.exe
argv[1]=foo
argv[2]=bar

C:\hxtest\win32>test2.exe test.exe foo bar ">test2.txt"

C:\hxtest\win32>type test2.txt
COMSPEC=X:\i386\system32\cmd.exe
argc=3
argv[0]=C:\hxtest\win32\test.exe
argv[1]=foo
argv[2]=bar

Interesting that argc=3 when running test2.exe on Windows, but argc=5 when running test2.exe on HXRT. This leads me to believe that the blame rests somewhere other than the OpenWatcom compiler.

> Interesting that argc=3 when running test2.exe on Windows, but argc=5 when
> running test2.exe on HXRT. This leads me to believe that the blame rests
> somewhere other than the OpenWatcom compiler.

For me it works as expected, never got argc=5. Perhaps you're using a very old HXRT?

These are the binaries and sources that I used:https://github.com/Baron-von-Riedesel/Testcases/blob/main/test2.zip

Note that for the DOS binaries I used spawn-hx.obj - this was formerly spawn386.obj ( created from a modified spawn386.asm ), contained in hxow.lib.

---
MS-DOS forever!

> For me it works as expected, never got argc=5. Perhaps you're using a very
> old HXRT?
>
> These are the binaries and sources that I
> used:https://github.com/Baron-von-Riedesel/Testcases/blob/main/test2.zip

I tried using your binaries and HXRT release 2.21, which i believe is the latest release. I am still getting argc=5 from the win32 version and argc=3 from the dos version.

C:\test2>test2w.exe testw.exe foo bar
COMSPEC=C:\FREEDOS\BIN\COMMAND.COM
argc=5
argv[0]=c:\test2\testw.exe
argv[1]=/c
argv[2]=testw.exe
argv[3]=foo
argv[4]=bar

C:\test2>type \hx.bat
set PATH=%PATH%;C:\BEN\HX\BIN
set DPMILDR=128
set HDPMI=32
hxldr32

C:\test2>echo %OS_NAME%
FreeDOS

C:\test2>echo %OS_VERSION%
1.3


By the way, i noticed that HXRT.TXT says:

> * to allow nested execution of Win32 applications it may be necessary
> to set environment variables HDPMI=32 *and* DPMILDR=8. For more details
> about this issue please read Doc\Compat.txt

I tried setting DPMILDR=8 but that didn't seem to make a difference.

Do you have other ideas for me to try? Thanks for your time!

> I tried using your binaries and HXRT release 2.21, which i believe is the
> latest release. I am still getting argc=5 from the win32 version and
> argc=3 from the dos version.

HXRT v2.21 is fine.

> C:\test2>echo %OS_NAME%
> FreeDOS
>
> C:\test2>echo %OS_VERSION%
> 1.3

I used MS-DOS 7.1 ( Win98SE ) with JemmEx ( Jemm should be irrelevant ).

> I tried setting DPMILDR=8 but that didn't seem to make a difference.

environment variables DPMILDR and HDPMI don't matter here, since the Win32 binaries are linked relocatable.

---
MS-DOS forever!

> I used MS-DOS 7.1 ( Win98SE ) with JemmEx ( Jemm should be irrelevant ).

I tried your binaries in MS-DOS 7.1 and they gave the expected argc=3, where they gave argc=5 on FreeDOS 1.3 and MS-DOS 6.22.

> I tried your binaries in MS-DOS 7.1 and they gave the expected argc=3,
> where they gave argc=5 on FreeDOS 1.3 and MS-DOS 6.22.

Ok, got the same result with MS-DOS 6.22.

I guess the problem is related to long filenames. Open Watcom encloses the contents of the COMSPEC variable in double quotes, appends the cmd tail ( without quotes ) and finally calls CreateProcess() with this string. The HX emulation code more or less copies this string to the CMDLINE environment variable. MS-DOS 7.1 can handle this, but obviously MS-DOS 6.22 gets confused by the quotes.

---
MS-DOS forever!

> Ok, got the same result with MS-DOS 6.22.
>
> I guess the problem is related to long filenames. Open Watcom encloses the
> contents of the COMSPEC variable in double quotes, appends the cmd tail (
> without quotes ) and finally calls CreateProcess() with this string. The HX
> emulation code more or less copies this string to the CMDLINE environment
> variable. MS-DOS 7.1 can handle this, but obviously MS-DOS 6.22 gets
> confused by the quotes.

Thanks for your analysis!

I installed 4dos 8.00 on FreeDOS 1.3. Now when i run your test2w.exe example, it gives the expected argc=3.

I posted mawk134-20240622b.zip

This includes mawkx.exe, an OpenWatcom win32 build that will run on HXRT.

> I posted mawk134-20240622b.zip
>
> This includes mawkx.exe, an OpenWatcom win32 build that will run on HXRT.

Keep in mind several things (from random experience):

In some programs (e.g. my fork of paq8o8), I noticed that OpenWatcom's malloc was much less efficient than DJGPP's.

UPX'd DJGPP .EXEs run much slower under (e.g. XP) NTVDM.

You can usually use WDOSX's "stubit" on OpenWatcom's Win32/PE files to run under DOS natively (since the PE has relocs, which most don't anymore). It will also compress it. However, there is no virtual memory swapping, and you can't undo the changes, so you have to keep a backup of the original. Also, if you shell out from a normal DJGPP (CWSDPMI) program, running a WDOSX program won't work. (Well, probably if you use HDPMI32 instead.)

CWSTUB.EXE is public domain with sources and much smaller than DOS4GW.EXE and less limited in memory and should enable SSE for you automatically, but if your app needs 37 MB of RAM and you only have 32 MB, it will swap the ENTIRE 37 MB out (unlike CWSDPMI, which will only swap the small part needed). However, at least Causeway seems to play nice with DJGPP.

Both WDOSX and CWSTUB have limited support for .DLL files.

> Keep in mind several things (from random experience):

Thank you for sharing from your experiences. I will try to be careful of those dark corners.

> > In theory, having only one binary that runs on two (incompatible) OSes
> > would be nice (even if relying on external API emulation).
>
> You mean like APE and Cosmopolitan?
>
> https://justine.lol/ape.html
> https://justine.lol/cosmopolitan/

Believe it or not, her APE build of AWK doesn't run on Windows 7 for me. (Admittedly it's an unsupported and old OS.)

> I don't think it is better. I would almost always prefer DJGPP. I meant "if
> you must use HX, stick to OpenWatcom".

> In 2008, Japheth recompiled the FTE text editor with HX. There he implied a
> "true flat memory model" was better somehow. (For DJGPP, I think for
> stability they restricted access to low DOS memory.) He also (rightly) said
> DJGPP was a bit bloated, having never been optimized for size, so his way
> gave a slimmer binary.

Recently I played with FTE a bit (again). And it's true that binaries created with Open Watcom may be pretty slow in memory management, at least if the DOS32 variants ( DOS4/GW and compatibles ) are used. As for FTE, the difference isn't really noticable if small to medium sized files are loaded, but loading a 100 MB file ( with 1.5 million lines ) took about 90 sec for FTE created as OW DOS32 binary, but just 2 secs for a variant linked statically with HX's Win32 emulation.

Here's the link so you can do your own tests:

https://github.com/Baron-von-Riedesel/FTE/releases

---
MS-DOS forever!

> I installed 4dos 8.00 on FreeDOS 1.3. Now when i run your test2w.exe
> example, it gives the expected argc=3.

The HX kernel32 emulation has been fixed - now cmdline handling will supply the correct results for non-LFN aware MS-DOS versions ( < v7 ).

---
MS-DOS forever!