Back to home page

DOS ain't dead

Forum index page

Log in | Register

Search:  
Back to index page
Thread view  Board view
kerravon

Ligao, Free World North,
07.01.2023, 03:56
 

clean dos extender executable format (Developers)

Post reply

I have a question that isn't quite resolved.

One of the DOS extender executable formats, can't
remember the name, but it allowed a 32-bit INT 21H
to be called.

I was almost happy to support that format, but I
saw that they both expected segment registers to
be set to particular values on entry, and they
manipulated segment registers on certain 32-bit
INT 21H calls. exec() I think.

This format was both available commercially, and
someone had written a freeware version.

But they came pretty close to what I consider to
be clean.

I don't remember what the executable format was.
Presumably not a.out like I use for my 32-bit
executables. Although I guess I could use PE if
I wanted to (ie even if I only do INT 21H, rather
than calling DLLs).

I might be willing to have an MSDOS stub that
invokes a DOS extender, so long as I can ignore
that stub on an actual 32-bit system (ie PDOS/386).

And so long as the stub itself doesn't contain
any functionality other than invoking the external
loader.

And it would be good if the stub cleanly reported
an issue if it wasn't able to find the external
loader or some other issue. My a.out executables
don't do that. They get treated as .com files and
hang or whatever. So this would be a step up from
what I am currently doing.

I like a.out because of its simplicity. I only have
simple needs.

So my question is - is there some room for compromise
here? A clean 32-bit executable format.

Not one that messes around creating 16-bit calls either.
Any such thing must be external to the executable.

I don't care at all what happens outside of my
executable. I'm just looking for executables that are
technically clean and unkludged.
kerravon

Ligao, Free World North,
07.01.2023, 04:18

@ kerravon
 

clean dos extender executable format

Post reply

> I might be willing to have an MSDOS stub that
> invokes a DOS extender, so long as I can ignore
> that stub on an actual 32-bit system (ie PDOS/386).
Actually, why can't it be done the same way that
HX manages to load Windows executables with no
intrusive code in the executables?

That seems to be the ultimate cleanliness.
DosWorld

08.01.2023, 15:51
(edited by DosWorld, 08.01.2023, 16:31)

@ kerravon
 

clean dos extender executable format

Post reply

> One of the DOS extender executable formats, can't
> remember the name, but it allowed a 32-bit INT 21H
> to be called.
Many DOS-extenders allow it.

> So my question is - is there some room for compromise
> here? A clean 32-bit executable format.
>
> Not one that messes around creating 16-bit calls either.
> Any such thing must be external to the executable.
IMHO, available problem with executable/object formats.
Let me describe it.

obj/omf. Support 16/32 bit. But it too difficult. My main complaint: the authors of the documentation could not describe in simple words. Perhaps this is the problem of technical writers of the old days. They invented terms and logical constructions that can be greatly simplified. So, here is i meet a wall.

rdf. Support 16/32 bit. But here is no place for dll import. (available record for dll, but i am never meet usage example)

aout. Support 32 bit. I am dislike 16-bit (8-byte import/export names), lack of segmented memory model for 16-bit. Also, we have 2 type of aout: linux and bsd.

Also, in object formats, will be great have a "module initialization" and "module deinitialization" code blocks (like in pascal units).

pe/lx/le. Support 32 bit only. over 100500 fields (non relevant for DOS) which make linker (and loader) not so simple.

(i am doest not touch elf and obj/coff, but i don't think there's anything better there)

So... IMHO, all choices is bad. We live in legacy world. We need Hercules to clear these Augean stables.

So far, producing rdf => pe/hx (or mz/exe) looks the most promising, for me.
(i am don't care about win32 + dll import. It's more important not to lose the ability to link mz/exe).

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.
kerravon

Ligao, Free World North,
08.01.2023, 18:15

@ DosWorld
 

clean dos extender executable format

Post reply

> aout. Support 32 bit. I am dislike 16-bit (8-byte import/export names),
> lack of segmented memory model for 16-bit. Also, we have 2 type of aout:
> linux and bsd.
as86/ar/ld86 from here:

https://github.com/robertapengelly?tab=repositories

use 16-bit a.out. I'm not aware of any
8-byte limitation on externals.

The lack of segmentation (not supporting the
"seg" keyword in assembler code) is something
I am happy to work around. I changed my
assembler code that was using that, and will
make sure the C compiler doesn't generate that.

I think that is the proper (clean) technical solution.

> pe/lx/le. Support 32 bit only. over 100500 fields (non relevant for DOS)
> which make linker (and loader) not so simple.
Yeah. I'm after something simple too. And a.out
seems to be simple. There's no "fat" at all in it
that I can see.

> So... IMHO, all choices is bad. We live in legacy world. We need Hercules
> to clear these Augean stables.
>
> So far, producing rdf => pe/hx (or mz/exe) looks the most promising, for
> me.
> (i am don't care about win32 + dll import. It's more important not to lose
> the ability to link mz/exe).
You're not happy to have independent 16-bit and
32-bit links?

Maybe we just need a.out with an MSDOS stub.

Which in fact was what EMX 0.9d does, isn't it?

But change the stub to the same as what PE uses.

And load it the same way that HX does. I assume
HX is intercepting the load and looking for the
"PE" signature. Maybe we need an "AO" signature
for a.out?

BFN. Paul.
DosWorld

08.01.2023, 19:09

@ kerravon
 

clean dos extender executable format

Post reply

> The lack of segmentation (not supporting the
> "seg" keyword in assembler code) is something
> I am happy to work around. I changed my
> assembler code that was using that, and will
> make sure the C compiler doesn't generate that.
aout, by design, have 1 data seg and 1 code seg.
Here is no any type of relocation for segments, so aout is applicable for
tiny or small memory models only.
imho, this is not enough for 16-bit DOS.

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.
tkchia

Homepage

08.01.2023, 19:11

@ DosWorld
 

clean dos extender executable format

Post reply

Hello DosWorld,

> aout, by design, have 1 data seg and 1 code seg.
> Here is no any type of relocation for segments, so aout is applicable for
> tiny or small memory models only.
> imho, this is not enough for 16-bit DOS.
My understanding is that kerravon's actual a.out will be a 32-bit protected mode program. (So any 16-bit code will only be used to start up a 32-bit environment for the a.out to run in.)

Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"
kerravon

Ligao, Free World North,
08.01.2023, 19:36

@ tkchia
 

clean dos extender executable format

Post reply

> Hello DosWorld,
>
> > aout, by design, have 1 data seg and 1 code seg.
> > Here is no any type of relocation for segments, so aout is applicable
> for
> > tiny or small memory models only.
> > imho, this is not enough for 16-bit DOS.
>
> My understanding is that kerravon's actual a.out will be a 32-bit protected
> mode program. (So any 16-bit code will only be used to start up a 32-bit
> environment for the a.out to run in.)
These are separate issues I think.

A proposal to use a.out for 32-bit, regardless of
how the 16-bit stub is produced.

And a proposal to use a.out for 16-bit. Actually I
was surprised Robert chose a.out for 16-bit, and I
wasn't aware that was actually a thing, or possible.

I believe he chose it because OMF wasn't well
documented.

Seems a bit odd to me, as this was an existing
solution, but maybe the shift to 16-bit a.out is
a good thing, and could be part of the rationalization
of DOS (given Microsoft are no longer involved, it
may as well be us that set any standards).
tom

Homepage

Germany (West),
09.01.2023, 18:49

@ kerravon
 

clean dos extender executable format

Post reply

> given Microsoft are no longer involved, it
> may as well be us that set any standards.
note the standardS.

even when Microsoft was involved, you were absolutely able to set your own 'standard'.

I'm not 100% sure, but there never was any 'standard' in DOS extender executable format beyond "the MZ portion of it will be executed and takes care to execute the 32-bit part". so there was neither a standard nor a standard needed.

just implement a working toolchain of compiler, linker, whatever, and have people use your toolchain.

there are multiple implementations of this available with source code.

why do you refuse to learn 'things to be learned' from others, indicating you have no clue at all about dos extenders in general or specifically?
kerravon

Ligao, Free World North,
09.01.2023, 19:51

@ tom
 

clean dos extender executable format

Post reply

> just implement a working toolchain of compiler, linker, whatever, and have
> people use your toolchain.
I already have that, and it relies on HX or equivalent.

> why do you refuse to learn 'things to be learned' from others, indicating
> you have no clue at all about dos extenders in general or specifically?
I thought I was learning from others right now.

Regardless, just because I already have something
working, relying on HX, doesn't mean that that is
the cleanest implementation.

All I did was copy Microsoft's Win32.

I'd like to independently derive something from
first principles, and using the benefit of hindsight.
If that is actually possible.
DosWorld

09.01.2023, 00:24

@ tkchia
 

clean dos extender executable format

Post reply

> My understanding is that kerravon's actual a.out will be a 32-bit protected
> mode program. (So any 16-bit code will only be used to start up a 32-bit
> environment for the a.out to run in.)
I am talk from historic point of view.
Initially, aout was a 16-bit file format (like Unix System V age) and was extended to 32-bit later. And more later, was split to Linux-aout and BSD-aout.

So, all 3 type of aout have no segment relocations. BSD-aout have more flexible relocation format (than in other aout). BSD-aout is good for DPMI, but for 16-bit... not so good. Not a "silver bullet". IMHO.

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.
kerravon

Ligao, Free World North,
09.01.2023, 02:32

@ DosWorld
 

clean dos extender executable format

Post reply

> I am talk from historic point of view.
> Initially, aout was a 16-bit file format (like Unix System V age) and was
> extended to 32-bit later. And more later, was split to Linux-aout and
> BSD-aout.
>
> So, all 3 type of aout have no segment relocations.
For what reason do you need "segment relocations"?

It just occurred to me - we might be talking cross-purposes.

Robert used a.out for object code format, but he still
necessarily produces standard MZ executables.

There was indeed an issue using a.out as object format
instead of OMF, but it wasn't a big deal.

Assembler code couldn't use the "seg" keyword, and
instead it was necessary to define a "dd" to create
an external reference, and that could be used to
extract the segment information.

Which may well be a nicer thing to do, to simplify
things.
DosWorld

09.01.2023, 03:22
(edited by DosWorld, 09.01.2023, 03:33)

@ kerravon
 

clean dos extender executable format

Post reply

> Robert used a.out for object code format, but he still
> necessarily produces standard MZ executables.
Sorry, i am not check Robert's work. I'll check as86.

(Right now, i'm sure he produce MZ in tiny or small memory model.
But need check - may be Robert invent something and escape from this restriction?)

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.
kerravon

Ligao, Free World North,
09.01.2023, 03:43

@ DosWorld
 

clean dos extender executable format

Post reply

> > Robert used a.out for object code format, but he still
> > necessarily produces standard MZ executables.
>
> Sorry, i am not check Robert's work. I'll check as86.
>
> (Right now, i'm sure he produce MZ in tiny or small memory model.
> But need check - may be Robert invent something and escape from this
> restriction?)
I've only used small (final) executables, but the assembler
code is all using far pointers (large memory model).

Perhaps you can explain why you think a restriction
exists while using a.out as an object code format.
kerravon

Ligao, Free World North,
08.01.2023, 19:31

@ DosWorld
 

clean dos extender executable format

Post reply

> > The lack of segmentation (not supporting the
> > "seg" keyword in assembler code) is something
> > I am happy to work around. I changed my
> > assembler code that was using that, and will
> > make sure the C compiler doesn't generate that.
>
> aout, by design, have 1 data seg and 1 code seg.
> Here is no any type of relocation for segments, so aout is applicable for
> tiny or small memory models only.
> imho, this is not enough for 16-bit DOS.
I was also wondering if this would be the case,
but it does not appear to be the case.

Robert has managed to get a large memory model
program of mine to work.

Admittedly the executable is small, so maybe
there is some fundamental problem I haven't
discovered yet.

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/makefile.wca

https://github.com/robertapengelly?tab=repositories
tkchia

Homepage

08.01.2023, 18:10

@ kerravon
 

clean dos extender executable format

Post reply

Hello kerravon,

> And it would be good if the stub cleanly reported
> an issue if it wasn't able to find the external
> loader or some other issue. My a.out executables
> don't do that. They get treated as .com files and
> hang or whatever. So this would be a step up from
> what I am currently doing.
Well, yes — Windows and OS/2 NE, LE, LX, PE, etc. files follow a rather basic sort of "fat binary" format:

(1) There is a classical MS-DOS MZ program stub tacked on at the front;

(2) and the MZ header has an extra offset field which points to the "actual" NE or PE (etc.) payload that Windows or OS/2 (etc.) can immediately run. (This field is usually named e_lfanew.)

The general scheme is kind of described in one of Microsoft's old (~ 1999) Developer's Notes: https://jeffpar.github.io/kbarchive/kb/065/Q65122/ .

You can probably apply the same scheme, or a similar one, to whatever protected mode (?) executable format you wish to stubify. Tack a classical MS-DOS program in front of it, and add an offset pointer to point to the a.out payload, and call it a day.

Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"
kerravon

Ligao, Free World North,
08.01.2023, 18:37

@ tkchia
 

clean dos extender executable format

Post reply

> You can probably apply the same scheme, or a similar one, to whatever
> protected mode (?) executable format you wish to stubify. Tack a classical
> MS-DOS program in front of it, and add an offset pointer to point to the
> a.out payload, and call it a day.
Sounds good to me!

So at offset x'3c' is the e_lfanew, which is x'80'
in my case:

C:\devel\pdos\src>hexdump \winpath\hexdump.exe 0 200
000000 4D5A9000 03000000 04000000 FFFF0000 MZ..............
000010 B8000000 00000000 40000000 00000000 ........@.......
000020 00000000 00000000 00000000 00000000 ................
000030 00000000 00000000 00000000 80000000 ................
000040 0E1FB409 BA1000CD 21B001B4 4CCD2100 ........!...L.!.
000050 496E7374 616C6C20 4858206F 72207570 Install HX or up
000060 67726164 6520746F 2050444F 532F3338 grade to PDOS/38
000070 36206F72 2057696E 65206574 630D0A24 6 or Wine etc..$

And that has the PE + NUL + NUL:

000080 50450000 4C010400 47286D61 00000000 PE..L...G(ma....
000090 00000000 E0000E02 0B010238 00040000 ...........8....
0000A0 00060000 00000000 00100000 00100000 ................
0000B0 00000000 00004000 00100000 00020000 ......@.........

It would probably be good for doing hexdumps if
my a.out was aligned on a 16-byte boundary.

So what do you think of making that say:

AO + NUL + NUL + 12 * NUL?

followed by the normal 32-bit a.out which does
INT 21H calls?

And then I will need to write an actual DOS extender
which does DPMI calls, right?

And in hindsight, does this look neater than existing
DOS extender formats?

BFN. Paul.
tkchia

Homepage

08.01.2023, 19:05

@ kerravon
 

clean dos extender executable format

Post reply

Hello kerravon,

> It would probably be good for doing hexdumps if
> my a.out was aligned on a 16-byte boundary.
> So what do you think of making that say:
> AO + NUL + NUL + 12 * NUL?
> followed by the normal 32-bit a.out which does
> INT 21H calls?
You could do that — or, you could just put the a.out there without any additional tagging. I believe a.out has its own magic number(s) too.

> And then I will need to write an actual DOS extender
> which does DPMI calls, right?
Well, or you could hack an existing DOS extender to recognize your a.out file format, and massage the argv[] and envp[] information into a form that it can use. Other than HX, extenders such as CauseWay, DOS/32A, and PMODE/W are now also open source.

(I happen to be tinkering with the CauseWay extender lately, for use with the gcc-ia16 toolchain. This is mainly because it supports running 16-bit protected mode programs (!). CauseWay is public domain too.)

> And in hindsight, does this look neater than existing
> DOS extender formats?
That I will leave to you to judge. :flower:

Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"
kerravon

Ligao, Free World North,
08.01.2023, 19:50

@ tkchia
 

clean dos extender executable format

Post reply

> You could do that — or, you could just put the a.out there without
> any additional tagging. I believe a.out has its own magic number(s) too.
Good point.

> > And then I will need to write an actual DOS extender
> > which does DPMI calls, right?
>
> Well, or you could hack an existing DOS extender to recognize your a.out
> file format, and massage the argv[] and envp[]
> information into a form that it can use.
Currently I call these functions:

PosGetCommandLine();
PosGetEnvBlock();

which are front-ends to INT 21H, AH=F6H, AL=3FH
and INT 21H, AH=F6H, AL=3BH respectively.

Any reason why I shouldn't do that?

> Other than HX, extenders such as
> CauseWay, DOS/32A, and PMODE/W are now also open source.
Thanks for that!

> (I happen to be tinkering with the CauseWay extender lately, for use with
> the gcc-ia16 toolchain. This is mainly because it supports
> running 16-bit protected mode programs (!). CauseWay is public domain
> too.)
I checked, and it is indeed - thanks for that!

https://github.com/amindlost/cw

Next question.

With PDOS-generic, I move the INT 21H calls (if
someone chooses such an implementation) outside
of the executable itself. And also provide a C
runtime library.

Would that be more appropriate?

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/generic/__os.h

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/generic/makecomm.w32

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/pgastart.c

(it would still be an MZ stub plus a.out)

ie I guess the question is - is it really necessary
to do a 32-bit INT 21H in order to be considered
legitimate?

Like, let's say it's 1986 or whenever the Compaq
was released - what would the goal be, in hindsight?
tom

Homepage

Germany (West),
08.01.2023, 20:35

@ kerravon
 

clean dos extender executable format

Post reply

> ie I guess the question is - is it really necessary
> to do a 32-bit INT 21H in order to be considered
> legitimate?
NO
kerravon

Ligao, Free World North,
08.01.2023, 20:46

@ tom
 

clean dos extender executable format

Post reply

> > ie I guess the question is - is it really necessary
> > to do a 32-bit INT 21H in order to be considered
> > legitimate?
>
> NO
Ok, then what alternatives exist in 1986?

Happy to use hindsight, so you may wish to say "use Windows PE format".

I would say the fundamental thing is that the MSDOS filesystem code is stable, so any solution needs to eventually call 16-bit MSDOS.

DPMI hasn't been invented yet, but we can invent it exactly as-is, or have a stop-gap measure. So long as the stop-gap measure is divorced from the executable itself, it can be changed with impunity.

People still want to run 16-bit MSDOS software, and that isn't changing.

I'm not a fan of DLLs, so I'd prefer not to go on that path.

The C language is taking off, so I'd like to see a C library provided to executables.

What sensible suggestions exist in 1986? What about 1990? No need to be first to market either. Possibly something that academia would approve of.
kerravon

Ligao, Free World North,
08.01.2023, 22:51

@ kerravon
 

clean dos extender executable format

Post reply

> What sensible suggestions exist in 1986? What about 1990? No need to be
> first to market either. Possibly something that academia would approve of.
Actually, 1986 would just be when the implementation
is done.

Academia could have predicted (in 1981 or whatever)
that there would be a 32-bit CPU coming out in due
course and this question would need to be answered.

They could have answered it, and maybe even written
the code, including on an emulator, so that as soon
as the exact instruction set was known they could
modify their compiler and have executables ready
a week after the Compaq hit the market.
tkchia

Homepage

09.01.2023, 00:59

@ kerravon
 

clean dos extender executable format

Post reply

Hello kerravon,

> Currently I call these functions:
> PosGetCommandLine();
> PosGetEnvBlock();
> which are front-ends to INT 21H, AH=F6H, AL=3FH
> and INT 21H, AH=F6H, AL=3BH respectively.
> Any reason why I shouldn't do that?
Huh? As I said, CauseWay, DOS/32A, PMODE/W, etc. are open source. This means you can freely read their source code or even (!) modify it. Use this power.

People say, "capitalism, it works!" I say, "use the Force, read the Source".

Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"
DosWorld

09.01.2023, 03:13
(edited by DosWorld, 09.01.2023, 03:26)

@ tkchia
 

clean dos extender executable format

Post reply

> Huh? As I said, CauseWay, DOS/32A, PMODE/W, etc. are open source.
PMODE/W is bad example due to license.
Nobody can continue development (only as part of other software): :-(

-----------------------------
PMODE/W License

Due to the fact that PMODE/W is no longer supported it is now free for use in
commercial and non-commercial applications, provided said applications are not
themselves DOS extenders. To put it bluntly, don't go ripping off our DOS
extender to make your own.

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.
tkchia

Homepage

09.01.2023, 10:49

@ DosWorld
 

clean dos extender executable format

Post reply

Hello DosWorld,

> PMODE/W is bad example due to license.
> Nobody can continue development (only as part of other software): :-(
Argh! I stand corrected, and PMODE/W is not really quite "open source" in that case. Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"
tkchia

Homepage

09.01.2023, 11:11

@ kerravon
 

clean dos extender executable format

Post reply

Hello kerravon,

> which are front-ends to INT 21H, AH=F6H, AL=3FH
> and INT 21H, AH=F6H, AL=3BH respectively.
> Any reason why I shouldn't do that?
Some considerations which you will almost certainly need to deal with at some point in time:

(1)
You probably want to decide early on how your a.out application is going to receive the argv[] and envp[] information. This will form part of the ABI "contract" between the OS (+ DOS extender) and applications, and you probably want the ABI to be stable.

The Linux IA-32 ABI places argv[] and envp[] on the stack at program startup, but if you prefer to create extra syscalls to do that, that is possible too. You also need to decide whether to split the command line arguments into separate words (i.e. argv[]) at startup, and how (!) — or to leave the task to the application.

Whichever interface you decide on, try to stick to it.

(2)
The MZ loader stub will likely be needed to run your a.out program(s) precisely in those situations where the OS does not already understand your new proposed syscalls. This means you will need to implement the new "syscalls" inside the DOS extender, for use in your protected mode app.

(CauseWay and DOS/4G both already implement an additional bunch of functions under int 0x21, ah = 0xff.)

Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"
glennmcc

Homepage E-mail

North Jackson, Ohio (USA),
08.01.2023, 19:10

@ kerravon
 

clean dos extender executable format

Post reply

> And then I will need to write an actual DOS extender
> which does DPMI calls, right?
>
> And in hindsight, does this look neater than existing
> DOS extender formats?
>
> BFN. Paul.
IMO, what you propose is (as with your previous suggestions),
yet another a case of....

http://glennmcc.org/images/reinvented-wheel.jpg

;-)

---
--
http://glennmcc.org/
kerravon

Ligao, Free World North,
08.01.2023, 19:26

@ kerravon
 

clean dos extender executable format

Post reply

> One of the DOS extender executable formats, can't
> remember the name, but it allowed a 32-bit INT 21H
> to be called.
DOS32A is what I was thinking of:

https://web.archive.org/web/20210726190857/https://dos32a.narechk.net/index_en.html

https://github.com/amindlost/dos32a
CandyMan

09.01.2023, 19:23

@ kerravon
 

clean dos extender executable format

Post reply

Here is an example of a simple use of the D3X dos extender and a modified version of FASM. Maybe that'll help you.

https://board.flatassembler.net/topic.php?t=14416
Back to index page
Thread view  Board view
22049 Postings in 2034 Threads, 396 registered users, 158 users online (0 registered, 158 guests)
DOS ain't dead | Admin contact
RSS Feed
powered by my little forum