DOS ain't dead - .DXE .DLL .SO (etc.)

Rugxulo

Usono,
14.12.2019, 14:00

.DXE .DLL .SO (etc.) (Developers)

(I'm splitting this off into a separate topic since it's unrelated to MSA2 and TP's .OVL files. Also, I don't directly have any new, useful comments on .OVL yet anyways.)

>> I know Go32v2 supports DXE1, but I've never used it much there.
>> In fact, I never used it much (except Matt Mahoney's old
>> C++/ASM code, paq8f, ten years ago!).
>> DXE3 is better but rarely used (e.g. Hexen2, whose code I never
>> studied closely).
>
> DXE is slightly different since afaik DXE can be created using
> separate generation. IOW it can be used to create plugins for
> already existing (compiled) programs.

I don't have a lot of direct experience with .DXE. I'm not sure it's
been used much by anyone. There were some very few builds of DJGPP
things using it, but even that was relatively rare. It's DJGPP 2.04
(/beta/ from 2003) and /current/ 2.05 from 2015 with improved DXE3.
Daniel Borca was responsible for DJELF with actual .so support,
which he considered superior. But even that I never heavily tested.
(I halfway meant to read an ebook by Ulrich Drepper on such stuff
but never found the motivation.) Oh, I think Juan's port of Lua 5.2.2
(/v2tk/ from 2013) used DXE3, also. So maybe that would prove
semi-interesting.

Just for comparison, the obvious alternatives would be OpenWatcom + HX
or maybe Causeway. FreeBASIC has some rudimentary DLL support for DOS
since 1.06 (almost a year ago), apparently, but I've not tried it.
Quite confusing stuff!

Laaca

Czech republic,
15.12.2019, 22:50

@ Rugxulo

.DXE .DLL .SO (etc.)

Post reply

Borland pascal 7.0 can use both - overlays in realmode and DLL libraries in DOS 16-bit protected mode. (and for windows, of course).
The DLL stuff is described quite well in the help system in IDE.

The bad thing in DLLs in DOS is that they are a little bit usesless. What advantage give it against static linking in the single-task OS?

However the good thing ih the BP way of the DLL implementation is that it is quite straitforward and not much more complicated ovet normal static-linked unit.

The DXE1 modules in the FPC has severe limitation that it has almost no connection with other parts of the program.

The EXE sees the inside of the DXE but the DXE sees nothing from EXE.
The practical impact is the the DXE modules can not use any procedures (or functions) from any system or user unit. Nothing from unit CRT, nothing from DOS, even nothing from unit SYSTEM.
So it is an extreme limitation for using this stuff.
The only way how to bypass this situation is using the procedural variables.

In the DXE can be something like that:
var Alloc:procedure(size:longint);
var DrawLine:procedure(x1,y1,x2,y2:longint);

And in the EXE you must initialize these variables:
DXE.Alloc:=@GetMem;
DXE.DrawLine:=@Line;

...and so on.

So it is so paintful that nobody uses it.
Maybe in theory could be possible the implement into FPC the DXE3 handling which should not suffer by this limitation but it would need some DOS-and-pascal-megaguru...

---
DOS-u-akbar!

Rugxulo

Usono,
16.12.2019, 20:04

@ Laaca

.DXE .DLL .SO (etc.)

Post reply

> The bad thing in DLLs in DOS is that they are a little bit usesless. What
> advantage give it against static linking in the single-task OS?

It would only be for replaceable (newer/alternate/faster/bugfixed/enhanced) code, maybe (un)loaded at runtime. So, if anything, I would think (oft-maligned) segmentation would help with this kind of thing! (Not for DJGPP, but you know what I mean, in 16-bit DOS.) Recompiling and redistributing an entire binary program (of multiple MB) just to change a few bytes is silly. Hex editing/byte patching/binary diffs are much more brittle but also unappreciated.

> The DXE1 modules in the FPC has severe limitation that it has almost no
> connection with other parts of the program.

It's still good for ("simple"?) leaf functions or even assembly (see paq8f with MMX or SSE2 speedups), especially if they will change later or target newer instruction sets (SSE 4.2?). But my patch made that optional (fallback to static code) so you could still use the .EXE without the external .DXEs. I think that's a better way and doesn't negate the stability and independence of static linking.

> The EXE sees the inside of the DXE but the DXE sees nothing from EXE.
> The practical impact is the the DXE modules can not use any procedures (or
> functions) from any system or user unit. Nothing from unit CRT, nothing
> from DOS, even nothing from unit SYSTEM.
> So it is an extreme limitation for using this stuff.
>
> So it is so painful that nobody uses it.

I agree it's limited, that's why I mentioned newer DXE3. (Remember that DJGPP 2.04 was never finalized beyond /beta/ since 2003, so barely kinda sorta 2.03p2 from 2001 was still official since it was /current/, even if you could still use both. But 2.05 in 2015 improved all of that, thankfully.)

> Maybe in theory could be possible the implement into FPC the DXE3 handling
> which should not suffer by this limitation but it would need some
> DOS-and-pascal-megaguru...

DXE3 is better, yes, but most people don't really understand how to use it or don't find it worth the hassle. Obviously I'm not quite smart enough to do everything myself. Still, I'm vaguely interested.

Everybody else is obsessed with .DLLs (and .so). It's considered old-fashioned and flawed to never use them. But clearly some compilers do without anyways, even on Linux (ACK, FPC, OpenWatcom)! I do think static is good sometimes, but mostly nobody cares. Snap/Flatpak/etc. will probably simplify distributing multi-file programs like this in the long run, if not already.

marcov

16.12.2019, 23:16

@ Laaca

.DXE .DLL .SO (etc.)

Post reply

> Borland pascal 7.0 can use both - overlays in realmode and DLL libraries in
> DOS 16-bit protected mode. (and for windows, of course).
> The DLL stuff is described quite well in the help system in IDE.
>
> The bad thing in DLLs in DOS is that they are a little bit usesless. What
> advantage give it against static linking in the single-task OS?

If you can't LoadLibrary() it is somewhat limited. Loadlibrary() is however difficult because then again you can only get main program symbols by probing for them.

> However the good thing ih the BP way of the DLL implementation is that it
> is quite straitforward and not much more complicated ovet normal
> static-linked unit.
>
> The DXE1 modules in the FPC has severe limitation that it has almost no
> connection with other parts of the program.

(it was straight port of Djgpp code by me, with some help of Pierre to get it debugged, I think I got it working within one week only working a good hour in the evenings or so)

> The EXE sees the inside of the DXE but the DXE sees nothing from EXE.
> The practical impact is the the DXE modules can not use any procedures (or
> functions) from any system or user unit. Nothing from unit CRT, nothing
> from DOS, even nothing from unit SYSTEM.
> So it is an extreme limitation for using this stuff.
> The only way how to bypass this situation is using the procedural
> variables.

Freepascal is working for packages, that means that all shared units must be in a DLL/BPL.

It is starting to work for windows, but the libraries get quite big, since there is no smartlinking. Go32V2 would be smaller, but still quite sizeable.

e.g. for win32 quoted sizes are:

2633984 rtl.dll
414820 rtl.objpas.dll
247060 rtl.extra.dll
364625 rtl.generics.dll
389888 fcl.res.dll
788664 fcl.base.dll
962560 fcl.xml.dll
953676 chm.dll

(extension is DLL, but has extras like Delphi BPL and also extra metadata files that needed during linking only)

A simple binary like chmls.exe then goes back from 285kb to 68kb. But the main advantage is not binary size, but that all DLL/BPL packages use the same libraries and library state, and can be used almost as if you would just divide the program into multiple binary on unit boundaries.

I haven't heard about loadlibrary use yet (but delphi can). Usually the trick is that both the EXE and the loadlibraried BPL/DLL statically link to another BPL/DLL that contains a registration of e.g. customer specific objects. (like forms etc)

This way you can plug in customer specific behaviour in already compiled programs.

> So it is so paintful that nobody uses it.
> Maybe in theory could be possible the implement into FPC the DXE3 handling
> which should not suffer by this limitation but it would need some
> DOS-and-pascal-megaguru...

And for 16-bit and 32-bit separately.

Laaca

Czech republic,
18.12.2019, 21:30

@ marcov

.DXE .DLL .SO (etc.)

Post reply

Yes, the DLL implementation in Borland pascal has nothing like LoadLibrary. All the used DLLs are automaticaly loaded during the .EXE initialization and they can't be unloaded or replaced.

On the other hand, even the DXE1 modules can be loaded and unloaded runtime.

The concept of packeges in interresting. I haven't known about it before (although is for years used in Delphi). The main advantage is that many units can be grouped in one file?
I am afraid about braking compatibility with my old code...
Yes, although not dynamicaly loaded I understand the possible advantage about changing the separate DLL modules without modifying the main EXE. I can imagine it in situation like demoversion/fullversion or "general game engine" / "customized game engine".
But the important question for me is: can be this feature be switched off?

---
DOS-u-akbar!

marcov

19.12.2019, 20:39

@ Laaca

.DXE .DLL .SO (etc.)

Post reply

> The concept of packeges in interresting. I haven't known about it before
> (although is for years used in Delphi). The main advantage is that many
> units can be grouped in one file?

You can say "this BPL/DLL contains these units". The units themselves don't change and if you don't work with BPL/DLL can be compiled normally.

IOW a package is like a DLL project, a mainprogram to build a DLL. The difference is in the metadata.

Then to use it, you have to explicitly say that you want to build a certain project with packages and list the packages to link. If you use packages, RTL is always in a package (since the other packages must link to it, so it can be shared).

> I am afraid about braking compatibility with my old code...

Packages won't be default. It is primarily useful for plugin architectures, just like DXE, just easier.

Things like IDEs use it to runtime load designtime packages and other IDE extensions.

Some RTL state will get an indirection, if you don't use it, not much changes otherwise.

> Yes, although not dynamicaly loaded I understand the possible advantage
> about changing the separate DLL modules without modifying the main EXE.

Note that this is only in the dynloaded loadlibrary case.

> I can imagine it in situation like demoversion/fullversion or "general game
> engine" / "customized game engine".
> But the important question for me is: can be this feature be switched off?

It is not even on by default. Not even in Delphi. The designtime packages you install in Delphi work via this system.

Rugxulo

Usono,
20.12.2019, 10:39

@ Laaca

.DXE .DLL .SO (etc.)

Post reply

> Yes, the DLL implementation in Borland pascal has nothing like LoadLibrary.
> All the used DLLs are automaticaly loaded during the .EXE initialization
> and they can't be unloaded or replaced.

There was a 16-bit DOS shareware Oberon compiler in the late '90s by Chris Peachment (Canadian) called Edipar (.LZH sfx). I came across it (before 2016) because I was trying various DOS compilers with my silly Befunge interpreter. It had a few bugs, but it did mostly work. I tested it under DOSEMU because it didn't work properly natively (memory bug?).

I know this isn't fully relevant here, but it did support loadable modules and could even combine them for a static .EXE. Very obscure in hindsight, but I just felt the trivial need to mention it, for completeness.

Note that this is not an OS, nor is it graphical, and it lacks garbage collection (must use DISPOSE). So it's just yet another cmdline compiler. Oh, and I think it was claimed to have been written in TopSpeed Modula-2 (for "286" [186?], no XMS needed, FPU or emulation).

marcov

20.12.2019, 18:06

@ Rugxulo

.DXE .DLL .SO (etc.)

Post reply

> TopSpeed Modula-2
> (for "286" [186?], no XMS needed, FPU or emulation).

TSM2 was my compiler before I went to FPC (then with targets go32v2, emx(OS/2) and Linux).

TSM2 was generally a 16-bit compiler but a later one than e.g. Borland Pascal 7.
There were several major versions (1,2,3), with TSM2 1.15/1.17 widely sold and used in education, even when later versions were out.

TS 3(.3x?) was the last product, but was offered as a bunch of modules and addons. Win3.1x, various protected mode options were all addon iirc.

It had real mode (roughly small,compact,large,xlarge), 286 protected mode and a 386 enhanced extender for 286 protected modem, lifting several limits and efficiency).

I never had the 386 addon, so don't know the exact details. I did have some of the win3.x stuff but never used it, since I never used win3 for anything serious.

Afaik it could generate 287 code, but the assembler didn't know any 386/387 opcodes. (requiring an external assembler to generate code for that). (386 db 66 based 32-bit integer support would have been very welcome, but few compilers support that)

sezeroz

16.12.2019, 06:15

@ Rugxulo

.DXE .DLL .SO (etc.)

Post reply

DXE3 is good enough, and we used it in DLL-requiring projects
such as quake2, sin (not public), and also hexen2. It handles
chain-loading of interdependent shared modules too as of 2.05
reliably. The djgpp 2.06 development version also handles the
extended coff relocations (more than 65535 relocs) and has an
updated linker script.

It works just fine in those projects for changing game modules
and changing refresh drivers (opengl / software) just like it
does on unix and windows.

Dxe3 has a serious inherent limitation, though: if dependencies
of the dxe changes (such as v1.00 of a dxe depends on malloc()
and free(), v1.02 additionally depends on strlen()), you will
need to either relink your exe with the new version's import lib,
or if you are dlopen()ing the dxe then add the new dependency
exports to your exe and rebuild it.