Back to home page

DOS ain't dead

Forum index page

Log in | Register

Back to index page

fritz.mueller

Munich, Germany,
07.01.2024, 21:20

Is there a DOS memory documentation available? (Users)

Hi,
I come closer and closer to final version 1.1.0 of FreeDOS help, see:

https://www.bootablecd.de/FreeDOS-Internet-version/help110/en/index.htm
(if you find a mistake there, please inform me.)

What I am still missing is a short and simple to understand explanation of the different DOS memory types, how it came to this and how to handle them (different config/autoexec configurations etc.).
Does anybody know such a free documentation that I can overtake for FD help?

Fritz

SuperIlu

Berlin, Germany,
08.01.2024, 00:35

@ fritz.mueller

Is there a DOS memory documentation available?

> Does anybody know such a free documentation that I can overtake for FD
> help?
Are you aware of the english wikipedia page about the subject?
https://en.wikipedia.org/wiki/DOS_memory_management

---
Javascript on MS-DOS? Try DOjS https://github.com/SuperIlu/DOjS
Fediverse: @dec_hl@mastodon.social

mceric

Germany,
08.01.2024, 00:58

@ fritz.mueller

Is there a DOS memory documentation available?

Hi and happy new year and thanks for all the htmlhelp updates :-)

Not sure if I understand your question correctly, but...

- normal low DOS memory is at most 640 kB. Sometimes minus EBDA which is an extra BIOS data area, see SWITCHES /E Option to manipulate if you know what you are doing.

- UMB upper memory blocks, which are the area between 640 kB and 1 MB accessible by all 8086 or newer CPU in theory, but reserved for VGA RAM, BIOS and similar in practice. Drivers like RDOSUMB, HIRAM or URAM can activate normal RAM in unused areas between those in chipset-dependent ways on various mainboards and all sorts of EMM386 style drivers can make RAM visible in those areas if you have 386 or newer CPU. Sometimes, UMB have limitations such as not working with DMA for floppy or harddisk. Extra drivers bundled with the UMB drivers, or extra options for EMM386 can be used to work around or try to work around the limitations. Sometimes you just want to avoid to load certain disk, network or USB drivers to UMB instead. You can tell the FreeDOS kernel to move some data structures to UMB with DOSDATA=... and you can use DOS=... to move some other kernel things to UMB. See also FILESHIGH, I guess.

- HMA high memory area is 64 kB big and starts where the first 1 MB ends. You can use it on 286 or newer CPU, usually by loading HIMEM or similar drivers. You can use DOS=... to move large parts of the kernel to HMA. Other drivers can use remaining HMA space, but this is not done as easily in FreeDOS as in other DOS. See also FILESHIGH, I guess. Because the accessbility of HMA depends on the infamous A20 address line gate, you may have to help HIMEM or EMM386 to treat that properly if autodetection has resulted in wrong A20 handling.

- XMS extended memory is what HIMEM and similar drivers give you. Note that JEMMEX is a driver containing both EMM386 and HIMEM in a single driver. XMS are memory areas which you can ask the driver to copy to or from DOS memory (or UMB, maybe even HMA) as needed. Protected mode apps can also access XMS directly at the original location beyond 1 MB after telling HIMEM to lock areas so they keep a constant absolute location. You need 286, with common drivers even 386 or newer CPU to use XMS.

- EMS expanded memory is what EMM386 usually gives you, but there also are chipset-dependent hardware drivers for certain old mainboards with hardware EMS support and there were hardware expansion cards for EMS for really old computers. Either way, EMS consists of originally 16 kB, later 4 kB sized pieces of memory which you can make visible (map) at locations below 1 MB. For the 4 kB style, you need EMS 4, the 16 kB style is EMS 3.2 and the latter also is limited to make the 16 kB chunks visible in any quarter of a 64 kB page frame at a location which has to be a multiple of 64 kB. That page frame takes away space which you could otherwise use for UMB, so there is a misleadingly named NOEMS option for EMM386 style drivers which actually means that you need no page frame. You can still map 4 kB to various multiples of 4 kB within the first megabyte if you use EMS 4 even when your EMM386 is in NOEMS mode and has no fixed page frame. Because you make memory appear at other places instead of actually copying it, EMS can fast when you want to swap around memory areas, maybe for handling libraries or background music data in a game, in situations where you do not want to copy a specific part of your data, but want a whole subset of your data to be available in case you want to use it. XMS, on the other hand, is good when you already know which specific data area you want to be copied where, because you do not have to take the extra step of mapping there.

- VCPI is a low-level interface for EMM386 which lets it co-exist with other protected mode software, so it also involves memory mapping and allocation. It is not very popular outside DOS extenders, I would say. Note that Windows can be called a DOS extender in this context.

- DPMI is a more high-level interface to manage memory in a protected mode context. It is so comfortable that the CWSDPMI DOS extender basically just makes sure that you get DPMI, if necessary distilled from a number of other (!) types of memory access available in pure DOS, because apps compiled with DJGPP and similar compilers already feel fine with DPMI as a DOS extender, roughly speaking. Windows also provides DPMI to apps, but not for example VCPI, because the latter would be too low-level for smooth coexistance with apps directly using that.

- DOS Extenders make sure that apps can use more than 1 MB of memory taken from any suitable source of memory, using protected mode. So they may differ a lot in which types of memory they support or not.

- This also involves INT15 memory, which is a BIOS based way to access memory above 1 MB which is not otherwise relevant to most DOS apps, because they prefer to use the more DOS specific types of memory instead. So INT15 is more something relevant for HIMEM and EMM386 to either use or coexist with. In general, the BIOS has provided a number of different methods to get to know which memory and where is available for which purposes, often INT15 related, so DOS memory drivers like HIMEM have to take care to query the newest available sub-method and they may have to resolve ambiguities and contradictions if the BIOS is not properly designed.

Note that very specific DOS extenders can also make it possible for apps to use more than 4 GB of RAM or more than 1 CPU core. Neither are widespread yet, so most implementations of HIMEM or EMM386 will be limited to between 2 GB and 4 GB of usable RAM even if you have more installed, because things like graphics memory or MMIO areas and extra BIOS stuff etc. will block some amount of memory within the first 4 GB. In particular graphics RAM windows can be quite large with modern graphics hardware and few apps or drivers for DOS take the effort to use 36- or 64-bit addressing to reach beyond the first 4 GB. You can assume that the system RAM which you cannot access below 4 GB because graphics RAM uses the address space will be made available at some place above 4 GB, so 64-bit aware drivers can use all your RAM. This is similar to the UMB problem mentioned above, RAM covered by BIOS or by VGA RAM and therefore not available for UMB can sometimes still be made available at another address for other purposes such as EMS depending on your hardware and drivers.

---
FreeDOS / DOSEMU2 / ...

Rugxulo

Usono,
08.01.2024, 09:46

@ mceric

Is there a DOS memory documentation available?

8086 could address 1 MB (conventional memory), but the IBM PC specifically (contrary to SCP's boards) would only use 640 kb (minus OS overhead) with ROM data (e.g. BIOS) in higher parts. EMS with bankswitching was optional, but some people (e.g. Jim Leonard) have physical hardware EMS, allowing a few extra MB.

XMS required at least a 286 (v2) or 386 (v3). The 286 could only address 16 MB (or 1 GB virtual, allegedly, IIRC). It fragmented a bit worse than EMS.

EMM386 was optional but helped get old EMS apps to work using V86 mode. VCPI (ring 0, 386 only) was a superset of this, supported by Win 3.x Standard mode (win.com /s), but DPMI was overall better.

DPMI was originally invented for Win 3.0 (Enhanced mode) in 1990 but later standardized and became common elsewhere (OS/2 2.1, Novell DOS 7). Unfortunately, most people only supported the 0.9 spec (usually for 32-bit apps only). But Borland did support 16-bit DPMI. So, just to be clear, unlike ring 0 32-bit exclusive VCPI, DPMI could also be ring 3 and either 286 or 386 compatible. Ring 3 is less prone to OS instability.

There are various bugs and limits in implementations of all of the above. Testing is more important than blind fidelity to specs.

marcov 08.01.2024, 22:40 @ Rugxulo	Is there a DOS memory documentation available? Post reply
	> XMS required at least a 286 (v2) (IIRC it required an AT, i.e. an 286 on a 286 mainboard. There were a bunch of 286's on XT derived mobos that weren't compatible with "286" software)

tauro

18.01.2024, 06:22

@ mceric

Is there a DOS memory documentation available?

Greetings, DOS experts.

Thank you for your explanations of the DOS memory structure and operation. I'm far from understanding all the details but this thread has been helpful.

I'm not sure about the validity of this but I've read about EMM386 being shunned because it puts the processor in Protected mode and thus the system runs slower and some incompatibilities may be introduced too. What's true and what's false about this statement? Why is it so? What are the cases where EMM386 breaks compatibility with other software? Is it because such software is not compatible with VCPI or Virtual/Protected mode?

On Windows 98 EMM386 doesn't need to be loaded as a DEVICE in CONFIG.SYS, but nevertheless Windows creates an EMM Page Frame for EMS compatibility, at least that's what's being shown in the DOS boxes, which I understand are partially emulated. How is this achieved, and how can the CPU unload EMM386 once you "Restart in MS-DOS mode" putting the processor back in Real mode? WIN is still loaded in conventional memory, does it have anything to do with it?

Is it possible to stop EMM386 on demand and completely unload it from memory on pure DOS?

mceric

Germany,
18.01.2024, 10:47

@ tauro

Is there a DOS memory documentation available?

> I'm not sure about the validity of this but I've read about EMM386 being
> shunned because it puts the processor in Protected mode and thus the system
> runs slower and some incompatibilities may be introduced too.

That is correct. EMM386 uses protected mode because this allows it to put DOS into a special task which simulates a normal DOS environment, but allows it to juggle with memory in software to provide EMS and UMB. There also was EMS hardware, sometimes as add-on card, sometimes as a chipset feature of the mainboard, but neither were widespread.

You also have UMB chipset drivers such as UMBPCI. Not all mainboards are supported by those and sometimes the UMB have limitations, such as not supporting DMA access for example related to disk/floppy/network/sound controllers. DMA can also be an issue with UMB and EMM386 if BIOS and drivers do not properly implement negotiations and preparations for them.

But back to EMM386: Because it simulates normality, there can be gaps in the simulation and somethings can be slower to simulate than others, such as the DOS style of handling hardware interrupts. Newer CPU had VME, virtual mode extensions, to speed up those again. The main compatibility issue with EMM386 is that by design, it cannot coexist with early protected mode DOS games which want to use protected mode themselves, but do not yet support various interfaces allowing EMM386 and the game to share it. More modern DOS games can even coexist with Windows through DPMI. Others cannot even coexist with HIMEM, but those are rare. Games with DOS extenders which cannot coexist with EMM386 (and Windows) were actually a thing for a while. Those would usually tell you that you have to reboot and not load EMM386 before playing them if they detected a loaded EMM386.

Which is one of the reasons why a default FreeDOS install will ask you at boot which standard set of drivers you want to load. Including choices where EMM386 is omitted.

Regarding VCPI: That is a very low level coexistence interface. It is used for example by Windows to replace the entire EMM386 by a Windows built-in implementation when it starts. Games may support it, but DPMI is far more app-oriented and smooth to use. Things compiled with DJGPP and related compilers often use CWSDPMI as DOS extender, so if CWSDPMI can use VCPI to load its own DPMI layer and then load your DJGPP-based app, it can use that route for running while EMM386 is loaded.

Games which do not care about protected mode will not usually have compatibility problems with EMM386, because EMM386 will only have to make things look like normal DOS processor mode, which is why the special task style used here is called virtual 8086 mode. In practice, EMM386 can let you use 32-bit computations as well, but it cannot let you mess with protected mode yourself. Your apps will have to use VCPI and play nice if they want that while EMM386 is loaded.

I would say the speed issue affects mostly interrupt-heavy apps, such as games which use soundcard IRQ many times per second. If your CPU and EMM386 support VME as mentioned above, the speed impact will be much less.

> On Windows 98 EMM386 doesn't need to be loaded as a DEVICE in CONFIG.SYS,
> but nevertheless Windows creates an EMM Page Frame for EMS compatibility

See above. Windows, even Windows 3, takes control of protected mode. While doing so, it is polite enough to also provide services normally offered by EMM386. If EMM386 is already loaded before Windows 3, then Windows needs to use VCPI to replace EMM386 with its built-in implementation on the fly.

The CPU does not unload EMM386 if you restart in DOS mode. You just actually restart and do not load it again that time, so you stay in real mode ;-)

In real mode, everybody is free to switch to protected mode any way they like. But if EMM386 is loaded, protected mode already is active, so it cannot allow you to do uncontrolled changes to protected mode.

Completely unloading EMM386 would involve rolling back all protected mode configuration. You would no longer have EMS and UMB, so it would only be possible if neither are used at that moment. Because it is far easier to just reboot and not load EMM386 again after that reboot, there would not be much use in EMM386 implementing a method to remove itself on the fly.

With JEMM386 and JEMMEX, you do have LOAD and UNLOAD command line options. Check whether they are sufficient for what you need, but note that you can only use them if you loaded JEMM... from autoexec or command line, not if you loaded it from config sys.

---
FreeDOS / DOSEMU2 / ...

Rugxulo

Usono,
19.01.2024, 01:25

@ mceric

Is there a DOS memory documentation available?

> > I'm not sure about the validity of this but I've read about EMM386 being
> > shunned because it puts the processor in Protected mode and thus the
> system
> > runs slower and some incompatibilities may be introduced too.
>
> That is correct. EMM386 uses protected mode

V86 mode (a special real-mode friendly form of protected mode), but yeah, since it's (always?) ring 3, it's more "protected" and thus prevents some things (for stability). The original Pentium (only), for instance, would not allow RDTSC under V86 mode. Others won't even let you write (or read!) a control register (unless they emulate that, I think JEMM does).

> because this allows it to put
> DOS into a special task which simulates a normal DOS environment, but
> allows it to juggle with memory in software to provide EMS and UMB.

Most DOS versions of EMM386 (except DR-DOS) required HIMEM.SYS to also be loaded first. And it was weird whether some would share XMS and EMS or not or how much.

> But back to EMM386: Because it simulates normality, there can be gaps in
> the simulation and somethings can be slower to simulate than others, such
> as the DOS style of handling hardware interrupts. Newer CPU had VME,
> virtual mode extensions, to speed up those again.

VME came with the Pentium (aka 586) in 1993.

> More modern DOS games can even coexist with Windows through DPMI.

Not with x64-only Windows 11. But yes, DPMI was invented by Windows 3.0 in 1990. Even XP had bugs that DJGPP 2.03p2 had to workaround, but it was most okay. With Vista, things went way downhill for DPMI support.

> Others cannot
> even coexist with HIMEM, but those are rare. Games with DOS extenders which
> cannot coexist with EMM386 (and Windows) were actually a thing for a while.
> Those would usually tell you that you have to reboot and not load EMM386
> before playing them if they detected a loaded EMM386.

I believe Quarterdeck (Desqview) was one of the main companies behind VCPI.

> Regarding VCPI: That is a very low level coexistence interface. It is used
> for example by Windows to replace the entire EMM386 by a Windows built-in
> implementation when it starts.

The so-called GEMMIS interface, right?

> Games may support it, but DPMI is far more
> app-oriented and smooth to use. Things compiled with DJGPP and related
> compilers often use CWSDPMI as DOS extender, so if CWSDPMI can use VCPI to
> load its own DPMI layer and then load your DJGPP-based app, it can use that
> route for running while EMM386 is loaded.

CWSDPMI is not a "DOS extender" in that it doesn't support unofficial int 21h extensions (which were never standardized). It's a "strict" 32-bit DPMI server only (no 16-bit DPMI). DJGPP v1 had various workarounds and extensions and a separate GO32.EXE "extender", but DJGPP v2 (circa 1996) is "DPMI only". CWSDPMI is just one of many ("standard") DPMI 0.9 hosts potentially supported. GO32.EXE is no longer needed at all.

> Games which do not care about protected mode will not usually have
> compatibility problems with EMM386, because EMM386 will only have to make
> things look like normal DOS processor mode, which is why the special task
> style used here is called virtual 8086 mode. In practice, EMM386 can let
> you use 32-bit computations as well, but it cannot let you mess with
> protected mode yourself. Your apps will have to use VCPI and play nice if
> they want that while EMM386 is loaded.

Any 32-bit cpu (e.g. 386) can use 32-bit computations (e.g. registers EAX etc.) even in real mode, assuming your BIOS isn't buggy and doesn't clobber the registers (which most don't).

> I would say the speed issue affects mostly interrupt-heavy apps, such as
> games which use soundcard IRQ many times per second. If your CPU and EMM386
> support VME as mentioned above, the speed impact will be much less.

So-called DOS-extended apps still have to switch back to DOS for the disk access. In theory, it's offset by an improvement in other speedups. Actually, switching to V86 mode from pmode (e.g. EMM386) should be faster than switching from pmode to real mode. Although the Pentium added "4 MB pages" for better speed, which CWSDPMI r7 optionally supports (by default), which is noticeably faster, but that's not supported under EMM386 because it's always 4 kb pages there (legacy requirement).

mceric

Germany,
19.01.2024, 10:42

@ Rugxulo

Is there a DOS memory documentation available?

> > That is correct. EMM386 uses protected mode
>
> V86 mode (a special real-mode friendly form of protected mode)

It is a special task type in the protected mode ecosystem and you need to activate protected mode in general to juggle with memory paging, which is what software-based EMS as in EMM386 needs. To be able to provide EMS and still run DOS requires EMM386 to encapsulate DOS in a V86 task, which feels similar to real hardware for DOS, but it is not the same, so EMM386 has to emulate some things. Apps which want to use other protected mode features (such as creating tasks using more than 1 MB of RAM) get in conflict with EMM386 while in V86, so they have to make VCPI requests instead of just messing with protected mode directly.

> VME came with the Pentium (aka 586) in 1993.

So with a VME-aware EMM386 version and a Pentium or similar or newer CPU, the hardware interrupt IRQ speed loss caused by EMM386 will be much smaller, yes.

> > More modern DOS games can even coexist with Windows through DPMI.

Of course only with Windows versions which still offer DPMI.

> > Regarding VCPI: That is a very low level coexistence interface.

Not to be confused with GEMMIS, but both are related, yes. VCPI allows you to take over protected mode powers from EMM386. GEMMIS allows in particular Windows to know about the internal state of EMS and UMB, helping it to take over the entire work of EMM386 instead of only grabbing some powers from it.

What makes VCPI worse than DPMI is that it is more for grabbing power and less for cooperation. DPMI is more cooperative, but also far more complex, so EMM386 usually only have the former built in. I think DR-DOS EMM386 had both built in. Normally, DPMI is provided by Windows or by a separate DPMI host driver. CWSDPMI provides DPMI, too.

> CWSDPMI is not a "DOS extender"

Well it is somewhat in between. Classic DOS extenders help you to port DOS apps to protected mode, offering protected mode wrappers for INT 21 etc.

CWSDPMI just makes sure you get DPMI and a few other goodies, which in combination "extends" DOS far enough to let DJGPP compiled and similar apps run. Those apps do not target classic real mode DOS anyway, they are already built for the DPMI protected mode ecosystem.

As Rugxulo says, all types of protected mode apps for DOS have some speed loss because BIOS and DOS kernel will work in V86 mode while the app uses protected mode. The VESA VBE graphics BIOS can also be used directly in protected mode.

The FD32 project uses a FreeDOS kernel running directly in protected mode, too, to speed up DPMI-based apps, but practical speed gains are limited.

In DOSEMU2, even the shell (a port of FreeCOM) runs in protected mode on the Linux (or other host operating system) side with some glue magic to make it visible from the DOS V86 or DOS emulated CPU world. Thanks to CPU emulation, DOSEMU2 can be ported to non-x86, non-amd64 CPU.

> Actually, switching to V86 mode from pmode (e.g. EMM386) should be faster
> than switching from pmode to real mode.

Probably, yes. You also had ancient protected mode DOS apps which just refrained from using DOS and the BIOS at all after loading, so they had to do all hardware access themselves, but avoided the effort of providing a V86 task for DOS and BIOS.

---
FreeDOS / DOSEMU2 / ...

Japheth

Germany (South),
19.01.2024, 11:57

@ mceric

Is there a DOS memory documentation available?

I have 2 remarks, which are a bit nitpicking, but to avoid generating myths:

> So with a VME-aware EMM386 version and a Pentium or similar or newer CPU,
> the hardware interrupt IRQ speed loss caused by EMM386 will be much
> smaller, yes.

Actually, VME only may speed-up software interrupts. Hardware interrupts and exceptions will generally switch to protected-mode, no matter what VME is telling.

> Actually, switching to V86 mode from pmode (e.g. EMM386) should be
> faster than switching from pmode to real mode.

This may only be true if there's no context switch involved ( like in Win3x/Win9x ). For DPMI hosts that run as VCPI clients things are different, because register CR3 has to be switched for every mode switch.

---
MS-DOS forever!

ecm

Düsseldorf, Germany,
19.01.2024, 16:42

@ mceric

Is there a DOS memory documentation available?

> In DOSEMU2, even the shell (a port of FreeCOM) runs in protected mode on the Linux (or other host operating system) side with some glue magic to make it visible from the DOS V86 or DOS emulated CPU world.

This is untrue. While the recommended dosemu2 shell, comcom32, does run in Protected Mode, some of your statements are inaccurate. I don't think comcom32 is a port of FreeCOM. Further, as of now comcom32 runs as a 32-bit DPMI client so depending on the modes used for DPMI (KVM, native Linux task on amd64/i386, or emulated) it may or may not run on the host machine. And on amd64 hosts comcom32 will (as the name suggests) still run as 32-bit code.

You're probably confusing comcom32 with dosemu2's default DOS kernel, fdpp. The fdpp kernel is a port of the FreeDOS kernel but most of its code runs as host code in what I believe is a "dosemu2 plugin". In a dosemu2 compiled as an amd64 long mode application, most of the matching fdpp will also run as 64-bit code.

---
l

mceric

Germany,
19.01.2024, 18:02

@ ecm

Is there a DOS memory documentation available?

Hmm indeed. A DPMI client is something different than something magically running on the Linux side, and an alternative is something different than a port. Also interesting that dosemu2 DPMI can be based either on Linux tasks, KVM or emulated CPU tasks?

> You're probably confusing comcom32 with dosemu2's default DOS kernel, fdpp.
> The fdpp kernel is a port of the FreeDOS kernel but most of its code runs
> as host code in what I believe is a "dosemu2 plugin". In a dosemu2 compiled
> as an amd64 long mode application, most of the matching fdpp will also run
> as 64-bit code.

Quite possibly the root of my confusion, yes. Thanks for the clarification!

Thanks to Japheth as well: VME only accelerating software interrupts, but not hardware IRQ, makes a significant difference that I should have known about.

---
FreeDOS / DOSEMU2 / ...

bretjohn

Rio Rancho, NM,
08.01.2024, 18:17

@ fritz.mueller

Is there a DOS memory documentation available?

> Hi,
> I come closer and closer to final version 1.1.0 of FreeDOS help, see:
>
> https://www.bootablecd.de/FreeDOS-Internet-version/help110/en/index.htm
> (if you find a mistake there, please inform me.)
>
> What I am still missing is a short and simple to understand explanation of
> the different DOS memory types, how it came to this and how to handle them
> (different config/autoexec configurations etc.).
> Does anybody know such a free documentation that I can overtake for FD
> help?

The descriptions given below by others are accurate, but I'm not sure are all that helpful to you for writing FreeDOS help. The descriptions sort of provide reasons as to why a PROGRAMMER might decide to use a particular type or combination of types of memory. What types of memory you may actually need to configure in FreeDOS at boot time depends on what programs you are going to run and what kind(s) of memory they use. No matter what you put in the help file, it will not be 100% accurate or complete. I applaud the effort, though.

I will also say that the descriptions provided below do not really address the memory requirements of TSRs and device drivers, which are VERY different than "regular" programs. E.g., "regular" programs almost never use Upper or High Memory, and TSRs and device drivers almost never use DPMI. In addition, DPMI can be incompatible with certain kinds of TSRs (see the thread:

http://www.bttr-software.de/forum/forum_entry.php?id=20131&page=0&category=0&order=last_answer

which talks about problems between DPMI and SBEMU).

fritz.mueller

Munich, Germany,
08.01.2024, 20:30

@ bretjohn

Is there a DOS memory documentation available?

> > Hi,
> > I come closer and closer to final version 1.1.0 of FreeDOS help, see:
> >
> > https://www.bootablecd.de/FreeDOS-Internet-version/help110/en/index.htm
> > (if you find a mistake there, please inform me.)
> >
> > What I am still missing is a short and simple to understand explanation
> of
> > the different DOS memory types, how it came to this and how to handle
> them
> > (different config/autoexec configurations etc.).
> > Does anybody know such a free documentation that I can overtake for FD
> > help?
>
> The descriptions given below by others are accurate, but I'm not sure are
> all that helpful to you for writing FreeDOS help. The descriptions sort of
> provide reasons as to why a PROGRAMMER might decide to use a particular
> type or combination of types of memory. What types of memory you may
> actually need to configure in FreeDOS at boot time depends on what programs
> you are going to run and what kind(s) of memory they use. No matter what
> you put in the help file, it will not be 100% accurate or complete. I
> applaud the effort, though.
>
> I will also say that the descriptions provided below do not really address
> the memory requirements of TSRs and device drivers, which are VERY
> different than "regular" programs. E.g., "regular" programs almost never
> use Upper or High Memory, and TSRs and device drivers almost never use
> DPMI. In addition, DPMI can be incompatible with certain kinds of TSRs
> (see the thread:
>
> http://www.bttr-software.de/forum/forum_entry.php?id=20131&page=0&category=0&order=last_answer
>
> which talks about problems between DPMI and SBEMU).

OK, I just uploaded the first version of memory.htm with a lot of links and comments from Rugxulo and bretjohn. If it is not ok, please tell me.
https://www.bootablecd.de/FreeDOS-Internet-version/help110/en/hhstndrd/help/memory.htm

tom

Germany (West),
08.01.2024, 21:20

@ fritz.mueller

Is there a DOS memory documentation available?

> OK, I just uploaded the first version of memory.htm with a lot of links and
> comments from Rugxulo and bretjohn. If it is not ok, please tell me.
> https://www.bootablecd.de/FreeDOS-Internet-version/help110/en/hhstndrd/help/memory.htm

it's not ok.

this is a text about a fairly difficult topic. overlaying this text with a COMPLETLY CONTENTFREE GRAPHIC IS NOT HELPFUL. Neither for me (expert) nor for the intended reader.

marcov

08.01.2024, 23:22

@ fritz.mueller

Is there a DOS memory documentation available?

> OK, I just uploaded the first version of memory.htm with a lot of links and
> comments from Rugxulo and bretjohn. If it is not ok, please tell me.
> https://www.bootablecd.de/FreeDOS-Internet-version/help110/en/hhstndrd/help/memory.htm

I'll also try to phrase something, since the above writeup is as confusing as wikipedia. I quite often see confusion between extended/expanded memory on one side, and XMS and EMS on the other.

I sometimes use the term AT, which is a more standardise motherboard architecture for a 286. There are really _weird_ ones that aren't, but those are rare.
----------

Basically "extended memory" is what we now consider memory. So simms or dimms plugged into the motherboard, directly accessible/addressable memory to the CPU. It is the same memory as the first 1 MB, but simply the part that is isn't accessible directly in 16-bit CPU modes. (*)

Expanded memory is basically _any_ memory, anywhere (typically a plugin card in some slot), , made available over a driver. In XT times this was a way to expand memory beyond 1MB while the CPU and mobo couldn't handle it.

But even while the 286+ (or better AT, so 286 on a suitable AT mobo, some very early 286's excluded) could address more (than 1MB+64kb), it couldn't in 16-bit mode.

To make these kinds of physical memory (extended mobo, or off it) accessible to then still dominant 16-bit apps, APIs were defined.

The oldest API (which is "EMS") for access to expanded memory, basically instruments the driver to move the data from the external memory to the 16-bit page frame and vice versa. (the page frame is typically an area of 64KB in the 640-1MB area that is accessible to the processor in 16-bit modes, QEMM defaults to C000-CFFF IIRC). The API doesn't expose too many details.

XMS is a similar API that allows Extended memory to be copied between the Dos 1MB(optionally +64KB, A20) and the memory above that area. It is a bit more transparent and 1:1, since extended memory as directly CPU wired memory has to be transparent anyway.

Now the confusion starts, since Expanded memory can be anywhere, it can also be in Extended Memory.

So extended memory can be used for EMS too, and usually except for XTs and some weird systems, this has been the norm since the AT/286 era. Stronger even, since the 386 features a MMU, for 16-bit applications EMS can be very efficient, since the copying to and from the page frame can be done by zero copy memory mapping rather than copying.

Finally, the API DPMI is usually used with dos-extenders that puts the processor in protected mode (only briefly returning for real mode interrupts), so the application can access extended memory (286: 16MB 386:4GB) directly. For 386 systems, the startup code allocates all available memory via XMS, and then the application accesses it as "direct" memory till it terminates.

The only 286 PM programs that I made (Borland/Turbo Pascal 7) afaik got heap memory on the fly by doing heap calls when allocated. I.e. it didn't allocate anything up front, but 286 PM program.

The reason for this difference is that the 386 with its MMU can map all available memory, but only really allocate it (from the free XMS memory pool) when the CPU accesses it, and simulate it by swapping to disk if the XMS pool is empty.

(*) Something similar happened again when 32-bit got too tight, and they expanded the memory bus to 36-bit and defined an API (PAE) to allow special 32-bit apps like Exchange and SQL Server to handle more than 4GB. The OS usually was fully 36-bit (using a segmentation model), but the apps were accessing an API to escape the 32-bit constraints.

ecm

Düsseldorf, Germany,
09.01.2024, 00:00

@ marcov

Is there a DOS memory documentation available?

> Basically "extended memory" is what we now consider memory. So simms or
> dimms plugged into the motherboard, directly accessible/addressable memory
> to the CPU. It is the same memory as the first 1 MB, but simply the part
> that is isn't accessible directly in 16-bit CPU modes. (*)
>
> Expanded memory is basically _any_ memory, anywhere (typically a plugin
> card in some slot), , made available over a driver. In XT times this was a
> way to expand memory beyond 1MB while the CPU and mobo couldn't handle it.
>
>
> But even while the 286+ (or better AT, so 286 on a suitable AT mobo, some
> very early 286's excluded) could address more (than 1MB+64kb), it couldn't
> in 16-bit mode.
>
> To make these kinds of physical memory (extended mobo, or off it)
> accessible to then still dominant 16-bit apps, APIs were defined.

It isn't technically accurate to refer to "16-bit CPU modes" when the modes you are pointing at are specifically "Real Address Mode" or Real/Virtual 86 Mode. (Which I call 86 Mode for short.) In these modes only 1024 KiB (with A20 wraparound) or 1088 KiB minus 16 B (without A20 wraparound) are accessible. It is true that the segment limits are generally 65_535 Bytes in 86 Mode, and that the Code Segment D bit (Default size) and the Stack Segment B bit (Big) are both zero, indicating a default 16-bit CS and a 16-bit SS.

However, 86 Mode isn't the only 16-bit mode. There is 16-bit Protected Mode since the 286, which allows to access the entire 16 MiB of the 286 address space, albeit only in 64 KiB chunks.

---
l

bretjohn

Rio Rancho, NM,
09.01.2024, 01:03

@ ecm

Is there a DOS memory documentation available?

> However, 86 Mode isn't the only 16-bit mode. There is 16-bit Protected Mode
> since the 286, which allows to access the entire 16 MiB of the 286 address
> space, albeit only in 64 KiB chunks.

Actually, you can use 16-bit protected mode with 386+ CPU's to access the entire 4 GiB of memory, but still only in 64 kiB chunks. 16-bit PM code is nice because it is almost exactly the same as 16-bit real mode code (the only real difference being that you are using selectors instead of segments). This lets you avoid the need to switch between 16-bit code and 32-bit code.

ecm

Düsseldorf, Germany,
09.01.2024, 09:38

@ bretjohn

Is there a DOS memory documentation available?

> > However, 86 Mode isn't the only 16-bit mode. There is 16-bit Protected
> Mode
> > since the 286, which allows to access the entire 16 MiB of the 286
> address
> > space, albeit only in 64 KiB chunks.
>
> Actually, you can use 16-bit protected mode with 386+ CPU's to access the
> entire 4 GiB of memory, but still only in 64 kiB chunks. 16-bit PM code is
> nice because it is almost exactly the same as 16-bit real mode code (the
> only real difference being that you are using selectors instead of
> segments). This lets you avoid the need to switch between 16-bit code and
> 32-bit code.

Well, yes, I should have clarified that I was referring to the entire address space of the 286 as a "16-bit CPU". Indeed you can access the 4 GiB address space using 16-bit segments on the 386.

However, even if you run in a CS with 16-bit default address/operand size, the 386 allows you to change the sizes of a single instruction to 32-bits using the asize and osize prefixes. This works in 86 Mode and a 16-bit CS in Protected Mode. However, on a 386 in Protected Mode you can easily set segment limits beyond FFFFh so 32-bit memory references are possible from a 16-bit CS, at which point the term "16-bit Protected Mode" perhaps doesn't fully apply any longer.

(Debug/X and lDebugX in fact do access 32-bit segments using asize prefixes from their 16-bit CS, when running on a 386. As you noted, 16-bit 86 Mode and 16-bit PM are similar, so most of the resident debugger is dual-mode code that can run in either mode. Other than selectors vs segments, which implies that 86 Mode segment arithmetic won't work, PM may also run the DPMI client in ring 3 as opposed to the Real 86 Mode running in ring 0. (Virtual 86 Mode must run in ring 3 I believe, but the VMM may emulate ring 0 features.))

---
l

bretjohn

Rio Rancho, NM,
09.01.2024, 17:35

@ ecm

Is there a DOS memory documentation available?

> However, even if you run in a CS with 16-bit default address/operand size,
> the 386 allows you to change the sizes of a single instruction to 32-bits
> using the asize and osize prefixes. This works in 86 Mode and a 16-bit CS
> in Protected Mode. However, on a 386 in Protected Mode you can easily set
> segment limits beyond FFFFh so 32-bit memory references are possible from a
> 16-bit CS, at which point the term "16-bit Protected Mode" perhaps doesn't
> fully apply any longer.

Correct, and as you stated you probably really can't call it 16-bit PM any more, though I'm not sure exactly what you would call it. There are quite a few modern mode "variations" that don't exactly fit any of Intel's "standard" nomenclature (e.g., I personally think "Unreal Mode" is a really bad, or at least non-descriptive, term to use).

Why this matters in my specific case is that I'm updating my TSR's so that they will use less memory. I'm doing this by automatically leveraging the many types of memory access that are available and useful for TSR's: Upper Memory, EMS (with a Page Frame), and DPMS (sort of a "TSR Sidekick to DPMI" that was created by Novell a long time ago). When DPMS is installed, I use 16-bit PM, when it's not DPMS I use Real/V86 mode. I also don't use any op-codes or registers that won't work on an 8086/8088 CPU unless it's a section of code that doesn't run unless I know I'm using an advanced CPU.

With the setup I'm using, the same basic 16-bit code is used (and accesses 16-bit-sized data segments) no matter which environment the TSR got installed in (Lower, Upper, EMS, or DPMS). It will also work whether the EMS is real (via an add-on card on an 8086/8088 CPU) or whether it is emulated through XMS via something like EMM286 or EMM386. The TSR's will work on any CPU and I really don't need to change anything.

99% of the code is the same no matter what the environment is -- the only part that changes is when I'm manipulating segments/selectors. With this methodology, I am able to create a really complicated, intricate, full-featured TSR and still have it require only a few kiB of conventional or upper memory (if I'm using an advanced CPU) while most of the TSR is "hidden" in other places.

Oso2k 09.01.2024, 20:09 @ bretjohn	Is there a DOS memory documentation available? Post reply
	> Correct, and as you stated you probably really can't call it 16-bit PM any > more, though I'm not sure exactly what you would call it. There are quite > a few modern mode "variations" that don't exactly fit any of Intel's > "standard" nomenclature (e.g., I personally think "Unreal Mode" is a really > bad, or at least non-descriptive, term to use). It's also been called "Flat Real Mode".

fritz.mueller

Munich, Germany,
09.01.2024, 22:21

@ Oso2k

Is there a DOS memory documentation available?

STOP!
It's always the same - nobody reads the subject!

Writing the help is a pain, there are some problems that happen again and again (and I am working for more than TWO years on this version now):
a) A command has NO documentation - with bad luck even no /? help. So you have to run tests, ask people, and anyhow you decide that your own text is the best. Years later you get a message "How could you forget this?"

b) A command has a documentation that is overlong and it is unable to take EVERYTHING into the help - so you have to reduce yourself to the minimum.
But it is no fun to read 200 pages of documentation for this.

c) A command has a short documentation that explains the most important effects and you can simply overtake it. GREAT!

d) I find a bug and try to report it to the programmer, with good luck I get a response, with bad luck I get none. Some bugs are fixed, some not. In the meantime I feel free enough to write a short sentence into the help why I think that the program is buggy.

e) I get a message that "an ugly picture" hides the whole documentation but
the creator forgets that htmlhelp and amb do not support background pictures,
they are only for usage in internet,

f) I ask the DOS group for giving me a free text, I get one (usually not) - and then the discussion begins - and continues and continues - and nobody remembers to my question. Nothing to say against, but is there anybody who can do the following:

Replace this sentence by this or that sentence so that I can simply overtake it?

No, there seems to be none. At the end I am as clever as before and must try to create my own text and implement all this additional feedback to a mixture. And later people come and say: You forgot this / you forgot that! What nonsense did you write there?

So: Please, please, correct the text but do not start a big discussion that is absolutely useless for me and makes more work than I would have had without asking you. I do not need the last details as only less USERS will understand them, I only need a text that I can simply overtake. Is this so hard to understand?
Sorry, I beg your pardon, but sometimes I am close to give up this project so close to end. Or I will simply remove this problematic texts.

Thanks for reading (and not sending a shitstorm).

Fritz

Zyzzle

10.01.2024, 04:47

@ fritz.mueller

Is there a DOS memory documentation available?

> c) A command has a short documentation that explains the most important
> effects and you can simply overtake it. GREAT!
OK, one command which has too short documentation in FreeDOS is in the SWITCHES syntax, specifically the /E switch. Apparently it does this:
/E enables moving of EBDA (Extended BIOS Data Area), optionally a size in kilobytes may be specified [xxx, in range of 48-1024]

Now, I have multiple newish laptops which only report ~625kb of total low (640kb memory) due to code which is in the EDBA. Nothing I've been able to do will get dos to report "655360 (640K) of total memory" because of the commandeering of the EDBA code which takes low memory, before anything else is apparently even loaded - including the DOS kernel itself.

So, my question is will using the /E switch somehow move that code out of low memory and into upper memory? XMS memory? Or what will it do, exactly.

I've also read conflicting reports that say the /E switch does the opposite of what I think it might do, that is, *disable* the kernel from moving EBDA BIOS code into upper memory / UMB / XMS. Others have claimed the /E switch is "broken" and has no effect on most systems. Further, why does MS-DOS have no such (equivalent) option?

And lastly, why can't memory ranges of less than 48 kb be specified for the switch. In my case, 32 kb, or even 16kb would be sufficient to reclaim my "lost" low memory taken by the BIOS code.

And, I do appreciate all the time you've spent on creating this documentation.

mceric

Germany,
10.01.2024, 11:56

@ Zyzzle

Is there a DOS memory documentation available?

> /E enables moving of EBDA (Extended BIOS Data Area), optionally a size in
> kilobytes may be specified [xxx, in range of 48-1024]

Documentation about SWITCHES=/E:nnnn could indeed be better. The size is specified in bytes, not in kilobytes. The size has to be a multiple of 16 bytes and at least 48 bytes, so the kernel rounds up the value you specify to the next multiple of 16. Example values:

/* allowed values: [48..1024] bytes, multiples of 16 * e.g. AwardBIOS: 48, AMIBIOS: 1024 * (Phoenix, MRBIOS, Unicore = ????) */ ... if (n >= 48 && n <= 1024) ... Config.ebda2move = (n + 15) & 0xfff0;

The EBDA is not code, it is data. Relevant code is in DoConfig and PreConfig2. If you do not specify /E:nnnn, the kernel will calculate the size in kilobytes based on BIOS data about EBDA segment and amount of low RAM (which is at most 640k). If you do specify /E:nnnn, it will cap the size to your specified amount of bytes, at most 1 kB, at your own risk.

unsigned ebdasize(void) { unsigned ebdaseg = peek(0, EBDASEG); unsigned ramsize = ram_top; if (ramsize == peek(0, RAMSIZE)) if (ramsize * 64 == ebdaseg && ramsize < 640 && peek(0, RAMSIZE) == ramsize) { unsigned ebdasz = peekb(ebdaseg, 0); /* sanity check: is there really no more than 63 KB? * must be at 640k (all other values never seen and are untested) */ if (ebdasz <= 63 && ramsize + ebdasz == 640) return ebdasz * 1024U; } return 0; }

Because EBDA is supposed to be data, the kernel can move it to other places, possibly in UMB (or HMA?) simply by copying it around and updating the EBDA segment value somewhere at 40:xx similar to updating a pointer.

If you see far less than 640k of RAM, you may have some code or data buffers at the end of your 640k. The kernel can NOT move those around. Think about MBR -based drivers for supporting harddisks larger than 504 MB or 8 GB if your BIOS is too old. Also think about viruses and maybe buffers of exotic network or disk controller ROMs.

You probably should not use the /E switch most of the time. The kernel can autodetect the size and you save less than 1 kB by manually specifying the size if the autodetected size was 1 kB. For other sizes, the kernel cannot help you. I think you can also specify size -1 or size 0 for special cases such as "autodetect" or "do not touch EBDA".

---
FreeDOS / DOSEMU2 / ...

ecm

Düsseldorf, Germany,
10.01.2024, 21:14

@ mceric

Is there a DOS memory documentation available?

> Because EBDA is supposed to be data, the kernel can move it to other
> places, possibly in UMB (or HMA?) simply by copying it around and updating
> the EBDA segment value somewhere at 40:xx similar to updating a pointer.

The HMA is not addressable using an offset of zero as the EBDA requires. The highest segment value is 0FFFFh which still does not point into the HMA at offset zero, rather it points to the last 16 Bytes of the ROM-BIOS image usually placed below 1 MiB. So you cannot move the EBDA into the HMA.

> If you see far less than 640k of RAM, you may have some code or data
> buffers at the end of your 640k. The kernel can NOT move those around.
> Think about MBR -based drivers for supporting harddisks larger than 504 MB
> or 8 GB if your BIOS is too old. Also think about viruses and maybe buffers
> of exotic network or disk controller ROMs.

Bootable debuggers such as lDebug may also decrease the size of the Low Memory Area as indicated by interrupt 12h in order to protect their allocation. (The int 12h simply returns the word value at 413h in AX. You modify the variable, not the interrupt handler.) As int 12h specifies the memory size in KiB, the debugger must round up the size to allocate for itself to happen upon a KiB boundary.

Barely related ad: For lDebug, the first paragraph of its bootloaded mode allocation starts with a small header that reads either NSYM or NDEB. The NDEB image identification paragraph is described a bit in https://hg.pushbx.org/ecm/ldebug/file/bbf39043e416/source/init.asm#l2114

---
l

bretjohn Rio Rancho, NM, 12.01.2024, 04:42 @ ecm	Is there a DOS memory documentation available? Post reply
	I've never messed with bootloaders, but what are NSYM & NDEB? I've never heard of those before.

ecm

Düsseldorf, Germany,
12.01.2024, 18:00

@ bretjohn

Is there a DOS memory documentation available?

> I've never messed with bootloaders, but what are NSYM & NDEB? I've never
> heard of those before.

They are simply what I chose to call the headers that bootable lDebug creates for itself. As depicted in the linked sources, NDEB has four letters as a signature, then two digits for a version number (always "00" so far), a checksum for the image identification, and a size. (The size must be given as an amount of paragraphs; the comments could be clearer on that.)

align 16, db 0 ; Image identification ; First dword: signature ; Next word: version, two ASCII digits ; Next word: checksum. adding up all words of the paragraph gives zero ; Next word: size of image (including this paragraph) ; Three words reserved, zero. imageident: db "NDEB00" .check: dw 0 .size: dw 0 times 3 dw 0

While NDEB covers the allocation of the bootloaded debugger instance / 'process' as such (not really a DOS application process of course), NSYM is used for an allocation of symbol tables when symbolic bootable lDebug is running. Other than the differing name, the definition of the NSYM block is similar to NDEB's.

The NSYM allocation will always be done using the int 12h / word 413h memory size mechanism as well. A single bootloaded debugger instance will only allocate exactly one NDEB and at most one NSYM. But the debugger can load another instance of the debugger, so there can be multiple NDEB for instance. This can be useful to debug the debugger itself, in which the inner debugger should be debuggable or conditionally debuggable.

The naming choice of NDEB stems from the internal, pre-release-0 times of when lDebug used to be called NDebug, inspired by the "Netwide Assembler". Unfortunately "NDEBUG" seems to be taken for a C-language-style preprocessor macro, which I seem to recall is one of the reasons I changed the name.

---
l

marcov

09.01.2024, 09:42

@ ecm

Is there a DOS memory documentation available?

> > very early 286's excluded) could address more (than 1MB+64kb), it
> couldn't
> > in 16-bit mode.
> >
> > To make these kinds of physical memory (extended mobo, or off it)
> > accessible to then still dominant 16-bit apps, APIs were defined.
>
> It isn't technically accurate to refer to "16-bit CPU modes" when the modes
> you are pointing at are specifically "Real Address Mode" or Real/Virtual 86
> Mode. (Which I call 86 Mode for short.)

You are right. Further in the text I already replaced 16-bit with real mode, missed this one apparently. But actually I'm happy with such detail errors, means the broad lines are at least correct :-)

It was written from memory mainly to get the difference between the APIs and real memory across, as they are so often mixed up.

boeckmann

Aachen, Germany,
19.01.2024, 10:09

@ fritz.mueller

Is there a DOS memory documentation available?

> Hi,
> I come closer and closer to final version 1.1.0 of FreeDOS help, see:
>
> https://www.bootablecd.de/FreeDOS-Internet-version/help110/en/index.htm
> (if you find a mistake there, please inform me.)
>
> What I am still missing is a short and simple to understand explanation of
> the different DOS memory types, how it came to this and how to handle them
> (different config/autoexec configurations etc.).
> Does anybody know such a free documentation that I can overtake for FD
> help?
>
> Fritz

While not directly usabe for the HELP, OSNEWS recently linked a nicely pictured article explaining the memory layout from 0 to 1 MB:

https://blogsystem5.substack.com/p/from-0-to-1-mb-in-dos

Back to index page

22049 Postings in 2034 Threads, 396 registered users, 196 users online (0 registered, 196 guests)
DOS ain't dead | Admin contact

powered by my little forum