DOS ain't dead - Howto receive 4GB memory as one chunk, DPMI + Watcom C

DosWorld

06.02.2019, 23:06

Howto receive 4GB memory as one chunk, DPMI + Watcom C (Developers)

Hello!

I had develop software (text editor) and I want open very huge files. My problem is - dpmi hosts does not return memory as one chunk. So, watcom memory manager use memory only from one chunk (~64mb, with causeway extender - 220mb). How to received full memory as one chunk?

My target - open 3-3.5gb text file. Now, I can open only little bit more 40mb :(
May be somebody have any advise..

As I understood, need something like virtual memory mapping. (But what is it and how prepare it?)

PS: One time I receive all memory as one chunk, but in very strange way: I compile programm for watcom "compact" memory model for dos4g-extender and run it via HX's dpmild32. I had read recomendations does not use this memory model (because library is not ready, but printf works for me!) and I dont known how to receive access to first megabyte (0000:004xx, b800:0000) in this memory model. At the same time, programm compiled for "hxdos" had 64mb chunks.

PPS: For memory size check I am use source code from Watcom Programmers Guide (I hope, many people see it).

PPPS: I had receive big speed degradation for watcom's malloc, after 500k allocated strings. I think, it will be unusable after 1m malloc's. It is looks like watcom manage memory via linked list structure. So, I want refuse using watcom malloc and switch to my own memory manager (may be he not so fast, but he remember where is stored first free memory block).

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

RayeR

CZ,
07.02.2019, 02:03

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

With DJGPP/CWSDPMI you should be able to malloc something around 2GB... There's also new API for 64bit long mode, see some older posts here...

---
DOS gives me freedom to unlimited HW access.

DosWorld

14.02.2019, 01:36

@ RayeR

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

> With DJGPP/CWSDPMI you should be able to malloc something around 2GB...
> There's also new API for 64bit long mode, see some older posts here...

Hmm... It will be "plan B". DJGPP is good compiler, but... too modern.

(it other words - djgpp build "hello world" >20s on my k6/300. I want see my texteditor.exe before I am die)

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

alexfru

USA,
07.02.2019, 09:55

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

> I had develop software (text editor) and I want open very huge files. My
> problem is - dpmi hosts does not return memory as one chunk. So, watcom
> memory manager use memory only from one chunk (~64mb, with causeway
> extender - 220mb). How to received full memory as one chunk?
>
> My target - open 3-3.5gb text file. Now, I can open only little bit more
> 40mb :(
> May be somebody have any advise..

If the problem is in the standard library, you should try allocating memory directly either via DPMI functions or HIMEM.SYS (possibly, before the main .EXE, and then map it via DPMI).

Alex

DosWorld

14.02.2019, 01:30

@ alexfru

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

> If the problem is in the standard library, you should try allocating memory
> directly either via DPMI functions or HIMEM.SYS (possibly, before the main
> .EXE, and then map it via DPMI).

yes :( (I have no enough expirience for it, now)

Also... What about SmallerC's dpstub.asm ? (is this mapping available? may be more easy way - just switch to SmallerC?)

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

alexfru

USA,
14.02.2019, 10:32

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

> > If the problem is in the standard library, you should try allocating
> memory
> > directly either via DPMI functions or HIMEM.SYS (possibly, before the
> main
> > .EXE, and then map it via DPMI).
>
> yes :( (I have no enough expirience for it, now)
>
> Also... What about SmallerC's dpstub.asm ? (is this mapping available? may
> be more easy way - just switch to SmallerC?)

First of all you should probably forget about editing any files larger than 2GB-1. That's a FAT16 maximum. FAT32 may let you have 4GB-1-long files, but...

... Secondly, in 32-bit compilers (especially old ones or developed for old systems) long and fpos_t likely won't let you handle more than 2GB-1 for file sizes and offsets. malloc() may also fail for larger sizes (due to possible implementation bugs, even if the underlying DPMI host allows it).

Third, I don't know what size limitations are in e.g. CWSDPMI or DOS4GW. There may be some. Btw, NTVDM on Windows Vista (or was it 7?) and up also limits DPMI memory by default to something like 32MB (can be fixed by poking in the registry, AFAIR).

In reality I wouldn't want to deal with anything larger than 1GB in DOS at all.

Fourth, Smaller C isn't particularly good if you want to do lots of processing. It optimizes nearly nothing. And its malloc() is primitive and not suitable for many allocations (very likely much worse than Watcom's or DJGPP's). Smaller C's <string.h> functions (actually, most of the library functions) are also implemented in slow C.

So, yeah, you'll likely need a custom memory manager for your task.

Another thing to point out is that you shouldn't be using primitive algorithms when working with large text files. You definitely don't want to move 1GB of data in memory every time you insert or delete a character in the middle of a 2GB file. https://en.wikipedia.org/wiki/Gap_buffer may help here (it's a very simple data structure with a simple implementation).

Speaking of algorithms, if you can't allocate most of free RAM as one block due to fragmentation, you may want to represent your text as https://en.wikipedia.org/wiki/Rope_(data_structure). Then you can use non-contiguous buffers. The rope is more complex than the gap buffer and needs more memory for the meta data.

As for dpstub.asm, if you specify a large value in -maxheap (when linking), you should be able to malloc() that much memory at once unless the DPMI host has some limitation of its own.

Laaca

Czech republic,
14.02.2019, 17:40

@ alexfru

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

Yes! For text editor is definitely not needed the allocation of one huge buffer. Much better is to split the file into more buffers.
For example in my editor Blocek is every line on own buffer and these buffers are connected via structure called linked-list (in my implementation is it encapsulated double linked list).
Other possible solution is to connect lines by dynamic array (has some advantages and disadvantages over linked lists).

I didn't want to enter this discussion earlier because I considered it more as a theoretical discussion about allocating giant memory blocks.

---
DOS-u-akbar!

DosWorld

14.02.2019, 22:05

@ alexfru

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

> First of all
> ... Secondly
> Third
> Fourth

Thanks for hightlithing of problems, I have solution for ALL of this problems.

> So, yeah, you'll likely need a custom memory manager for your task.

I am already have.

> Another thing to point out is that you shouldn't be using primitive
> algorithms when working with large text files. You definitely don't want to
> move 1GB of data in memory every time you insert or delete a character

solved.

> Speaking of algorithms, if you can't allocate most of free RAM as one block
> due to fragmentation, you may want to represent your text as
> https://en.wikipedia.org/wiki/Rope_(data_structure).

It is solved - it work into my custom memory manager (something sort of "rope" technique). BUT, for this custom memory manager's sanbox- need memory. One big piece of memory. Only this is my problem.

---
Make DOS great again!

Carthago delenda est, Ceterum censeo Carthaginem delendam esse.

CandyMan

14.02.2019, 18:08
(edited by CandyMan, 14.02.2019, 18:22)

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

Tomasz Grysztar (author of flat assembler) wrote a text editor using a one-way list. Each line of text consists of blocks of "segments" of fixed length. The first block points to the next one and the next block to the next one. As a result, a continuous memory area is not needed.
In the long mode it is possible to obtain a continuous memory area of up to 64GiB thanks to mapping.
It is possible to download this text editor from (search for the file FASMD3X.EXE in archive FASM.7Z):
https://drive.google.com/drive/folders/0B_wEiYjzVkC0ZGtkbENENzF1Nms

Rugxulo

Usono,
26.10.2019, 18:13

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Post reply

Sorry I'm late! (FYI, I'm pretty much a noob, so I can only provide vague hints.)

> I had develop software (text editor) and I want open very huge files.

*nix "Ed" kept the whole file in memory! Fast but bad if you want to edit huge files (since it requires enough RAM, but that's less of a problem these days). For huge files, something like Sed was recommended (a line at a time!), and it only copied, didn't truly edit in-place.

In-place editing is almost impossible for plain text files anyways due to added or deleted lines or changing line length. If you wanted to hex edit a few bytes in-place (change 0xF to 0xA), that would work, but that kind of editing is somewhat rare (especially if done via batch, aka non-interactively).

> My problem is - dpmi hosts does not return memory as one chunk.
> So, watcom memory manager use memory only from one chunk (~64mb, with causeway extender - 220mb).

Be careful! Causeway supports virtual memory (but will swap out the *entire* amount, unlike CWSDPMI which only requests the extra it needs).

> How to received full memory as one chunk?

XMS is easily fragmented. EMS is probably worse. Virtual memory also makes everything more complicated!
So you can only grab a big chunk, not literally everything in one piece. Heck, BIOS holes make it
where you only get 2.9 GB or such on a 4 GB machine anyways (among other problems ...)!

* http://www.delorie.com/djgpp/doc/dpmi/api/310500.html
"Get Free Memory Information"
"Largest available free block in bytes"
"[T]he information returned by this function should only be considered as advisory."

> PPPS: I had receive big speed degradation for watcom's malloc, after 500k allocated strings.
> I think, it will be unusable after 1m malloc's.

I noticed back in 2010 that DJGPP 2.04's malloc was much better than OpenWatcom (at least, more efficient in space). DJGPP 2.05 switched to CBFalconer's nmalloc (faster for lots of free() calls). CWSDPMI r7 (2010) also added optional support for 4 MB pages (faster on supported 586+ cpus but can't swap, doesn't play well with EMM386 as it's non-standard size). More complications, but at least it doesn't overflow the page table (or whatever) in low/conventional memory as easily!

(historical only! but still interesting)
* http://www.delorie.com/djgpp/malloc/

> With DJGPP/CWSDPMI you should be able to malloc something around 2GB

CWS said to call sbrk() twice to get more than 2 GB total since DJGPP libc does assume 2 GB address space.
(Of course, he also offered to help anyone use 686+ PAE, if desired, but no one did.)

> djgpp build "hello world" >20s on my k6/300. I want see my texteditor.exe before I am die)

This is a separate problem. Not totally dire, but you have to use an appropriate GCC version for your machine (sadly). Also, enable Ultra DMA, use a software cache, use a RAM disk, use a well-written GNUmakefile (to avoid redundant compilations), use libraries (*.a instead of a billion separate *.o), adjust your PATH, split up larger files (-Q to see what funcs are being compiled, IIRC), etc. etc. etc. GCC 3.4.6 is probably max newest you should consider (oh, and you don't have to compile every module with optimizations nor everything with -O2, you can even preprocess and then compile later separately).

> First of all you should probably forget about editing any files larger than 2GB-1. That's a FAT16 maximum.
> FAT32 may let you have 4GB-1-long files, but...

DJGPP 2.04+ should (maybe?) support 4 GB files, but it wasn't widely tested (at least, not by me). I believe that can only be done with int 21h, 716Ch. I don't think NT/XP supported it properly either.

To be honest, unless you have external/third-party data, you shouldn't really have files that big (especially not for text). I'm saying don't do what's hard if you don't have to, then you don't have to support it. You don't have to support literally everything, only reasonable things. It's best to keep texts small and self-contained (especially source code). You can only see a screen of text at a time anyways, so it makes little sense to fill your whole memory up (except for speed, which doesn't matter except with huge batch operations, changing all matching lines or whatever).