DOS ain't dead

I just upgraded my old 1TB HDD to 4TB (beside a SSD for system, attached to intel P67 onboard SATA 3.0 controller in IDE mode). I like to have most of the capacity accessible for my legacy OSes (DOS, Win9x, NT, XP...). On old 1TB I had nonLBA ext. partition with logical drivers that is accessible in MS-DOS 6.22 and the rest was big FAT32. Now I hit the 2TB barrier of MBR. But I wonder that I hit even lower. Neither FD-FDISK nor WinXP disk manager is able to create partition larger than 1,6TB. WTF? Why not the whole 2TB? Then I found that is seems my BIOS is too stupid to report limited EDD capacity that then fools the FDISK and even the Windows (I would expect WinNT/XP use their own disk drivers and don't rely on EDD values. Here is it:

EDD version: 2.1
EDD features: Fixed disk access, EDD support
EDD flags: 0000h
Physical drive #1 geometry LBA sectors: 3519069872 (1803 GB), sector=512 B
Device Path Information: absent
Device Parameter Table Extension pointer: F000:8420h
DPTE revision: 1.1 [OK]
DPTE flags: 2897h
Controller I/O base address: 1F0h, control port: 3F6h
IRQ: 14, DMA channel: 2
PIO mode: 4, DMA mode: 0, Ultra DMA: enabled
32-bit transfer mode: enabled, LBA: enabled, LBA48: enabled
ATA read/write multiple command max sectors: 16, enabled
ATAPI: no, uses command packet interrupt: no, nonremovable media
Host Protected Area: inactive

As you can see LBA sectors: 3519069872 (1803 GB) instead of real drive capacity that is exactly 7814037168 sectors. So it's an evident EDD bug - 7814037168 mod 2^32 = 3519069872 so BIOS do only 32b arithmetics...
Maybe it would be easier to fix some table in memory after boot than BIOS patching but less universal. For WinXP there seems it may be a solution to replace some *.sys drivers from Win 2k3 SP2 ther should correctly handle LBA48. Linux (even old 32b Debian) seems to habdle it correctly.

Then need to deal with 2TB MBR barrier. I'm thinking to create a hybrid MBR with GPT. The GPT would mirror the existing MBR-defined partitions and adds one GPT-only partition behind 2TB barrier that will be accessible only for newer OSes (probably NTFS). I think it should be safe when MBR and GPT will keeps synced. Linux gdisk should have options to create this. Win98 should have working LBA48 fix for ESDI_506.PDR, WinXP should use GPT via Win2k3 drivers but not sure about NT4. Any idas?

---
DOS gives me freedom to unlimited HW access.

Hi!

> Physical drive #1 geometry LBA sectors: 3519069872 (1803 GB), sector=512 B

> As you can see LBA sectors: 3519069872 (1803 GB) instead of real
> drive capacity that is exactly 7814037168 sectors. So it's an evident EDD
> bug - 7814037168 mod 2^32 = 3519069872 so BIOS do only 32b arithmetics...

In theory, int 13, ah=48h, dl=drive, ds:si pointer to buffer is supposed to return a 64-bit value at offset si+10h. I can easily imagine a BIOS failing to return all bits here, though. Which tool did you use to view the EDD data?

> Then need to deal with 2TB MBR barrier...

As people already started to work on this, I suggest to contribute to DOS GPT support if you really want > 2 TB for DOS.

Does hybrid GPT-MBR really work? Or is the MBR on a GPT disk just a dummy to keep non-GPT OS from causing problems?

---
FreeDOS / DOSEMU2 / ...

> As people already started to work on this, I suggest to contribute to DOS
> GPT support if you really want > 2 TB for DOS.

This is the correct approach.

> Does hybrid GPT-MBR really work? Or is the MBR on a GPT disk just a dummy
> to keep non-GPT OS from causing problems?

Short answer is no, it doesn't work. It's called a Protective MBR and it's just there, as the name implies, to try and keep non-GPT OS's from screwing everything up. There were a few attempts to try and have hybrid MBR's, but there were never any standards so there were incompatibilities and problems everywhere. It's just a bad idea.

If you try and be hybrid you need to keep the MBR and GPT sides synchronized properly so they are telling the same stories regarding the partitions, and the other OS's won't be sympathetic. Bottom line is that you can't avoid implementing GPT, which I think is part of your reason for wanting to go hybrid.

Also, it is possible to read GPT partitions in DOS if things are handled correctly. I do it with my USB drivers. But booting from GPT is another matter altogether -- that is what I think some people are working on.

But there are people which use hybrid MPR/GPT partitioning.
These partition tables change only very rarely, only if you do some partition size tools. Otherwise they are static (maybe except partition boot flags) si the MBR and GPT should not desynchronize.
I see bigger problem in the boot manager itself which must jump between MBR booting code and GPT booting code.
Also do not forget that Rayer wants not only DOS on his new harddisk but also Win98 and WinNT.

---
DOS-u-akbar!

> In theory, int 13, ah=48h, dl=drive, ds:si pointer to buffer is supposed to
> return a 64-bit value at offset si+10h. I can easily imagine a BIOS failing
> to return all bits here, though. Which tool did you use to view the EDD
> data?

Many years ago I wrote a small utility EDDINFO. I looked in my source and found that I ignored the high Dword and printed only the low DWord because of old Borland C don't support printing format %llu (is there some specific format string for BC or any?). So I quickly modified the code to just print both DWords separately but I still get 0 in high Dword so yes, the BIOS/EDD is really buggy/too old...

I already tried to replace disk.sys, partmgr.sys and mountmgr.sys from Windows Server 2003 SP2 as suggested on many forums but nothing has changed! After some digging info through AI it told me, that in my specific case - SATA controller in IDE compat. mode when it use standard IDE driver (instead of e.g. AHCI driver or RAID driver), then the LBA is passed to the NT kernel via NTDETECT.COM / NTLDR from EDD and it has HIGHER priority than what driver might detect via ATA identify. I got a suggestion that I can try patch EDD runtime via GRUB4DOS and then chainload NTLDR - grub command geometry (hd1) should detect the correct LBA, then if drivers from W2K3 will work as expected I should see full capacity...

I also tried to look at the HDD from Win7-x64 and the same tools like 32-bit HDTune reported 4TB correctly, so this tools rely on what they get from the OS. Using Win7-x64 Disk manager I was able to create one smaller partition ~370GB that filled the rest from 1,6 to 2TB limit. It sees next 2TB of unallocated space but don't allow to create partition there while still having MBR-only. Maybe if I filled the values in MBR manually... But first I'd like to solve EDD...

> As people already started to work on this, I suggest to contribute to DOS
> GPT support if you really want > 2 TB for DOS.

I don't need so much to access that 2nd partition over 2TB, it would be probably NTFS anyway. But just need that DOS wouldn't complain and I can access all below 2TB.

> Does hybrid GPT-MBR really work? Or is the MBR on a GPT disk just a dummy
> to keep non-GPT OS from causing problems?

There are 2 kind of MBR when GPT involded. Normally all tools generate protective MBR that is just dummy, useless for DOS. But Linux gdisk can (as I belive) generate valid MBR from GPT partitions and vice versa automatically so you could have up to 4 partitions defined same as in MBR and GPT so both legacy and modern OSes should access it....

---
DOS gives me freedom to unlimited HW access.

> But there are people which use hybrid MPR/GPT partitioning.

I belive they are.

> These partition tables change only very rarely, only if you do some
> partition size tools. Otherwise they are static (maybe except partition
> boot flags) si the MBR and GPT should not desynchronize.

Yes, I don't plan to change partition layout after it's done. The bootloader is placed on smaller SSD so nothing should randomly change the MBR and GPT on data HDD.

I tried also Win98SE and DOS 7.1 - they can see both 1,6TB and ~370GB partitions below 2TB without problem. But WinNT4 has some serious problem - they didn't corretly recognize DOS extended partition neither big FAT32 1,6TB partition and show total garbage in disk manager (240GB SSD is displayed correctly). Even I'm using UniATA driver...

---
DOS gives me freedom to unlimited HW access.

> ... (maybe except partition boot flags) ...

And you don't think which partition is marked as the boot partition matters (and both the MBR & GPT must agree on this)? You are correct that the partitions themselves rarely change after initial setup, but which one is bootable can change a lot. Especially if you want to try and run multiple OS's off the same disk, and some of them only understand one or the other (MBR or GPT). And even if an OS understands both they will normally only use one or the other (probably GPT) and not cross-check to make sure they agree. You _can_ get it to work but it is asking for trouble and probably will trash your data at some point.

If I understand Rayer correctly he does not plan to use this 4TB disk for booting. He will boot from smaller disk and this one will be for data only.

---
DOS-u-akbar!

> If I understand Rayer correctly he does not plan to use this 4TB disk for
> booting. He will boot from smaller disk and this one will be for data only.

Yes, I have 240GB SSD as a system disk for those zoo of DOSes, Win and Linux... I normally don't need change boot flags, NTLDR and GRUB4DOS are enough for me to boot it all...

---
DOS gives me freedom to unlimited HW access.

> Now I hit the 2TB barrier of MBR.
I'm still trying to figure out how to bypass 2 TB MBR limit! I think with 4k sectors, one should be able to access 16TB on an mbr partition, but both bios and DOS must support this. Raw 4k sector mode, not hybrid 512/4k mode. So far no one has taken this approach. I'm not sure if any BIOSes even support CSM mode / MBR mode with native 4k sectors. I know no DOS format utilities do. They all assume 512-byte sectors. Sadly. Even FreeDOS which has been updated very often doesn't support native 4k sectors on an MBR partition, booting to baremetal DOS.

And bypassing 2gb/4gb filelimit; I know we've got NTFS and exFAT readers for DOS, but all DOS programs and utilities I've tried do not read past 2gb-1 and only one or two will even read to 4gb-1 in native baremetal DOS 7.1 even on NTFS / exFAT drives.

I would indeed be great to get 16tb in DOS / MBR. RayeR's modulo error is typical of the buggy BIOSes and incompatibility we see in trying to press the limits. The partitions don't have to be bootable, just recognizable when one boots with a FAT / FAT32 usb memory stick, which is of course < 2TB and MBR.

bret, when you say your USB utilities enable GPT partition reading in DOS, this is only for USB devices, correct? You do not patch DOS to enable GPT reading of all non-bootable partitions attached to the system (including SATA / AHCI internal drives?)

> bret, when you say your USB utilities enable GPT partition reading in DOS,
> this is only for USB devices, correct? You do not patch DOS to enable GPT
> reading of all non-bootable partitions attached to the system (including
> SATA / AHCI internal drives?)

That is correct. The USB driver add entries in the various DOS internal tables that keep track of drive letters and such, responds to INT 13h requests (including the 64-bit extensions), and inserts block device drivers into the device driver chain. It does not modify anything about DOS itself -- just creates new entries in the appropriate places.

In addition, since somebody brought it up, the USB driver will work with sector sizes other than 512 bytes if the DOS supports it (which MS-DOS does if you set it up correctly but AFAIK none of the others DOS's do). Unfortunately with DOS, you need to add support for non-512-byte sectors not just to the kernel but also to ALL of the disk-related utilities (and there are a BUNCH of those). And even though the MS-DOS kernel supports sector sizes other than 512 bytes, I don't think any of the MS utilities do (though a few might).

I'm also adding support for CD/DVD/BD to the USB driver, but still have some more work to do. What I have got to work (that I think is pretty cool) is accessing a DVD-RAM disk, which (like all common optical media) has a 2k sector size, formatted with FAT32. I can mount this in MS-DOS as a regular drive letter and MS-DOS treats it as a FAT32 hard drive with 2k sectors and doesn't know it's a DVD-RAM. I used Windows to format the DVD-RAM since there aren't any DOS utilities that will do it (and it's possible that newer versions of Windows won't do it either). Unlike most read-write optical media which have a pretty limited number of writes, a DVD-RAM is actually designed to be written to many times -- it's sort of like a really big floppy drive. Unfortunately, nobody seems to make new DVD-RAM media any more and it's really hard to find.

> But WinNT4 has some serious problem -
> they didn't corretly recognize DOS extended partition neither big FAT32
> 1,6TB partition and show total garbage in disk manager (240GB SSD is
> displayed correctly). Even I'm using UniATA driver...

Win2000 was the first NT to support FAT32. So NT 4 is too old.

> Win2000 was the first NT to support FAT32. So NT 4 is too old.

I tried to update to the latest UniATA 0.47B from 2021 and now NT4 sees partitions correctly. Essential is also to NOT use WinInternals FAT32.SYS that incorrectly handles big FAT32 partitions and may lead to data loss. Instead there's some Bearwindows's alternative FASTFAT.SYS driver that can handle 1,6TB FAT32 partition properly. For me better worked the version for NT3.51 than one for NT4. So I got the same level of support on DOS 7.x, FreeDOS, Win98, NT4 and XP - can use all up to 2TB...

---
DOS gives me freedom to unlimited HW access.

I don't want to go via bigger sectors way - it's too trouble with many systems. Even the HDD pretend to use 512B sectors and no idea how to change it. It's viable for external USB drives where it's not problem to have bigger sectors...

I would like to prove if NT/XP belives in BIOS/EDD values forwarded at boot so I'm looking for a way how to hook int13 and modify returned EDD (turned out that GRUB4DOS can't do this)- is there some existing code that can be reused for this? I also could temporarly try the AHCI mode if it changes something. In theory, in AHCI mode the UniATA driver should rely more on ATA identify than EDD in IDE mode but still overflow can happen somewhere else in the system...
Also I think that AHCI BIOS may be different implementation than when IDE mode is used so there's a little chance it may have better EDD...

---
DOS gives me freedom to unlimited HW access.

>
> I would like to prove if NT/XP believes in BIOS/EDD values forwarded at boot
> so I'm looking for a way how to hook int13 and modify returned EDD (turned
> out that GRUB4DOS can't do this)- is there some existing code that can be
> reused for this?

You could use an Extension for lDebug targetting my pre-boot loader. It's described some in http://svardos.org/?p=forum&thread=1764705038

fdbplace.eld is the only loader Extension that I created yet. To modify it for your use, you may want to add some way to identify its presence. In the int 13h handler then modify the 13.48 return buffer as you desire.

If you can't assemble or modify the loader and extension I may look into drafting something like this.

---
l

I tried temporary switch to AHCI and it has different AHCI BIOS that supports EDD 3.0 and reports the correct value of total LBA sectors. WinNT4 with UniATA booted in AHCI mode without a problem bu they still see only 1,6TB so even when EDD fixed it is still truncated somewhere else so no need further EDD experiments...

I also tried to manually create another primary partition that fills the rest from 1,6TB to the end (under W7-x64) and found there's visible aliasing behind 2TB of absolute LBA address! So even that Windows sees this partition normally it would cause data loss when I try to write there. The same in Win98 and DOS7/FreeDOS. So it doesn't make sense to expose 2nd partition in MBR to legacy OSes but safer would be if it will be defined only in GPT for newer OSes only (via hybrid MBR).

---
DOS gives me freedom to unlimited HW access.

> I also tried to manually create another primary partition that fills the
> rest from 1,6TB to the end (under W7-x64) and found there's visible
> aliasing behind 2TB of absolute LBA address!

I assumed you wanted to report FFFF_FFFFh sectors for the size, and use partitions that completely fit in the 32-bit space (first nearly 2 TiB).

The aliasing beyond 2 TiB is to be expected if you use a driver that doesn't support 33-bit LBA. That is likely all drivers, except for lDOS boot, lDOS iniload, and bootable lDebug. (Excluding lDOS kernel just like all kernels I'm aware of.)

---
l

Have you tried the HDAT2 program?
https://www.hdat2.com/
It has a buch of options. Maybe it is possible to switch off the shadowing.

---
DOS-u-akbar!

I'm a bit disappointed of various partition resizing tools (to expand existing FAT32 from 1,6TB to 2TB). I tried many ~10 tools and was surprised about the problems. Many of them just reported invalid partition or invalid size of current partition and so didn't allow to resize. Some others started resizing but hanged with zero disk activity after some time or ended with some error while partition (and data) left untouched. I run them under w7-x64 to have proper 64b arithmetics. The last I tried was Paragon Partition Community Edition x64 [ https://www.paragon-software.com/free/pm-express/# ] relative new SW - it resized the partition after cca 1/2 hour but screwed the data. Root dir looks fine but when I opened subdirs there was a garbage. Even it screwed my extended partition with FAT16 logical drives a crazy way they can be accessed under Windows and linux but DOS6/7 throws error when acesing them. Record in partition table seems OK. So I just going to wipe all that crap, repartition manually and load from a backup that takes several hours on such amount of data...
So just beware such tools promising "partition resize without data loss". In the past I used Powerquest Partition Magic (the latest 8.05 for DOS) to resize FAT16/32 partitions and I never had a problem but this old tool can't handle such big drivers.

---
DOS gives me freedom to unlimited HW access.

Unfortunatelly my hybrid MBR attempt failed. Crucial problem is that Win7-x64 and XP32 (with W2k3 drivers) recognize GPT only if the MBR contains protective partition type EEh in the 1st partition table entry and the rest of part.table MUST be cleared to zeros. If there's added any other partition or if EEh partition is not located in the 1st entry then Windows just ignore GPT and use MBR instead. Only Linux can read such disk correctly prefering the GPT regardless other partitions in MBR. DOS itself seems has no problem with EEh partition, just ignore it, old fdisk show unknown partition.
I also tried to use GRUB4DOS to subvert GPT MBR image from file to memory when windows boots but it failed when windows loader switch to pmode and vanish faked MBR from memory (and read real one from HDD).

It would be possible to do manual MBR swap at boot but it can easily leave MBR with unwanted version when system crashes and it's not swapped back.

Also there's still 2TB limit in WinXP32 even with W2K3 drivers that was successfully enabled to accept GPT but it failed to access the last partition beyon 2TB to the end while 32b Linux and W7-x64 can access it correctly.

So I'm disappointed. I cannot find any hacked windows drivers. Maybe I could try some IFS or imaging driver that can mount any physical partition according to given LBA instead of partitions provided by system, not sure if such exist...

---
DOS gives me freedom to unlimited HW access.

This site contains a little bit of info on the various partition types that can be in an MBR:

https://thestarman.pcministry.com/asm/mbr/PartTypes.htm

EEh (GPT Protective MBR) is the "standard" GPT, and there is also EFh (EFI/UEFI System Partition) which seems to only be used by Windows and I'm guessing somehow interacts with the EEh.

Officially, EEh is supposed to be a single partition that encompasses the _entire_ drive, or at least the first part of the drive that the MBR can handle (~2TB if the sector sizes are 512 bytes). The other entries are supposed to be 0. The GPT is _supposed to be_ the only place that contains real partition information. Various vendors will handle an "incorrect" protective MBR's differently, as you have discovered. A hybrid MBR is _not_ universal and is _almost guaranteed_ to cause you grief at some point. Even in my USB disk drivers, when I see the EEh I do a little bit of checking (though far from complete) to see if it really looks like a valid protective MBR, and use the GPT instead of the MBR. If I see the EEh and it doesn't look like a valid protective MBR, I assume it's a regular MBR and the EEh is a mistake or something weird is going on and it's not really GPT.

I know that hybrid MBR is some kind of grey zone, just hoped it would work well as under Linux. To precise previous observation about Win7, I found that they probably checks only one Byte of MBR partition entry - the partition type. If I zero this Byte for remaining 3 partition entries (leaving CHS and LBA values in place) then Win7 recognizes valid GPT. So instead of swapping whole MBR it would be enough to just change these 3 Bytes...

Another possible approach:
do you know about some program like OSFMount or ImDisk but that can mount a part of physical disk instead image file? AI halucinated me that it could be possible selecting physical disk in file open menu and entering specific LBA of start partition and size but I didn't see such option - only mounting image files. But technically it is be possible...

---
DOS gives me freedom to unlimited HW access.

Oh sh... even worse with Win7 and hybrid MBR. I found that Windows is actively corrupting the GPT at LBA 2 (offset 400h) whenever I run command "attributes volume clear hidden" in diskpart - this is necessary to unhide my legacy MBR partitions because Windows hides automatically them at boot when detect hybrid MBR with EEh entry. So definitely not possible both legacy and GPT partition together. I hate this windows behavior to silently write something somewhere in a God/good intention. All that crap started by Win95 unexpectedly writting to floppies...

So I wiped the GPT and roll back to old good MBR. I just defined 4th partion entry to some dummy to prevent fdisk and similar to create new partition on unused space. Legacy OSes just ignore it.
Under Win7x-64 I use IMDISK to mount the hidden (undefined) NTFS partition, a bit rude with hardcoded offset but I don't expect to change partitions layout further...
imdisk -a -P -f \\.\PhysicalDrive1 -b 2199022206976 -x 63 -y 255 -o hd -m N:
Under WinXP it's not possible to enter 64b offset (exptected that)...

Linux is smart it already has offset option so I can just do
mount -o offset=$((512*4294965248)) /dev/sdb /mnt/n
or place in my /etc/fstab
/dev/sdb /mnt/n ntfs,iocharset=iso8859-2 loop,offset=2199022206976

---
DOS gives me freedom to unlimited HW access.