DOS ain't dead

CandyMan 30.08.2023, 18:15	FAT32 drive on GPT (Miscellaneous) Post reply
	I have a 500GB GPT hard drive with a FAT32 drive on it. What will I need for DOS to see this disk? Is it possible at all?

RayeR

CZ,
30.08.2023, 20:21

@ CandyMan

FAT32 drive on GPT

Post reply

> I have a 500GB GPT hard drive with a FAT32 drive on it. What will I need
> for DOS to see this disk? Is it possible at all?

AFAIK any current DOS kernel doesn't support GPT. Theoretically it shouldn't be so hard to add such support into FreeDOS, it would need only some minimal parsing of GPT to find FAT16/32 and read the LBA offset where the partition starts. Booting could be done via some GRUB with GPT support.
Alsou you coud save this troubles using hybrid MBR/GPT scheme as your disk size is below 2TB...

---
DOS gives me freedom to unlimited HW access.

bretjohn

Rio Rancho, NM,
31.08.2023, 01:27

@ RayeR

FAT32 drive on GPT

Post reply

> Alsou you coud save this troubles using hybrid MBR/GPT scheme as your disk
> size is below 2TB...

While this is possible, it is not a good idea. There is no generally accepted standard for creating a hybrid disk (e.g., Apple does/did it a little differently than anybody else) and you need make sure you ALWAYS change BOTH partition tables any time even the smallest change is made. Just as a simple example, there can be only one partition marked as bootable on a disk and GPT and MBR do this in completely different ways but they are still supposed to match (and they usually don't once a disk starts actually being used).

tom

Germany (West),
31.08.2023, 10:31

@ RayeR

FAT32 drive on GPT

Post reply

> > I have a 500GB GPT hard drive with a FAT32 drive on it. What will I need
> > for DOS to see this disk? Is it possible at all?
>
> AFAIK any current DOS kernel doesn't support GPT.

as said in 2022,

http://www.drivesnapshot.de/download/kernel.sys

should show and use existing FAT/FAT32 partitions if they are below 2TB.

of course the kernel must be loaded from floppy or GRUB

> Theoretically it
> shouldn't be so hard to add such support into FreeDOS, it would need only
> some minimal parsing of GPT to find FAT16/32 and read the LBA offset where
> the partition starts.
even practically it took no more then ~2 hours of work to implement this.

> Alsou you coud save this troubles using hybrid MBR/GPT scheme as your disk
> size is below 2TB...

it's MUCH better to have a kernel that supports GPT partitioning natively.

boeckmann

Aachen, Germany,
31.08.2023, 14:52

@ tom

FAT32 drive on GPT

Post reply

> > > I have a 500GB GPT hard drive with a FAT32 drive on it. What will I
> need
> > > for DOS to see this disk? Is it possible at all?
> >
> > AFAIK any current DOS kernel doesn't support GPT.
>
> as said in 2022,
>
> http://www.drivesnapshot.de/download/kernel.sys
>
>
> should show and use existing FAT/FAT32 partitions if they are below 2TB.
>
> of course the kernel must be loaded from floppy or GRUB
>
> > Theoretically it
> > shouldn't be so hard to add such support into FreeDOS, it would need
> only
> > some minimal parsing of GPT to find FAT16/32 and read the LBA offset
> where
> > the partition starts.
> even practically it took no more then ~2 hours of work to implement this.
>
>
> > Alsou you coud save this troubles using hybrid MBR/GPT scheme as your
> disk
> > size is below 2TB...
>
> it's MUCH better to have a kernel that supports GPT partitioning natively.

Are there any plans to incorporate this into the official Kernel? I did not find a hint of this in the Github Kernel repo, but guess you modified initdisk.c to scan for partitions on GPT.

What is your strategy regarding enumerating the drive letters? Old MBR style partitions first and GPT last? There seem to be different ways to do the enumeration, and I am wondering what would be the "right" way to do it. To stay compatible with old DOS versions, GPT last would be the most obvious I think. But that may clash with the way Windows enumerated the drives.

tom

Germany (West),
01.09.2023, 17:32

@ boeckmann

FAT32 drive on GPT

Post reply

> Are there any plans to incorporate this into the official Kernel?

I have not the faintest idea what the official plans are.

> I did not
> find a hint of this in the Github Kernel repo, but guess you modified
> initdisk.c to scan for partitions on GPT.
exactly.

>
> What is your strategy regarding enumerating the drive letters? Old MBR
> style partitions first and GPT last? There seem to be different ways to do
> the enumeration,

GPT disks last, although this might be a SYS configurable option.

>and I am wondering what would be the "right" way to do it.

> To stay compatible with old DOS versions, GPT last would be the most
> obvious I think. But that may clash with the way Windows enumerated the
> drives.

probably the booted disk should have C: ( or A:), then MBR disks, then GPT.

however I don't see any major trouble ahead in this as the machine has to be booted in an uncommon way, through USB or GRUB.

CandyMan 31.08.2023, 18:27 @ tom	FAT32 drive on GPT Post reply
	Unfortunately it doesn't work for me. Are you sure this is the correct link for this kernel? When booting from a USB drive, it displays the following message: "SVN build 2042 OEM 22 Sep 2016" and doesn't see the drive I'm talking about.

tom

Germany (West),
01.09.2023, 17:24

@ CandyMan

FAT32 drive on GPT

Post reply

> Unfortunately it doesn't work for me.
>
> Are you sure this is the correct link for this kernel? When booting from a
> USB drive, it displays the following message:
>
> "SVN build 2042 OEM 22 Sep 2016"
>
> and doesn't see the drive I'm talking about.

I have no idea what went wrong, but now there is http://www.drivesnapshot.de/download/kernel.sys and
http://www.drivesnapshot.de/download/initdisk.c

it dumps some debugging information about GPT disks, but then recognizes them as data partitions.

CandyMan 01.09.2023, 19:11 @ tom	FAT32 drive on GPT Post reply
	> I have no idea what went wrong, but now there is > http://www.drivesnapshot.de/download/kernel.sys > and > http://www.drivesnapshot.de/download/initdisk.c > > it dumps some debugging information about GPT disks, but then recognizes > them as data partitions. Thanks. I built the kernel and now it works. However, it also adds other disks from the GPT partition (also NTFS) with zero free space.

boeckmann

Aachen, Germany,
01.09.2023, 19:29

@ CandyMan

FAT32 drive on GPT

Post reply

> > I have no idea what went wrong, but now there is
> >
> http://www.drivesnapshot.de/download/kernel.sys
> > and
> >
> http://www.drivesnapshot.de/download/initdisk.c
> >
> > it dumps some debugging information about GPT disks, but then recognizes
> > them as data partitions.
>
> Thanks.
>
> I built the kernel and now it works. However, it also adds other disks from
> the GPT partition (also NTFS) with zero free space.

For GPT entries, there is no distinction between FAT and NTFS partitions. Both have partition type EBD0A0A2-B9E5-4433-87C0-68B6B72699C7.

If the non-FAT partitions do not get a drive letter assigned at boot, there is no way to reformat a such partition under FreeDOS with a FAT filesystem, at least with the format utility currently provided by FreeDOS.

Perhaps this behaviour should be configurable?

tom

Germany (West),
01.09.2023, 22:52

@ boeckmann

FAT32 drive on GPT

Post reply

> > I built the kernel and now it works. However, it also adds other disks
> from
> > the GPT partition (also NTFS) with zero free space.
>
> For GPT entries, there is no distinction between FAT and NTFS partitions.
> Both have partition type EBD0A0A2-B9E5-4433-87C0-68B6B72699C7.
>
> If the non-FAT partitions do not get a drive letter assigned at boot, there
> is no way to reformat a such partition under FreeDOS with a FAT filesystem,
> at least with the format utility currently provided by FreeDOS.
+1

a distinction could be made between "microsoft reserved", "uefi boot", and "plain data partitions", it's not clear to me what the behaviour should be.

>
> Perhaps this behaviour should be configurable?

possibly.
however since the users of this feature has currently a group size of 1 and he is able to recompile the kernel: let him check this out and tell us what he thinks.

CandyMan

02.09.2023, 00:14

@ tom

FAT32 drive on GPT

Post reply

> > > I built the kernel and now it works. However, it also adds other disks
> > from
> > > the GPT partition (also NTFS) with zero free space.
> >
> > For GPT entries, there is no distinction between FAT and NTFS
> partitions.
> > Both have partition type EBD0A0A2-B9E5-4433-87C0-68B6B72699C7.
> >
> > If the non-FAT partitions do not get a drive letter assigned at boot,
> there
> > is no way to reformat a such partition under FreeDOS with a FAT
> filesystem,
> > at least with the format utility currently provided by FreeDOS.
> +1
>
> a distinction could be made between "microsoft reserved", "uefi boot", and
> "plain data partitions", it's not clear to me what the behaviour should
> be.
>
> >
> > Perhaps this behaviour should be configurable?
>
> possibly.
> however since the users of this feature has currently a group size of 1 and
> he is able to recompile the kernel: let him check this out and tell us what
> he thinks.

An option to skip NTFS drives would be useful.

I'm not sure but the creator of fatfs http://elm-chan.org/fsw/ff/00index_e.html somehow distinguishes these types of file systems.

bretjohn

Rio Rancho, NM,
02.09.2023, 03:39

@ boeckmann

FAT32 drive on GPT

Post reply

> For GPT entries, there is no distinction between FAT and NTFS partitions.
> Both have partition type EBD0A0A2-B9E5-4433-87C0-68B6B72699C7.

In GPT, these are all classified as "Microsoft Basic Data" (which also covers exFAT). Even though there are plenty of bits to distinguish between the various partition types (way more than the single byte provided by MBR) MS decided for some reason to ignore that capability.

> If the non-FAT partitions do not get a drive letter assigned at boot, there
> is no way to reformat a such partition under FreeDOS with a FAT filesystem,
> at least with the format utility currently provided by FreeDOS.
>
> Perhaps this behaviour should be configurable?

In my USB program, by default it only assigns drive letters to compatible (FAT-formatted) partitions. To identify what's in the partition, though, it can't use the GPT entry. It must look at the data in the first sector (volume boot record) of the partition to see what's actually there (if anything).

This is different than MBR, where you can (at least theoretically) tell how a partition/volume is _supposed_ to be formatted (even if it hasn't yet been formatted) by the partition type value in the MBR.

The USB program also has a command-line option to mount non-compatible partitions (like NTFS or exFAT or non-formatted partitions) as drive letters so they can (at least theoretically) be (re)formatted by a DOS program. Of course, that's a dangerous thing to do unless you're _really_ sure you know what you're doing.

It seems like it might also be possible to create a FORMAT program that doesn't require a drive letter as input. I know there are some partition managers that will format the partitions they create (FDISK doesn't do that). Maybe that's the better approach, but it seems like it would also be possible to create a "smarter" FORMAT program that could make up for some of the shortcomings in FDISK?

boeckmann

Aachen, Germany,
02.09.2023, 12:54

@ bretjohn

FAT32 drive on GPT

Post reply

> In my USB program, by default it only assigns drive letters to compatible
> (FAT-formatted) partitions. To identify what's in the partition, though,
> it can't use the GPT entry. It must look at the data in the first sector
> (volume boot record) of the partition to see what's actually there (if
> anything).
>
> This is different than MBR, where you can (at least theoretically) tell how
> a partition/volume is _supposed_ to be formatted (even if it hasn't yet
> been formatted) by the partition type value in the MBR.

Exactly, the supposed file system type is encoded by the type number of the partition table entry. For NTFS partitions, it is a different number than for the FAT types (there are several). But even here, Microsoft started assigning several file system types to one type number, namely HPFS / NTFS / ExFAT, which all have partition type 7. Luckily, that is not of interest to DOS, as long as no one comes along and wants to implement ExFAT :-)

The actual FAT type "is determined solely by the count of clusters on the volume", and not by the partition type id, according to the Microsoft FAT specification [1], page 14.

> It seems like it might also be possible to create a FORMAT program that
> doesn't require a drive letter as input.

Yes, this proof has been provided several times :-)

For example, RANISH partition manager [2] is capable of it, but it is MBR only. I do not know any partition manager that a) supports GPT, b) runs under DOS and c) can format partitions.

> Maybe that's the better approach, but it seems like it would also
> be possible to create a "smarter" FORMAT program that could make up for
> some of the shortcomings in FDISK?

I think a "better" approach would be to store some FreeDOS specific hidden flag for the partitions the user does not want to see. Then everything else could stay as it is.

But it will be difficult, if not impossible, to store this information in the GPT itself without breaking compatibility. So some other location has to be found that a) does not break things and b) is accessible early in the boot phase while the Kernel enumerates the drives.

Bernd

[1] https://academy.cba.mit.edu/classes/networking_communications/SD/FAT.pdf
[2] https://codeberg.org/boeckmann/ranish/releases

ecm

Düsseldorf, Germany,
02.09.2023, 14:07

@ boeckmann

FAT32 drive on GPT

Post reply

> The actual FAT type "is determined solely by the count of clusters on
> the volume", and not by the partition type id, according to the
> Microsoft FAT specification [1], page 14.
>
>
> [1]
> https://academy.cba.mit.edu/classes/networking_communications/SD/FAT.pdf

This is what they claim in the specifications, but most drivers actually support "small FAT32" (with less than 65_536 data clusters) and only distinguish FAT32 from FAT12-or-FAT16 by testing whether the 16-bit Sectors per FAT field is zero. (The Amount of Root Entries field is also zero on FAT32, nonzero on FAT12-or-FAT16. It is uncertain what would happen if one of these fields is zero and the other nonzero.)

I also tested that EDR-DOS supports small FAT32, in https://www.bttr-software.de/forum/forum_entry.php?id=20608

I did quote some mtools discussion on this in https://github.com/dosemu2/fdpp/pull/202#issuecomment-1264597486 :

mtools just had this edge case fixed. The truth is that the word (16-bit) "sectors per FAT" field being zero indicates using both the larger EBPB and FAT32. It doesn't really matter how many clusters there are, you can create a FAT32 file system with less than 64 Ki clusters. That seems to be what most drivers, including Microsoft's, actually do.

Quoting from the mailing list for mtools: https://lists.gnu.org/archive/html/info-mtools/2022-09/msg00003.html

> >>> Actually, according to Microsoft's specification, number of FAT bits of
> >>> an existing filesystem is SOLELY determined by the number of clusters.
> >>> This applies to all *three* bit numbers: 12/16/32
>
> Yes, the (in)famous fatgen103.pdf document, retired from Microsoft's own
> site, but still available at
https://web.archive.org/web/20210723100623/http://...ba512-40e2-4cc9-843a-923143f3456c/fatgen103.doc
>
> It sternly says at page 14:
>
> "It is really quite simple how this works. The FAT type— one of FAT12,
> FAT16, or FAT32—is determined by the count of clusters on the volume
> and nothing else."
>
> But see below...:
>
> [...]
> > But as Alain pointed Microsoft FAT implementation (fastfat.sys) for
> > FAT32 detection does not use number of clusters, but rather explicit
> > marking of FAT32. Same behavior has Linux kernel implementation
> > (msdos.ko and vfat.ko) and same in dosfstools project. Hence this
> > non-standard FAT32 can be still created by mkfs.fat.
>
> Actually, what I meant in my previous mail was that *everything else*
> that is referred to in the FAT32 extended boot block (root directory
> location, etc.) would depend on this explicit marking
(fatLen/bigFatLen), but not the number of FAT bits themselves.
>
> However, your remark got me thinking, and so I tried it out on a Windows
> 10 VM.
>
> ... and lo and behold: the mere presence of a FAT32 extended boot block
> (as signaled by fat_len being 0) was enough to make it use 32 fat bits,
> even if the number of clusters was too low for FAT32.
>
> Sorry for having been to easily misled by an assertive, but nonfactual,
> Microsoft statement.
>
> Given this finding, I'll shortly make a new mtools release that will
> consider presence of FAT32 extended boot block as well for picking
> appropriate number of FAT bits.

---
l

boeckmann

Aachen, Germany,
02.09.2023, 15:17

@ ecm

FAT32 drive on GPT

Post reply

> > [1]
> > https://academy.cba.mit.edu/classes/networking_communications/SD/FAT.pdf
>
> This is what they claim in the specifications, but most drivers actually
> support "small FAT32" (with less than 65_536 data clusters) and only
> distinguish FAT32 from FAT12-or-FAT16 by testing whether the 16-bit Sectors
> per FAT field is zero. (The Amount of Root Entries field is also zero on
> FAT32, nonzero on FAT12-or-FAT16. It is uncertain what would happen if one
> of these fields is zero and the other nonzero.)

When "specification" meets reality :-D I am wondering if this small FAT-32 thing was an implementation side-effect or by intention.

The implementation I choose for RANISH is to explicitly forbid the creation of such partitions, even if the partition type was set to FAT-32. Better play save in this respect.

tom

Germany (West),
02.09.2023, 21:51
(edited by tom, 02.09.2023, 22:45)

@ bretjohn

FAT32 drive on GPT

Post reply

> It seems like it might also be possible to create a FORMAT program that
> doesn't require a drive letter as input.
absolutely.

FORMATing some disk area is like
READ/WRITE some stuff to the some sectors -
relative to the first sector of the partition

now if a new kernel learns new tricks to detect partitions - on GPT or/and even beyond 2TB - these FORMAT's and also file transfer will gain.

> I know there are some partition
> managers that will format the partitions they create (FDISK doesn't do
> that). Maybe that's the better approach, but it seems like it would also
> be possible to create a "smarter" FORMAT program that could make up for
> some of the shortcomings in FDISK?

FDISK is simply not able to handle GPT disks.

AND GPT disks don't come out of nowhere.
So I recommend to handle these problems in the Windows/Linux environments where these disks came from.

It's unlikely that FDISK will ever support GPT style partitioning, as this requires MUCH more effort to simply support detection.

RayeR

CZ,
02.09.2023, 21:14
(edited by RayeR, 03.09.2023, 01:58)

@ boeckmann

FAT32 drive on GPT

Post reply

> For GPT entries, there is no distinction between FAT and NTFS partitions.
> Both have partition type EBD0A0A2-B9E5-4433-87C0-68B6B72699C7.

How it can be so stupid? They expanded one byte partition type to GUID with bambilion of values but didn't differentiate fat and ntfs?

---
DOS gives me freedom to unlimited HW access.

0ffer

02.01.2024, 20:05
(edited by 0ffer, 02.01.2024, 20:15)

@ CandyMan

FAT32 drive on GPT

Post reply

> > I have no idea what went wrong, but now there is
> >
> http://www.drivesnapshot.de/download/kernel.sys
> > and
> >
> http://www.drivesnapshot.de/download/initdisk.c
> >
> > it dumps some debugging information about GPT disks, but then recognizes
> > them as data partitions.
>
> Thanks.
>
> I built the kernel and now it works. However, it also adds other disks from
> the GPT partition (also NTFS) with zero free space.

When booting with the shell=4dos.com parameter, the error message was displayed
Bad or missing Command Interpeter: dos.com
Enter the full shell command line:

When booting from ndos.com by Symantec Norton DOS (modified copies of 4DOS) everything is OK!

Kernel error message: Bad or missing Command Interpeter in ST-DOS replaced by Kernel prompt:
KERNEL>

Great idea for Free DOS kernel.sys

ecm

Düsseldorf, Germany,
03.02.2024, 19:08

@ 0ffer

FAT32 drive on GPT

Post reply

> When booting with the shell=4dos.com parameter, the error message was
> displayed
> Bad or missing Command Interpeter: dos.com
> Enter the full shell command line:
>
> When booting from ndos.com by Symantec Norton DOS (modified copies of 4DOS)
> everything is OK!
>
> Kernel error message: Bad or missing Command Interpeter in ST-DOS replaced
> by Kernel prompt:
> KERNEL>
>
> Great idea for Free DOS kernel.sys

I misremembered that you listed this bug with loading 4DOS on one of the FreeDOS mailing lists. In any case, I reported the bug to https://github.com/FDOS/kernel/issues/121

As a workaround, you can use shell=\4dos.com. If you already had the wrong line read, entering 4dos.com at the "Enter the full shell command line:" prompt also works.

---
l

RayeR

CZ,
02.09.2023, 21:19
(edited by RayeR, 03.09.2023, 01:58)

@ tom

FAT32 drive on GPT

Post reply

> as said in 2022,
>
> http://www.drivesnapshot.de/download/kernel.sys

Good work, how do we know about this fork? Is it based on latest code of official kernel? Why not to be official? I think this is important feature that people should know about.

Of course hybrid MBR is not ideal but only way for older OSes together with newer on the same disk.

---
DOS gives me freedom to unlimited HW access.

mceric

Germany,
30.08.2023, 22:54

@ CandyMan

FAT32 drive on GPT

Post reply

> I have a 500GB GPT hard drive with a FAT32 drive on it. What will I need
> for DOS to see this disk? Is it possible at all?

I think Bret's USB drivers already support GPT, so if you do not need to boot from it, you could put it into an USB enclosure and accept the slow I/O that you get because only USB 2 is supported?

Note that USB legacy support by the BIOS would make USB disks with MBR partitions visible to the kernel out of the box, but to use GPT, you would have to avoid them. Unless you have actual PS/2 keyboard and mouse, this means that you may have to tune the DOS USB driver configuration in a way which keeps those under BIOS legacy control while using the DOS driver for the disk, on another controller. Mainboards often have multiple controllers.

Also, you can probably boot DOS from many things if you use GRUB and a bootable MEMDISK boot floppy image. This would then have to contain drivers to help DOS to get access to the actual disk. One example would be loading NTFS4DOS from the virtual floppy to proceed using NTFS partitions in DOS, with only the boot floppy being FAT formatted.

---
FreeDOS / DOSEMU2 / ...

FAT32 drive on GPT (Miscellaneous)

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT

FAT32 drive on GPT