DOS ain't dead

I played around with lDebugX and the dpmimini example in my local Linux box's dosemu2, and found that an iretd (or o32 iret) which sets the VM flag apparently tries to load a stack pointer from the PM stack, I get a Stack fault if this cannot succeed. In case it does succeed, the DPMI host apparently returns a General protection fault.

Is it possible to enter V86M using just iretd in DPMI protected mode? If yes: How much does the DPMI host get confused by this, and can it avoid that?

---
l

> I played around with lDebugX and the dpmimini example in my local Linux
> box's dosemu2, and found that an iretd (or o32
> iret) which sets the
> VM flag apparently tries to load a stack pointer from the PM stack, I
> get a Stack fault if this cannot succeed. In case it does succeed, the DPMI
> host apparently returns a General protection fault.
>
> Is it possible to enter V86M using just iretd in DPMI
> protected mode? If yes: How much does the DPMI host get confused by this,
> and can it avoid that?

If you run CWSDPMI in ring 0, you should definitely be able to IRET into virtual 8086 mode. But then you'll also need to prepare things to go back.

Also, real-mode code may be executed in virtual 8086 mode.

What are you trying to achieve?

Thanks for the reply! Can a ring3 CS never iret to V86M?

> What are you trying to achieve?

Nothing, just idle curiosity. The most use I would have for this is hardening the debugger to avoid this situation or handle it in some way.

---
l

> Thanks for the reply! Can a ring3 CS never iret to V86M?

Looks like, the only option is if already in virtual 8086 mode, which is CPL=3 too.

Hello ecm,

> Thanks for the reply! Can a ring3 CS never iret to V86M?

Just to be sure(r), I looked up Intel's relevant documentation for the iret instruction.


PROTECTED-MODE:
    ...
    IF OperandSize = 32
            THEN
                        EIP := Pop();
                        CS := Pop(); (* 32-bit pop, high-order 16 bits discarded *)
                        tempEFLAGS := Pop();
            ELSE (* OperandSize = 16 *)
                        EIP := Pop(); (* 16-bit pop; clear upper bits *)
                        CS := Pop(); (* 16-bit pop *)
                        tempEFLAGS := Pop(); (* 16-bit pop; clear upper bits *)
    FI;
    IF tempEFLAGS(VM) = 1 and CPL = 0
            THEN GOTO RETURN-TO-VIRTUAL-8086-MODE;
            ELSE GOTO PROTECTED-MODE-RETURN;
    FI;
...
RETURN-TO-VIRTUAL-8086-MODE:
    (* Interrupted procedure was in virtual-8086 mode: PE = 1, CPL=0, VM = 1 in flag image *)
    (* If shadow stack or indirect branch tracking at CPL3 then #GP(0) *)
    ...
    shadowStackEnabled = ShadowStackEnabled(CPL)
    ...
    EFLAGS := tempEFLAGS;
    ESP := Pop();
    SS := Pop(); (* Pop 2 words; throw away high-order word *)
    ES := Pop(); (* Pop 2 words; throw away high-order word *)
    DS := Pop(); (* Pop 2 words; throw away high-order word *)
    FS := Pop(); (* Pop 2 words; throw away high-order word *)
    GS := Pop(); (* Pop 2 words; throw away high-order word *)
    ...

So it does look like only ring 0 code can iret into Virtual 8086 mode. Another interesting fact is that an iret to V86 mode will also pop additional registers from the ring 0 stack.

(Another way I can think of to transition from ring 3 PM to V86 mode, is to jump to a task gate or a TSS which has been set up for V86 mode. But in this case the ring 0 supervisor would need to have set up the TSS beforehand, to allow such a thing...)

Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"

> So it does look like only ring 0 code can iret into Virtual
> 8086 mode. Another interesting fact is that an iret to V86
> mode will also pop additional registers from the ring 0 stack.

Yep. It's similar to transferring control between different CPLs in regular protected mode.

> (Another way I can think of to transition from ring 3 PM to V86 mode, is to
> jump to a task gate or a TSS which has been set up for V86 mode. But in
> this case the ring 0 supervisor would need to have set up the TSS
> beforehand, to allow such a thing...)

Yes, this too.