DOS ain't dead

Can a DPMI program enter V86M with iretd instruction? (DOSX)

posted by tkchia , 21.01.2023, 14:06

Hello ecm,

> Thanks for the reply! Can a ring3 CS never iret to V86M?

Just to be sure(r), I looked up Intel's relevant documentation for the iret instruction.


PROTECTED-MODE:
    ...
    IF OperandSize = 32
            THEN
                        EIP := Pop();
                        CS := Pop(); (* 32-bit pop, high-order 16 bits discarded *)
                        tempEFLAGS := Pop();
            ELSE (* OperandSize = 16 *)
                        EIP := Pop(); (* 16-bit pop; clear upper bits *)
                        CS := Pop(); (* 16-bit pop *)
                        tempEFLAGS := Pop(); (* 16-bit pop; clear upper bits *)
    FI;
    IF tempEFLAGS(VM) = 1 and CPL = 0
            THEN GOTO RETURN-TO-VIRTUAL-8086-MODE;
            ELSE GOTO PROTECTED-MODE-RETURN;
    FI;
...
RETURN-TO-VIRTUAL-8086-MODE:
    (* Interrupted procedure was in virtual-8086 mode: PE = 1, CPL=0, VM = 1 in flag image *)
    (* If shadow stack or indirect branch tracking at CPL3 then #GP(0) *)
    ...
    shadowStackEnabled = ShadowStackEnabled(CPL)
    ...
    EFLAGS := tempEFLAGS;
    ESP := Pop();
    SS := Pop(); (* Pop 2 words; throw away high-order word *)
    ES := Pop(); (* Pop 2 words; throw away high-order word *)
    DS := Pop(); (* Pop 2 words; throw away high-order word *)
    FS := Pop(); (* Pop 2 words; throw away high-order word *)
    GS := Pop(); (* Pop 2 words; throw away high-order word *)
    ...

So it does look like only ring 0 code can iret into Virtual 8086 mode. Another interesting fact is that an iret to V86 mode will also pop additional registers from the ring 0 stack.

(Another way I can think of to transition from ring 3 PM to V86 mode, is to jump to a task gate or a TSS which has been set up for V86 mode. But in this case the ring 0 supervisor would need to have set up the TSS beforehand, to allow such a thing...)

Thank you!

---
https://gitlab.com/tkchia · https://codeberg.org/tkchia · 😴 "MOV AX,0D500H+CMOS_REG_D+NMI"

Complete thread:

• Can a DPMI program enter V86M with iretd instruction? - ecm, 20.01.2023, 17:52
  • Can a DPMI program enter V86M with iretd instruction? - alexfru, 21.01.2023, 03:35
    • Can a DPMI program enter V86M with iretd instruction? - ecm, 21.01.2023, 09:03
      • Can a DPMI program enter V86M with iretd instruction? - alexfru, 21.01.2023, 09:50
      • Can a DPMI program enter V86M with iretd instruction? - tkchia, 21.01.2023, 14:06
        • Can a DPMI program enter V86M with iretd instruction? - alexfru, 21.01.2023, 17:13