DOS ain't dead

CM, All : I'm uploading a new test keyboard driver including completely revised memory management - putting an end to the (relative) memory waste when loaded low. Had to redesign from scratch since I can't lay my hands on old code of mine. Not a bad thing per se, and with the help of some aspirin...

For the TESTING COMFORT of non-French users, I made KBZERO.EXE a dummy which will NOT mess with your existing keyboard translations but otherwise behaves like the real thing and importantly will allow you, valued Testers, check memory management under the most varied circumstances...

Download : <http://www.rootshell.be/~ninho/KBTEST15.ZIP> 11,361 bytes

Test package includes :

1- ready made KBFR and KBZERO executables, v. 1.5

Use : {lh} (KBFR | KBZERO) {/F /U}

/F : suppresses int 2F resident hook, sparing 32 bytes (and prevents unloading)
/U : unload if possible

Upper memory is no more used automatically, instead loadhigh is permitted.

2- source assemblies KBINST2.ASM KBRES2.ASM KBFR.INC

Most new and hopefully interesting stuff in the KBINST2 module, installation part. I've left a lot of "int 3"s if you want to fast-forward in a debugger.

Additional notes for the valiant who would "make" from sources :

- if building the "dummy", set ?KBZERO = TRUE in "include" file,
otherwise (to build the French kb) : ?KBZERO= FALSE

- assemble with MASM (I used 6.15) : ml -c kbinst2.asm ^ ml -c kbres2.asm

Under true DOS, must provide your DPMI server (Japh's e.g. as: DPMILD32 masm -c ....)

- link, using your favored OMF linker (I used tlink.exe).
The important is to present the objects in the following order, imperative! (opposite to the order required for linking previous versions!)
tlink KBINST2+KBRES2, my_keyb.exe

- optional bit of hokus-pokus to adjust and shrink the header of the resulting EXE (left out for lack of room in the margins of this article).

Cheers, have Fun !

---
Ninho

> Upper memory is no more used automatically, instead loadhigh is permitted.

Without looking at the program this tells me it uses a boring method.

I'll review this properly later.

> > Upper memory is no more used automatically, instead loadhigh is
> permitted.
>
> Without looking at the program this tells me it uses a boring method.

Well, boring or not it's worth mentioning the scheme is aimed at loading *low* efficiently which was not done previously. It happens to work also when loading high, almost by accident :=)

The previous scheme, load high from low, has been removed from this test on purpose in order not to obscure the source. We may put it back in later, for better fit in very small UMBs.

> I'll review this properly later.

Please do, at your pace and schedule.

--
Ninho

> For the TESTING COMFORT of non-French users, I made KBZERO.EXE

Appreciated.

> assemble with MASM (I used 6.15)

Would it work with JWASM too? I don't really care right now, though if I ever wanted to assemble it...

> important is to present the objects in the following order

Shouldn't it be possible to make it independent of the order if you relocate the entire process? Depending on the memory layout it might be possible.

> optional bit of hokus-pokus to adjust and shrink the header of the resulting EXE

Just use flat .COM executables if the .EXE header size worries you


Now here's what I determined during testing:

As you noted in KBRES.ASM, the "trans" function can be inlined to save some bytes.

The "drvflgs" variable in the Int2F resident part is apparently not used.

I do not see why the "psp" variable has to be in the resident part, either. It's apparently used with 21.49 by the deinstall function, however, isn't it sufficient if the function determines itself which memory block to free?

I see that you do relocate your PSP and all. Why didn't you choose to change the UMB link status and allocation strategy while allocating the memory block for the resident part then? I consider a program's ability to relocate itself into a UMB a feature.

It's true that DOS doesn't use the PSP beyond offset 50h. I don't see a reason not to relocate the entire PSP though.

I prefer activating the new PSP via process termination. Theoretically, if any resident program associates resources with our PSP address, my method might prevent leaks. In practice it probably doesn't matter though.

You use 21.4A to resize the part of the process which contains the old PSP. This means your new PSP is in free memory after DOS's MCB resizing finished. (It doesn't matter that even the DOS algorithm could overwrite your new PSP since any resident code interrupting your program's run at that time is allowed to allocate and overwrite the memory block anyway.) The best solutions to this are in my opinion: (a) do not split memory blocks at all, instead, relocate the entire process to an entirely new memory block, or (b) if you choose to split memory blocks, do not use this unsafe 21.4A method; instead, prepare the new MCB yourself and modify your old MCB to point to that one. The new MCB must be created with the letter read from the old one, and you have to set the old one's letter to 'M' afterwards.

You free your process's memory block partially after relocating the PSP. However, you have not relocated your program code. Your installer executes in free memory after that. This is far worse than the previous issue because you do issue 21.48 (to allocate the TSR's memory block) yourself while executing code in free memory. If DOS happened to allocate a memory block overlapping your program the latter might get overwritten. Note that the critical part of your program which must not be overwritten here includes the TSR's image that you copy into its block later.

The 21.48 request to allocate memory for your resident program definitively needs an error check.

Why do you set the TSR's MCB to "SC", owner 8? I prefer to set my TSR's MCBs so I can distinguish them: self-owned with a more or less descriptive name.

Related insignificant nitpick: Your new PSP's MCB's name is uninitialized. That's of course not important since that name is never visible except in debuggers.

Even if you fixed the issues with using free memory, your method might fail to place the TSR at the optimal position if your resident memory requirements grew larger than the size of the old process's and PSP's memory block that you free. To overcome this limitation, you have to relocate the entire process to a new memory block.

Other insignificant nitpicks: Why do you calculate the size of the TSR at run time? I see that you have to dispatch between the value for the removable TSR and the non-removable one, but that still doesn't explain why you don't use the assembler to do the shifting and such at build time. Similarly, why did you hardcode the "32 bytes" message? Just let the assembler calculate the memory block size difference between the removable and non-removable TSR and write that into the message.

---
l

(no text)

So many, detailed, comments, wow !

Just to put things into perspective, since I started modifying Japheth's program incrementally, I'm like a painter retouching an existing picture rather than one drawing onto a fresh canvas. I just sit in front of my frame, uh, screen and type code in the text editor, without separate notes of any kind. Now, the mods have taken us much farther from the starting point than I first envisioned. And since it is also my pleasure to deliver things even in an unfinished (yet I hope somewhat usable) state, I am aware of imperfections like unused, left over, variables (I've removed some but not all).

>> assemble with MASM (I used 6.15)

> Would it work with JWASM too? I don't really care right now, though if I
> ever wanted to assemble it...

No idea, welcome to try. I went out of my way myself to download MASM, since this is what Japheth wrote for, usually I program in TASM.

>> important is to present the objects in the following order
> Shouldn't it be possible to make it independent of the order if you
> relocate the entire process? Depending on the memory layout it might be
> possible.

I find it more elegant to relocate the resident in one move, and not have to ask for an auxiliary memory block, yielding complications if DOS couldn't allocate one (since the resident part is so small it may be not be a problem in practice but it might when installing high into a small umb for instance).

>> optional bit of hokus-pokus to adjust and shrink the header of the
> resulting EXE

> Just use flat .COM executables if the .EXE header size worries you

There are several ways to skin that cat. Yep, my pref would have gone to a .com too. Especially using MASM as opposed to TASM, I'm not always comfortable with the assembler's choice of segment: (or group:) origin for offset calculation especially when we start moving segments around... As a consequence I feel obliged to doublecheck the binaries in the debugger; I must say this MASM 6.15 has suprised me favorably as being not too "quirky",
I remembered older versions of the microsoft assembler doing strange unpredictable things sometimes :=)

> Now here's what I determined during testing:

> The "drvflgs" variable in the Int2F resident part is apparently not used.

True : legacy.

> I do not see why the "psp" variable has to be in the resident part, either.
> It's apparently used with 21.49 by the deinstall function, however, isn't
> it sufficient if the function determines itself which memory block to
> free?

The badly named "psp", really the resident segment value, is returned by the communication routine (int 2F). Legacy again.

Altogether that whole int 2F hook is doing little useful and it even renders uninstallation more problematic than need be; it is going to disappear, clearing the above 2 remarks.
I'll have another method for uninstalling, plus an option for disabling without removal.

> I see that you do relocate your PSP and all. Why didn't you choose to
> change the UMB link status and allocation strategy while allocating the
> memory block for the resident part then? I consider a program's ability to
> relocate itself into a UMB a feature.

This version was delivered to test and evaluate the "load low" option. It had all UMB management removed.

> It's true that DOS doesn't use the PSP beyond offset 50h. I don't see a
> reason not to relocate the entire PSP though.

Elegance for one. And memory usage, as I would have to enlarge the stack correspondingly (hence the whole executable memory footprint) if I were to copy the entire PSP. 50h bytes may be even too much for our short lived PSP here ;=) I guess 40h bytes would suffice (we don't call DOSVER :)

Ah, now comes the more serious stuff :

> I prefer activating the new PSP via process termination. Theoretically, if
> any resident program associates resources with our PSP address, my method
> might prevent leaks. In practice it probably doesn't matter though.

That's splitting hair. Would a foreign TSR hijack our PSP ? Any interrupting TSR which allocates resources for itself must activate its *own* PSP first, it would seem. It's not like we are *calling* a TSR as a service

> You use 21.4A to resize the part of the process which contains the old PSP.
> This means your new PSP is in free memory after DOS's MCB resizing
> finished.

Yep, well spotted!

> (It doesn't matter that even the DOS algorithm could overwrite
> your new PSP since any resident code interrupting your program's run at
> that time is allowed to allocate and overwrite the memory block anyway.)

I'm convinced. I could do direct MCB fixes, like you suggest under b) later, and like, I think, I did in earlier attempts years back. But this time I found it interesting to try and put DOS functions to work whenever possible rather than direct manipulation of structures.

This bug I think is simply the order of the steps. Simple fix (referring to the numbered comments in the code) : do steps 3.2 & 3.3 /before/ step 3. In fact I can't believe I couldn't see this before submitting.

> The best solutions to this are in my opinion: (a) do not split memory
> blocks at all, instead, relocate the entire process to an entirely new
> memory block,

avoided for reasons already told

> (b) if you choose to split memory blocks, do not use this
> unsafe 21.4A method; instead, prepare the new MCB yourself and modify your
> old MCB to point to that one....

Easily done, yet I believe 21/4A will be safe if done in the revised order.

> You free your process's memory block partially after relocating the PSP.
> However, you have not relocated your program code. Your installer
> executes in free memory after that. This is far worse than the previous
> issue because you do issue 21.48 (to allocate the TSR's memory block)
> yourself while executing code in free memory. If DOS happened to allocate a
> memory block overlapping your program the latter might get overwritten.
> Note that the critical part of your program which must not be overwritten
> here includes the TSR's image that you copy into its block later.

This is serious. My own use of 48h is not a danger, size of the requested block is accounted for so it won't trash anything still needed.
OTOH you're right asynchronous allocation (by a TSR ?) could, in theory, ruin the game after we free most of our own container (at #3.4)

I was aware of this but after (too little) thought, it didn't seem to matter... and since in practice nothing exploded I counted on the review process, honest !

Step 3.4 (freeing our old, previously shrinked, container) appeared necessary because allocation of the resident block (step 3.5) must be allowed to step over the start of our program. That was the central idea. I wanted to avoid having to request extra memory, as already said.

Thinking aloud, I could, instead of freeing all of my container, free the first (size of resident) bytes only, by direct MCB handling, that would reduce the surface of the problem you have uncovered but not solve it in theory. Is there a way to make : (free, then allocate) into a "critical section" of sorts ? I don't suppose "CLI" would suffice, would it ?

> The 21.48 request to allocate memory for your resident program definitively
> needs an error check.

I don't think so, since the steps I've taken ensure this allocation can't fail. An error check can't hurt anyway (you saw the comments) ;=)

> Why do you set the TSR's MCB to "SC", owner 8? I prefer to set my TSR's
> MCBs so I can distinguish them: self-owned with a more or less descriptive
> name.

This was left as per Japheth. If it turns out later on that we have to walk the MCBs for some reason, we may revise it. As we don't ATM it's a no brainer.

> Related insignificant nitpick: Your new PSP's MCB's name is uninitialized.
> That's of course not important since that name is never visible except in
> debuggers.

and except when debugging, the PSP in question lives only for a fraction of a second anyway :=)

> Even if you fixed the issues with using free memory, your method might fail
> to place the TSR at the optimal position if your resident memory
> requirements grew larger

Can't happen, not a limitation for this application.

> Other insignificant nitpicks: Why do you calculate the size of the TSR at
> run time? I see that you have to dispatch between the value for the
> removable TSR and the non-removable one, but that still doesn't explain why
> you don't use the assembler to do the shifting and such at build time.

Code reuse again !

> Similarly, why did you hardcode the "32 bytes" message?

It's only printed by the test Kbzero version, I think.

> assembler calculate the memory block size difference between the removable
> and non-removable TSR and write that into the message.

Moot point since that particular option is going away and the resident size will be fixed, probably.


Did I thank you for the review, comments and time passed ? Much appreciated!

---
Ninho

> > Shouldn't it be possible to make it independent of the order if you
> > relocate the entire process?
>
> I find it more elegant to relocate the resident in one move,

I don't think this can be properly done; not generally, anyhow. If your TSR is smaller than the PSP, or the TSR fits into the memory block that you would split off after the PSP's relocation (this can include some of your program code), then this might be sufficient. This appears to me to be more complicated and restricted though.

> and not have
> to ask for an auxiliary memory block, yielding complications if DOS
> couldn't allocate one (since the resident part is so small it may be not be
> a problem in practice but it might when installing high into a small umb
> for instance).

Just set a proper low-only, last-fit allocation strategy (21.5803.bx=1 then 21.5801.bx=2) and you'll allocate a proper temporary memory block in all cases. I found this method to be the least complicated one. It always causes the least fragmentation, assuming the largest low memory block is at the top of the low memory area, which is a reasonable assumption.

Note that a process loaded from a flat .COM executable into the low memory area has to resize its process memory block first since otherwise the subsequent 21.48 call usually fails to allocate a memory block in the expected position.

Note that relocating something to the top of the low memory area may overwrite a shell's transient code which would require the shell's resident code to reload that from the executable, XMS, or whatever they use for swapping. This can be avoided by allocating, or trying to allocate, a temporary memory block that is X bytes too large for the installer's purposes and not modifying the additionally allocated part; where X is a number assumed to be larger than the shell's transient code size. Note that this is entirely a performance optimization; the only disadvantage of not doing this, or choosing an X that is too small, is that the shell has to reload its code.


> The badly named "psp", really the resident segment value, is returned by
> the communication routine (int 2F).

It's not, that handler reasonably returns its own CS. The only time the resident "psp" variable was accessed in an installed TSR was from the deinstaller.


> This version was delivered to test and evaluate the "load low" option. It
> had all UMB management removed.

However, see my remark regarding complete process relocation to a temporary memory block. For optimal results, independently of whether LH was used, one has to specify both strategy and UMB link status using 21.58 anyhow. Otherwise, the temporary allocation (even if allocated with strategy 2, supposedly, quoth RBIL, "Low memory last fit") might just end up in a UMB. So some of the UMB management would have to be included for that anyway.


> > It's true that DOS doesn't use the PSP beyond offset 50h. I don't see a
> > reason not to relocate the entire PSP though.
>
> Elegance for one.

Don't see it here.

> And memory usage,

I see. But I would consider such constraints a disadvantage of storing the relocated PSP in space that is a part of the process's existing stack.

> enlarge the stack ... (hence the whole executable memory footprint)

Yes? I don't think your executable..., oh. Got me there. When you say "executable" I always think of the program's file. I say "process" or "program" when I mean the live program's memory usage.


> That's splitting hair.

Told you it's theoretical.

> Would a foreign TSR hijack our PSP ? Any
> interrupting TSR which allocates resources for itself must activate its
> *own* PSP first, it would seem.

I thought of some resident code tracking some kind of resource for every process, like, say, the environment. Now the normal DOS environment memory block is well known to everyone so it's easy to free that specifically, but if there was, say, a "hidden environment"-like resource associated with each process, then 21.4C relocation avoids leaking such resources. I don't know of any software creating such hidden resources, but it's a possibility.

More generally, I think changing the active PSP via 21.50, or direct DOS internal data manipulation, is fine if you go back to the previous PSP later on, like, say, a TSR or debugger setting its own PSP with 21.50 while it interrupts the currently running program. When it's done, the interrupting code uses 21.50 again to reset the active PSP. However, both the debugger and the TSR should ensure that their PSPs as well as the running program's PSP are properly terminated by process termination DOS functions.

> It's not like we are *calling* a TSR as a service

That's true. And as I said I'm not aware of any concrete program creating hidden resources for other processes on its own.


> But this
> time I found it interesting to try and put DOS functions to work whenever
> possible rather than direct manipulation of structures.

That's generally useful to keep in mind, but I think splitting memory blocks cannot be properly done without MCB manipulation if you want to keep the upper part.

Back when I did more of this TSR stuff (2009?) I found that the least complicated method remained complete process relocation. Of course you have to modify MCB names and owner fields as usual, but you don't have to create an MCB from scratch, as you have to do if you want to split a memory block and keep the upper part.

> This bug I think is simply the order of the steps. Simple fix (referring to
> the numbered comments in the code) : do steps 3.2 & 3.3 /before/ step 3. In
> fact I can't believe I couldn't see this before submitting.

No, you don't appear to understand the problem. If you modified the installer like that, it would still depend on (a) data in free memory to stay unchanged and (b) the free memory MCB to be large enough, when, in fact, DOS could have created this MCB with a data size of zero.

Now about (b), some will tell you that a DOS that doesn't behave exactly like MS-DOS in memory allocation functions isn't compatible enough. I don't think so. I don't want to depend on the assumption that the created MCB is of a proper size. In any case, resident code can interfere with this in any imaginable way still - that's of course a consequence of (a) as well.

Another more common problem with (b) is that in fact the free MCB created in the 21.4A request might be way too large. I don't remember the details, but it might be the case that MS-DOS doesn't collect free memory blocks after 21.4A. Again though, I don't want to consider this a DOS compatibility requirement.

> Easily done, yet I believe 21/4A will be safe if done in the revised
> order.

I do not see how it can be used safely in this way.


> Step 3.4 (freeing our old, previously shrinked, container) appeared
> necessary because allocation of the resident block (step 3.5) must be
> allowed to step over the start of our program. That was the central idea. I
> wanted to avoid having to request extra memory, as already said.

You'll have to split your memory block at a lower address for this to work properly, in such a way that the TSR image and the remaining installer code will remain inside allocated space. This will make it depend more on the program's memory layout, or generally speaking it makes things complicated.

> Is there a way to make : (free, then allocate) into a "critical
> section" of sorts ?

No, and that's not necessary if you properly keep all the data you need inside allocated space.

> I don't suppose "CLI" would suffice, would it ?

It wouldn't suffice if you called 21.48 since DOS or other Int21 software might enable interrupts or modify MCBs itself.


> steps I've taken ensure this allocation can't fail

I don't think you can really ensure it won't fail given the possibilities of other TSRs. Of course failure is improbable here!


> > Why do you set the TSR's MCB to "SC" ...
>
> This was left as per Japheth. If it turns out later on that we have to walk
> the MCBs for some reason, we may revise it. As we don't ATM it's a no
> brainer.

It's more convenient for the user to see what program uses what memory block in MEM's display.


> > Similarly, why did you hardcode the "32 bytes" message?
>
> It's only printed by the test Kbzero version, I think.

True, the French text says "32 octets" instead


> Did I thank you for the review, comments and time passed ? Much
> appreciated!

Hah, I sure wouldn't do this if I wasn't interested in it myself.

---
l

> Hah, I sure wouldn't do this if I wasn't interested in it myself.

Kind of you! Then let me summarize the result of my "meditating" over your sharp analysis.

> I found that the least
> complicated method remained complete process relocation.

I admit, concede and agree that relocating the entire process to the top of 'conventional' memory as a first step will be the simplest, safest, if a little... boring? way to get the job done.

Out of interest however still seeking a reasonably simple way to save my solution from the evil TSR problem. As we saw previously :

>> Is there a way to make : (free, then allocate) into a "critical
>> section" of sorts ?
> No (...)
>> I don't suppose "CLI" would suffice, would it ?
> It wouldn't suffice if you called 21.48 since DOS or other Int21 software
> might enable interrupts or modify MCBs itself.

Exactly. But if we walked the MCB chain and fixed them MCBs by hand without calling DOS, under CLI it would be completely correct, right ? Alternatively, what about temporary cheating the INDOS flag "up" (instead of CLI) ?

---
Ninho

> I admit, concede and agree that relocating the entire process to the top of
> 'conventional' memory as a first step will be the simplest, safest, if a
> little... boring? way to get the job done.

Touché.

Though I guess optimal placement is still an improvement over boring traditional TSRs (21.31, shudder) which don't relocate anything and fragment and waste memory.


> But if we walked the MCB chain and fixed them MCBs by hand without
> calling DOS, under CLI it would be completely correct, right ?

Assuming you kept track of what data to leave unmodified. But in that case you would have to replace all of DOS's 21.48 code (including UMB handling) and would have to free the MCB yourself instead of using 21.49 as well.

As usual with critical sections requiring IF zero, it would also fail to ensure your data's integrity if running the program in a debugger.

> Alternatively, what about temporary cheating the INDOS flag "up" (instead
> of CLI) ?

I don't think that would really be of any use. It might prevent interference by some TSRs which check for InDOS before issuing 21.48/.49/.4A but it's not a good solution. (In fact, I think protecting critical sections by the InDOS flag is generally not a good idea and should only be done if really necessary.)

All in all, design it so that there are no critical sections during which it leaves data in free memory. Then you won't have to think up kludges to protect such critical sections either.

---
l

>> Alternatively, what about temporary cheating the INDOS flag "up"> (instead
>> of CLI) ?

> I don't think that would really be of any use. It might prevent
> interference by some TSRs which check for InDOS before issuing
> 21.48/.49/.4A but it's not a good solution. (In fact, I think protecting
> critical sections by the InDOS flag is generally not a good idea and should
> only be done if really necessary.)

Aren't you pushing it for the sake of argument ? I mean, your hypothetical TSR monitoring other programs' use of resource cannot even try to interfere in the middle of a DOS operation (except for well known cases). Even if its purpose is only to record, not modify things, doing so while INDOS would me meaningless.

> All in all, design it so that there are no critical sections during which
> it leaves data in free memory. Then you won't have to think up kludges to
> protect such critical sections either.

Now the price being cases where it won't be able to run for lack of memory. Of course that is more theoretical than practical for our little keyboard helper!

Just FYI finalising will be delayed by important personal matters, meanwhile potential users can rely on the last version adverstised in the Announcements section.

---
Ninho

> > I don't think that would really be of any use. It might prevent
> > interference by some TSRs which check for InDOS before issuing
> > 21.48/.49/.4A but it's not a good solution. (In fact, I think protecting
> > critical sections by the InDOS flag is generally not a good idea and
> > should only be done if really necessary.)
>
> ...

I wasn't referring to the hypothetical TSR associating resources with other processes here, I was referring to very real TSRs which might pop up pretty much any time interrupts are enabled and which could interfere with your memory reallocation if you relied on every TSR to check for InDOS before it modifies MCBs one way (Int21) or another (direct access).

> Now the price being cases where it won't be able to run for lack of memory.
> Of course that is more theoretical than practical for our little keyboard
> helper!

If your program takes up that much memory swap it to a file

Seriously though, the only case in which the complete process relocation would fail was if there wasn't enough space available to relocate the complete process. If you trim the process to be relocated as much as possible first, you'll only have to allocate 80 bytes for the reduced PSP and enough space for the remaining installer code (including the image to install later on) plus some stack space. Assuming a 512 byte image (I think your installed TSR currently is about that large) the relocated installer process can take up less than 1024 bytes.

> meanwhile potential users can rely on the last version advertised in the
> Announcements section.

Yes, I'd agree that this version is the best for end users right now.

---
l

>> ...

> I wasn't referring to the hypothetical TSR associating resources with other
> processes here, I was referring to very real TSRs which might pop up pretty
> much any time interrupts are enabled and which could interfere with your
> memory reallocation if you relied on every TSR to check for InDOS before it
> modifies MCBs one way (Int21) or another (direct access).
>

No, this time I won't agree. Any TSR which accesses DOS structures such as MCBs in memory asynchronously (even as to read only), without first checking INDOS, is just buggy and potentially breaks pretty much everything, not just KBFR v1.5... won't you agree ? I repeat myself, you seem to be stretching your (otherwise good) points to an extent which is threatening to disserve your case.

---
Ninho

Just a quick passing by to announce I /fixed/ my low memory loading scheme using a bit of stolen free time. It should now be fully correct and TSR resistent, thanks in large part to CM's thought provoking comments. This is achieved without the need to first relocate the installer to the top of low memory.

Main good news :

- never execute from freed memory (duh!)

- freeing/splitting of mem blocks done by direct MCB access, protected
by CLI. (No unsafe int 21 4A nor 49)

- allocation of the final block for TSR placement is done by DOS 21/48, rather than duplicating its function in my code, still inside the section protected by CLI.
It doesn't matter that int 21 will internally do STI, since interrupting TSRs will be prepared to recognise when they were invoked in DOS (having hooked int 21 presumably) and behave accordingly. This is unlike my previous idea ("fake" INDOS while not actually in DOS, which was feeble and indeed unnecessary).

As soon as I've more free time I'll pack and upload an updated testing pack.

See you all

---
Ninho

> CM's thought provoking comments.

Agree.

> - never execute from freed memory (duh!)

Agree.

> - freeing/splitting of mem blocks done by direct MCB access, protected
> by CLI. (No unsafe int 21 4A nor 49)

Agree.

> - allocation of the final block for TSR placement is done by DOS 21/48,
> rather than duplicating its function in my code, still inside the section
> protected by CLI.
> It doesn't matter that int 21 will internally do STI, since
> interrupting TSRs will be prepared to recognise when they were invoked in
> DOS (having hooked int 21 presumably) and behave accordingly. This is
> unlike my previous idea ("fake" INDOS while not actually in DOS,
> which was feeble and indeed unnecessary).

Possibly disagree. Have to see source code.

> As soon as I've more free time I'll pack and upload an updated testing
> pack.

Agree.

> See you all

Agree.

> Possibly disagree. Have to see source code.

Here you have it as written this morning, raw cut&paste - the whole installation section, I can't seem to be able to copy the whole source into this box nor is it necessary - refer to the v 1.5 for context. It's unpolished, untested and just to give you crumbles to attack until I give you more food :=)


;-------------------------------------------- installation
align 16

; everything before this point /might/ be trashed by further processing,
; offset (from initial PSP) MUST be >= max resident's size (currently 1F0h)
.ERRNZ $-Firstb LT 0F0h ; resident size minus 100h bytes
NMCB DB "M" ; or "Z"
NOwner DW 0 ; owner
NParas DW 0 ; size
DB 11 dup (?)

NPSP DW 40h/2 dup (?) ; new mini-PSP while installing
Shft EQU 10h+(NPSP-firstb) shr 4 ; #paras from old to new PSP
Shftb = Shft shl 4 ; #bytes dø
doinstall:
if ?ATCHECK
; (skipped)
endif ; ?ATCHECK

; KBINST3 : enhanced conventional memory use.

; : copy MCB and PSP (to NMCB)
; : adjust seg reference in copied PSP
; : adjust new MCB, mark self-owned.
; : activate new PSP
; : split our own container at newMCB
; : *CLI*
; : *manually* mark our low part, and environment (if any) MCBs owner-free
; : DOS-allocate resident block (from low) see note 2 below
; : *STI*
; : copy resident part down into new block.
; : set ptrs to previous ints 15 & optional 2F (in new code)
; : activate new ints
; : done! back to caller (DOS) by int 21/4C

;Notes:
;1- we never execute from freed memory,
;2- we use int 21/48 rather than duplicate its function. We are not worried that
; DOS will STI internally, TSRs interrupting int21 should know how to behave.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; 1 - copy MCB+PSP out of the way (10h+40h bytes !) -> NMCB,NPSP
mov cx,(10h+40h)/2 ; size of MCB+size of mini-PSP
CLD
MOV AX, CS:[psp]
DEC AX ; AX: seg oldMCB
MOV DS, AX ; Assume DS: oldMCB
mov BX, AX
add BX, Shft ; BX: seg newMCB
MOV ES, BX ; Assume ES: newMCB
sub si, si
sub di, di
rep movsw ; copy 5 paras, from MCB (really)
inc BX ; BX: seg newPSP
mov es:[10h+36h], BX; adjust seg(handles pointer)

; 2 - prepare new MCB, keeping copied M or Z signature
mov es:[1], BX ; owner (self)
sub word ptr es:[3], Shft ; size

; 3 - make copied PSP active :
mov ah, 50h
int 21h ; ASSUME BX:newPSP

; 4 - *CLI*. Adjust Original MCB to cover only up to NewMCB, and mark it FREE.
CLI ; protection
mov byte ptr ds:[0], 'M' ; signature
mov word ptr ds:[3], Shft-1 ; paragraphs
mov word ptr ds:[1], 0 ; mark free !

; 5 - Mark our environment block, if any, FREE :
mov CX,DS:[10h+2Ch]
jcxz @F
DEC CX
mov ES, CX
MOV word ptr ES:[1], 0 ; owner=None
@@:
; 6 - ask DOS for a suitable block; make it owned by system
; calculate resident size.
if ?SUPPINT2F
mov BX, offset EndofInt2F
test byte ptr inst2f,01
jnz @F
endif
mov BX, offset EndofInt15
@@:
mov al, BL
shr BX, 4
and al,0fh
jz @F
inc BX
@@:
mov prglen, BX ; local store (BP rel)

mov AH, 48h ; DOS allocate for resident code
int 21h ; will STI internally, not a problem
; JC XXXX ; ... error? shouldn't happen :=) TODO
...rec mov CS:[psp], AX ; to be kept by resident copy
push AX
dec ax ; MCB of new block
mov ES, AX
mov bx, 0008
mov es:[0001], bx ; owner = System
xor si,si
xor di,di
mov word ptr es:[BX],"BK" ; indicator = "KBF"
mov word ptr es:[BX+2]," F"
pop ES

; 7 - Copy resident part to its final place :
INT 3
; ES: newly allocated for Resident Segment
mov BX, KBRES ; segment
mov DS, BX ; source
mov cx,prglen ; paragraphs
shl cx,3 ; -> words
rep movsw

; 4.5 - set pointers to previous ints in resident code :
push ES
pop DS ; resident segment

ASSUME DS: KBRES ; in fact, DS: *new* Resident Segment
mov ax, 3515h
int 21h
mov word ptr orgint15+0, bx
mov word ptr orgint15+2, es

if ?SUPPINT2F
test byte ptr inst2f,01
jz @F
mov ax, 352Fh
int 21h
mov word ptr orgint2f+0, bx
mov word ptr orgint2f+2, es
ASSUME DS: NOTHING

@@:
endif

; 5 - Activate interrupt hook(s) :
if NOT ?KBZERO
if ?KBFR ; make sure dead key related flags are reset :
PUSH DS
push 40h ; BIOS data seg
pop ds
ASSUME ds:BIOSSEG
and Cflags, not 3
POP DS
ASSUME DS:nothing
endif
endif

INT 3
; ASSUME DS: new (resident) segment
mov dx,offset introu15
mov ax,2515h
; push ds
int 21h
; pop ds
if ?SUPPINT2F
test byte ptr inst2f,01 ; local storage
jz @F
mov dx,offset introu2f
mov ax, 252fh
int 21h
@@:
endif
INT 3

; 6 - Installation done, return to DOS.

mov ax,4c00h
int 21h

main endp
; ---------------------------------------------------------------------

---
Ninho

This looks good. Some comments:

You could free the environment block before entering the "critical" part, and use 21.49 as usual.

If you changed your MCB splitting code to this, I don't think you would need to clear the interrupt flag:

mov word [3], newsize
mov byte [0], 'M'

You could also free the MCB using 21.49 then. The rationale behind this is that if the MCB contained 'M' initially, then the first instruction will resize the MCB appropriately. The second will effectively always be a NOP then, since the resized MCB must still point to the one you created. (It could hardly contain 'Z' suddenly.) If the MCB however contained 'Z' initially, then you're either at the very end of the MCB chain or you are modifying the MCB that points to the first UMCB (at segment 9FFFh if 640 KiB of low memory are in use). In both cases, making the MCB point to your new MCB while temporarily leaving the letter as 'Z' has none or no critical drawbacks. (If it pointed to the first UMCB and a TSR interrupts between the two instructions and that TSR allocates memory using DOS, DOS's behaviour with allocating UMBs would be erratic - however, no critical failure would occur.) At least that was my reasoning for executing this code without clearing the interrupt flag when I wrote MCB splitting code.

You could of course still include the clearing of the interrupt flag.

cli
mov word [3], newsize
mov byte [0], 'M'
sti

Exchanging the instructions in your code to this advantageous order where the MCB's size is adjusted first is still useful for tracing that code in a debugger, as debuggers disable any protection you get by clearing the interrupt flag.

Assuming your TSR image is still stored behind the installer and assuming that your stack is behind that, you do not need to leave the interrupt flag disabled during your 21.48 call, with or without this adjustment. As you noted, you do not execute in free memory and do not depend on data in free memory. Therefore, protecting the 21.48 call is not necessary. Attempting to justify your (arguably ineffectual) protection scheme isn't necessary either.

Other than that, you still seem to keep the "psp" variable around in the TSR and you still set the TSR's MCB's owner to 8 (so that MEM probably won't show your name properly). The variable is an optimization that isn't required, and the owner is a minor display issue that doesn't do any harm otherwise. So, as I said, this looks good.

---
l

snippety snip....
> Assuming your TSR image is still stored behind the installer and assuming
> that your stack is behind that, you do not need to leave the interrupt flag
> disabled during your 21.48 call, with or without this adjustment. As you
> noted, you do not execute in free memory and do not depend on data in free
> memory. Therefore, protecting the 21.48 call is not necessary. Attempting
> to justify your (arguably ineffectual) protection scheme isn't necessary
> either.

I think it is useful to attempt to protect (CLI or otherwise) the whole section which {frees some of our memory, then DOS-allocate} and the rationale for that in fact was suggested by our prior discussions. Indeed if some interrupt-activated resident chimes in after freeing but before allocating, it could itself allocate from one or both of the blocks we've just given back ! That would not be an error, as far as DOS is concerned, but could spoil our hopes for optimal placement of our own resident (this seems to apply equally to your system, as far as I've understood it without actually seeing an example).

Putting the section under the protection of CLI will prevent this from happening, because as I commented, although DOS /will/ STI (a good thing anyway), a well written TSR that would be invoked while int 21 has been entered shall either postpone its popping up until better times, or if it does activate it shan't try to reenter DOS or access internal DOS structures (unless first swapping, not just the usual SDA but pretty much the whole DOS "list of lists").

> So, as I said, this looks good.

I hope so. Returning to the protection problem, which is what kept me awake for part of last nights, there would be at least 2 other, more radical ways to solve it if the solution above was not enough : 1- do the allocation by hand, without calling DOS (tedious.) An advantage is that we can program the search so it "knows" about the blocks we are ready to provide, so that we could do without actually freeing them and without the CLI.
Alternate method 2- don't CLI around the section, but mask interrupts at the PICs. I don't like this too much. We would unfortunately need to mask all IRQs, because while the keyboard, timer and RTC irqs are fixed, the mouse (or other contraption) IRQ whose number is impredictable.

Anyway and fortunately, the current proposed method looks "just right",
if implemented properly :=)

---
Ninho

> Indeed
> if some interrupt-activated resident chimes in after freeing but before
> allocating, it could itself allocate from one or both of the blocks we've
> just given back ! That would not be an error, as far as DOS is concerned,
> but could spoil our hopes for optimal placement of our own resident (this
> seems to apply equally to your system, as far as I've understood it without
> actually seeing an example).

Yes, it does apply to my system. Yes, it could temporarily allocate memory in such a way as to prevent optimal allocation.

However, as you said, that would not be an error (not just for DOS, in fact, it wouldn't be an error for anyone). The system cannot possibly crash that way. The worst that could happen is that the TSR allocated so much memory that our 21.48 call would fail. (I told you you should check for an error there!) Otherwise, it might still cause less optimal placement as you said.

While the possibility exists, this is also somewhat unlikely.

Additionally, if the TSR's allocation could be said to be permanent in some way, then our 21.48 would not make a "less optimal" allocation - because, in fact, the TSR's allocation would have changed the memory layout then. Hence I said that specifically a TSR temporarily allocating memory could cause less optimal placement.

Because the system isn't unstable after such an incident, the user could easily unload our TSR and install it again - probably achieving optimal placement this time.

All in all, I am aware of such possibilities but do not consider them a big issue. And it's hard to completely work around them.

> Putting the section under the protection of CLI will prevent this from
> happening, because as I commented, although DOS /will/ STI (a good thing
> anyway), a well written TSR that would be invoked while int 21 has been
> entered shall either postpone its popping up until better times, or if it
> does activate it shan't try to reenter DOS or access internal DOS
> structures (unless first swapping, not just the usual SDA but pretty much
> the whole DOS "list of lists").

I have to admit you are right in that TSRs mustn't muck with MCBs (too much) while in DOS since DOS could be processing a 21.48 or 21.4A call. However, observe:

int21:
; IISP header or whatever here.
        ...
; Dispatcher here.
        ...
; Some kind of processing here.
        ...
; Ah! I need some memory.
; Should be safe to call DOS because I'm processing Int21 here
; and the service I've hooked is one that would enter DOS anyway.
        mov ah, 48h
        mov bx, somesize
        int 21h
; etc.

So resident code hooking Int21 might access memory precisely because of your Int21 call.

> which is what kept me awake for part of last nights,

I hope this didn't inconvenience you!

> Returning to the protection problem, ...
> there would be at least 2 other, more radical ways
> to solve it if the solution above was not enough : 1- do the allocation by
> hand, without calling DOS (tedious.) An advantage is that we can program
> the search so it "knows" about the blocks we are ready to provide, so that
> we could do without actually freeing them and without the CLI.

However, resident code could still prevent optimal allocation if it did its allocations earlier and if the optimal place for the TSR isn't where our environment or our process is placed. (In fact, I don't think you can prevent resident code interference with any loading scheme. Even if it was built into DOS. To ensure optimal placement in any case, you would have to ensure that all resident code would never allocate memory.)

> Alternate method 2- don't CLI around the section, but mask interrupts at
> the PICs. I don't like this too much. We would unfortunately need to mask
> all IRQs, because while the keyboard, timer and RTC irqs are fixed, the
> mouse (or other contraption) IRQ whose number is impredictable.

Ah yeah. I don't like this either. Too hardware-dependent. And would this even work in V86-mode, if running, say, Jemm?

And it still wouldn't avoid resident Int21 handler code interference.

---
l

> Here you have it as written this morning, raw cut&paste - the whole
> installation section, I can't seem to be able to copy the whole source into
> this box nor is it necessary - refer to the v 1.5 for context. It's
> unpolished, untested and just to give you crumbles to attack until I give
> you more food :=)

Ooops, for some reason the important STI ending the "critical section" has been left out. It should be just before the line labeled #7 :

STI ; end critical section
> ; 7 - Copy resident part to its final place :

--
Ninho

Passing by
>> Indeed
>> if some interrupt-activated resident chimes in after freeing but before
>> allocating, it could itself allocate from one or both of the blocks we've
>> just given back ! That would not be an error, as far as DOS is concerned,
...
> Yes, it does apply to my system. Yes, it could temporarily allocate memory
> in such a way as to prevent optimal allocation.

> However, as you said, that would not be an error (not just for DOS, in
> fact, it wouldn't be an error for anyone).

Not an error, but yes a failure due to a rude TSR not acting responsibly! Being able to preempt both the foreground task and the OS itself is a privilege, with privilege comes responsibility - at least I was brought up thinking so.

"Alternate method 1" sketched in my previous post, under CLI, would completely prevent the potential for such nuisance.

BTW just noticed the STI closing the critical section was unfortunately missing from the sample cut&pasted up thread. It should come precisely before section #7

---
Ninho

> Ooops, for some reason the important STI ending the "critical section" has
> been left out.

I didn't even notice. How unprofessional of me!

Process termination re-enables interrupts anyway.

---
l

> Not an error, but yes a failure due to a rude TSR not acting responsibly!
> Being able to preempt both the foreground task and the OS itself is a
> privilege, with privilege comes responsibility - at least I was brought up
> thinking so.

In my method, there isn't any way for the TSR to determine properly that it acts "not responsibly". In your method of protecting the section by clearing the interrupt flag, there arguably isn't either: If the protection works as intended, the TSR won't pop up. If it fails (21.48 call, running in debugger, running in V86-mode?) the TSR has no way of determining you intended to leave the interrupt flag disabled.

> "Alternate method 1" sketched in my previous post, under CLI, would
> completely prevent the potential for such nuisance.

No, because as I alluded to, resident code could allocate the memory that would be required for your optimal allocation (say, a UMB of just the right size) before you enter your protected section of code.

---
l

> > "Alternate method 1" sketched in my previous post, under CLI, would
> > completely prevent the potential for such nuisance.
>
> No, because as I alluded to, resident code could allocate the memory that
> would be required for your optimal allocation (say, a UMB of just the right
> size) before you enter your protected section of code.

Oh no, that is not considered a problem, it's no different than if the resident had done that allocation 5 minutes /before/ we launched KBFR ! Nobody can or wants to prevent that :=)

What poses a problem is if the resident could try and succeed to reserve memory while our critical section executes, only.

---
Ninho

> What poses a problem is if the resident could try and succeed to reserve
> memory while our critical section executes, only.

Why? What's the difference between the other allocation occurring five minutes earlier, while you are setting up the critical section, or during the critical section?

---
l

> Why? What's the difference between the other allocation occurring five
> minutes earlier, while you are setting up the critical section, or during
> the critical section?

The critical section ensures that we obtain a suitable free block
*no higher than our old PSP* whatever may happen behind our backs...
Because I don't relocate the installer to the top of low memory, I want
this feature badly ;=)

---
Ninho

Only if you use one of the "First fit" allocation strategies.

---
l

> Only if you use one of the "First fit" allocation strategies.

Not necessarily. If I replace the int 21 function 48h by mine, I needn't observe DOS's "strategy" - assuming it has one ;=) But I will set the UMB "link" state, yes.

Seriously I'd like to hear of others' opinion on the polite controversy over what TSRs are or aren't supposed to do. Specifically, I maintain a TSR which intends to sanely 1. call DOS functions and, or, 2. access DOS's internal state directly (whether to modify it or just for monitoring), MUST refrain from doing so if DOS has been entered - except well known cases of a few DOS 1 console functions IF INDOS flag=1. I've been stunned by CM's assumptions of TSR's allowed to basically mess with DOS in whatever way they fancied. Other developers speak up please, your silence has been deafening :=)

Aside from this diversion, I have devised a new scheme based on int 15/ 4F only, for enabling/disabling/testing the state of the keyboard driver. Hope to implement this the sooner.

All this is entertaining, how much I regret I can't get a little more time to try and finalise all the interesting ideas...

---
Ninho

> If I replace the int 21 function 48h by mine

I assumed you didn't want to do that.

> Seriously I'd like to hear of others' opinion on the polite controversy

I see no controversy. I too would welcome any input on TSR development though.

> I've been stunned by CM's assumptions of TSR's allowed to basically
> mess with DOS in whatever way they fancied.

Nah, as I mentioned my assumption was wrong. While in DOS (potentially in code messing with MCBs itself: 21.48 or 21.4A or 21.5803 or 21.4B or 21.67) TSRs must not alter MCBs. Exceptions to this might exist, for example, I think a TSR can probably free its own MCBs at any time without risk.

> All this is entertaining, how much I regret I can't get a little more time
> to try and finalise all the interesting ideas...

Don't we all?

---
l