DOS ain't dead

[I'm transferring this argumentative thread to the Developers' section rather than Announcements!]

>> brand new int 15h method is used to communicate with an installed
>> instance of KBFR.

CM wrote :
> I'm not a fan of overloading interfaces with magic values like that.

>Your interface still contains some unused fields?

Good morning ?

As I wrote, I shall document the whys and hows of this imho neat and secure interface. And indeed some fields are reserved for further uses. And there is cause for "overloading" 15/4F, and for the choice of al=F4 too. You might want to wait before blindly hammering at it all :=)

skimming and quick answering

> orgint15's address is hardcoded in the uninstaller. Why? ...

> The trans function still isn't inlined, as you noted in the comments.

Implementation details which do not affect function, and might or might not change between versions.

Although the communications interface itself is generic enough to allow for passing a number of parameters, as a rule I won't to turn the resident into a chemical factory.

As the resident's effective size happens to be now an integral # of "paragraphs", I felt little incentive to inline trans ... for no gain ;=)

>Your interrupt handler still isn't using the interrupt sharing protocol.
>Your uninstaller still doesn't walk interrupt sharing protocol handler >chains and doesn't query AMIS TSRs for their chain entries.

Won't do. Size matters :=)

> You sometimes discard the saved flags by returning with "retf 2" from an > interrupt handler.

Not "sometimes", it's in the one place iirc - where it returns from the API call. Are you suggesting I made a mistake ? I don"t think so.

> Your arguably ineffectual critical section protection doesn't appear to be >necessary any longer, as you aren't using any data in the freed area. You >can use 21.49 to free the previous PSP's area then.

>You still free the environment only inside the (non-)critical section, and >manually. I'd use 21.49 again.

I thought we discussed this at length, and I thought I'd won the argument. Apparently not ? Anyways, I'm content with how it stands.

>Where it says "calculate resident size", the calculation always appears to >yield the same value

I know, you've told this N times ! It's in my development version already.
Implementation trifle again!

>Still not checking for errors after 21.48. Can't happen?

Appropriate checks are added as I reestablish auto highloading. I have to verify things will default to working properly under old (pre MS DOS 5 e.g.) DOS versions as well.

>I guess that's all for now.

You might have chased a few typos, too... just kidding. Thanks for the punctilious peeking !

--
Ninho

> [I'm transferring this argumentative thread to the Developers' section
> rather than Announcements!]

Alright.

> You might want to wait before blindly hammering at it all :=)

I have to admit that I'm no expert on the 15.4F interface, so others who know that better should judge the overloading of it with this interface.

However, I don't see why you implemented the unused fields the way you did. Wouldn't it suffice to let, say, cx contain a function number, and then indicate success (if the function number is supported) by changing some register's value? For example, you could change the signature in bx to indicate that the interface is supported, and then you could change the function number in cx to something to indicate whether the particular function number is supported.

The current buffer method seems unnecessary. And if you re-use and extend it, you would theoretically have to check for older versions which will not touch the currently unused fields. (You probably will not implement that, but whatever.)

> Implementation details which do not affect function, and might or might not
> change between versions.

Yes, hardcoding TSR data offsets in the uninstaller only affects cross-version compatibility. Bad enough.

> As the resident's effective size happens to be now an integral # of
> "paragraphs", I felt little incentive to inline trans ... for no gain ;=)

The saved bytes could easily gain you a paragraph after further (or future) changes.

> >Your interrupt handler still isn't using the interrupt sharing protocol.
> >Your uninstaller still doesn't walk interrupt sharing protocol handler
> >chains and doesn't query AMIS TSRs for their chain entries.
>
> Won't do. Size matters :=)

Extending (only) the uninstaller's code will not increase your resident size. And you could implement the header for the interrupt handler (about 18 additional bytes) as a build option so that users who would prefer a cooperative TSR instead of one squeezing out the last byte can create that from the source.

> > You sometimes discard the saved flags by returning with "retf 2" from an
> > interrupt handler.
>
> Not "sometimes", it's in the one place iirc - where it returns from the API
> call.

Ah, yes.

> Are you suggesting I made a mistake ?

I am suggesting you are discarding the flags, for example, the trace flag (could still be off even though it was on before the call), the direction flag (could still be on, ie, "downwards"), the interrupt flag (could still be off). (Note that the trace flag issue isn't actually an issue with Japheth's debugger since it will skip interrupts by placing a breakpoint behind that opcode. It might be a problem with some programs tracing through other code though.)

> > critical section protection doesn't appear to be necessary any longer
>
> I thought we discussed this at length, and I thought I'd won the argument.
> Apparently not ? Anyways, I'm content with how it stands.

I don't remember that.

What I did say back here already was this:

> > > Is there a way to make : (free, then allocate) into a "critical
> > > section" of sorts ?
> >
> > No, and that's not necessary if you properly keep all the data you need
> > inside allocated space.

As you seem to be keeping all used data inside allocated space all the time now, the need for the critical section (however well it is implemented) goes away. (As I have shown in this post, any critical section implementation which still uses function 48h can hardly insure that no other resident code modifies MCBs before the function call reaches DOS. So even preventing resident code interference that could result in non-optimal installation doesn't hold as purpose of the critical section.)

And you said here, before that:

> > > try and put DOS functions to work whenever possible rather than
> > > direct manipulation of structures.

So I thought, without the need for a critical section, why not use function 49h to free the memory?

> It's in my development version already.

What kept you from uploading that? Just curious.

> >Still not checking for errors after 21.48. Can't happen?
>
> Appropriate checks are added as I reestablish auto highloading. I have to
> verify things will default to working properly under old (pre MS DOS 5
> e.g.) DOS versions as well.

What does that have to do with error checking? Yes, obviously you'll have to make sure that it will work properly even in DOS versions not supporting UMBs via the DOS memory allocation functions, and even when a DOS version that does support UMB management in this way doesn't currently manage any UMBs. However, you have to check for errors after pretty much any function 48h call independently of the exact allocation method and of DOS-managed UMB availability.

> You might have chased a few typos, too... just kidding.

Obviously I would chase source code comments that are not written in English first.

> Thanks for the punctilious peeking !

What can I say, my pleasure

---
l

Quickie reply

> I have to admit that I'm no expert on the 15.4F interface, so others who
> know that better should judge the overloading of it with this interface.

It was given much consideration so as to avoid confusing other users and issuers of that software interrupt (be it the BIOS or even Brett Johnson)

An alternative (int 15/8F, apparently unused elsewhere) was considered and reserved in case, the interest of overloading, as you put it, 15/4F being it saves us a few code bytes w/o sacrificing correctness, as I'm sure we'll see.

> However, I don't see why you implemented the unused fields the way you did.

Precisely because of the "overloading" : int 15/4F callers and hookers must not be disturbed. You'll see when I publish supplementary doc.
...

> Yes, hardcoding TSR data offsets in the uninstaller only affects
> cross-version compatibility. Bad enough.

I think you're mistaken, it's the installer portion that for convenience uses some hard coded offset, not the uninstaller. BICBW.

>> As the resident's effective size happens to be now an integral # of
> > "paragraphs", I felt little incentive to inline trans ... for no gain
> ;=)

> The saved bytes could easily gain you a paragraph after further (or future)
> changes.


> I am suggesting you are discarding the flags, for example, the trace flag

Indeed. I don't make it a sovereign goal to be tracer-friendly :=)

- the infamous critical section :

> (As I have shown in this post, any
> critical section implementation which still uses function 48h can hardly
> insure that no other resident code modifies MCBs before the function call
> reaches DOS. So even preventing resident code interference that could
> result in non-optimal installation doesn't hold as purpose of the critical
> section.)

But you'll remember I modified my implementation such that interrupts inside of the critsec can only occur inside of DOS (function 48) - I won't insult the reader by reminding that IRET back from DOS to my code restores a cleared interrupt flag. And you agreed there a TSR awoken during int 21/48 would not be well advised to mess with DOS mem allocation.

> ... my assumption was wrong. While in DOS (potentially in
> code messing with MCBs itself: 21.48 or 21.4A or 21.5803 or 21.4B or
> 21.67) TSRs must not alter MCBs...

Indeed they mustn't ! Ergo I think my critsec /is/ playing its role against unwanted MCB tampering.

> And you said here, before that:
>>>> try and put DOS functions to work whenever possible rather than
>>>> direct manipulation of structures.

I must have been right then :=) But as in all matters, circumstances command

> So I thought, without the need for a critical section, why not use function
> 49h to free the memory?

Even if we weren't caring for criticality - we have to access those MCBs in the course of installation anyway, as it stands it is a gain rather than a cost to emulate DOS functions 49 and 4A. Emulating fn 48 OTOH, would be costly even a simplified version (dealing with low memory only and strategy #0).
>
>> It's in my development version already.

> What kept you from uploading that? Just curious.

Would you like if I uploaded a new package for every little change I do ? I don't think so ;=)

So long... (pun intended of course)

---
Ninho

> An alternative (int 15/8F, apparently unused elsewhere) was considered and
> reserved in case

The problem with overloading any function is that there's no main authority where you can reserve function numbers. (In fact, there never was. Off-topic.) By which I mean: There could be other programs already using 15.4FF4 or 15.8F for other purposes. And they can't detect that your program is servicing those instead, because they can't know your program's interface. That is basically the same problem as that of interrupt 2Fh usage as TSR multiplex interrupt.

> Precisely because of the "overloading" : int 15/4F callers and hookers must
> not be disturbed.

In what way would they be disturbed more if you specified cx as additional parameter and used it to verify the return values too?

> I think you're mistaken, it's the installer portion that for convenience
> uses some hard coded offset, not the uninstaller. BICBW.

I wasn't entirely mistaken, and you were wrong: The uninstaller hardcodes the address of introu15 (to determine whether the current topmost interrupt 15h handler is that of the resident code) and as I said it hardcodes the address of orgint15 (to move the next interrupt 15h handler to the top if successfully uninstalling the resident code).

Of course I don't know whether your development version contains other code there, I'm just looking at the week-old source code that you uploaded recently.

(Note that hardcoding addresses inside resident in the installer is alright: The installer knows that this particular resident must be in use because it does the installing itself.)

> I don't make it a sovereign goal to be tracer-friendly :=)

Still leaves the direction and interrupt flag. The latter arguably isn't critical. The former however, left in the wrong state (which could have been set by a previous interrupt 15h handler that chained to your handler via jumping and assumed you would iret properly instead of using retf 2), can easily cause a program to crash.

> I won't insult the reader by reminding that IRET back from DOS to my
> code restores a cleared interrupt flag.

Unless DOS or another resident interrupt 21h handler uses retf 2 to return from the interrupt call too

> And you agreed there a TSR
> awoken during int 21/48 would not be well advised to mess with DOS mem
> allocation.
>
> > ... my assumption was wrong. While in DOS (potentially in
> > code messing with MCBs itself: 21.48 or 21.4A or 21.5803 or 21.4B or
> > 21.67) TSRs must not alter MCBs...
>
> Indeed they mustn't ! Ergo I think my critsec /is/ playing its role against
> unwanted MCB tampering.

No. You misunderstood the subject of my agreement there. TSRs are not to mess with MCBs (too much) while the InDOS flag in DOS's SDA is non-zero. However, resident code (in particular any interrupt 21h handler as shown previously) is free to call down into DOS while that flag is zero. When an interrupt 21h handler is executed, the InDOS flag (which is usually modified by DOS) is still zero. Resident code that was invoked by intercepting your interrupt 21h function 48h call can assume that it is okay to call the DOS memory allocation functions right now (even without checking the InDOS flag*) - for your purposes, and for its own purposes as well.

* Because the intercepted call is one that would call DOS's function 48h memory allocation code.

> Even if we weren't caring for criticality - we have to access those MCBs in
> the course of installation anyway, as it stands it is a gain rather than a
> cost to emulate DOS functions 49 and 4A.

First, to be precise, you are not emulating function 4Ah because function 4Ah is not the "split a memory block into two and free the lower block" function. In splitting memory blocks (without freeing either), you are emulating some particular functionality that is (presumably) internally implemented in DOS's function 4Ah handling as well. There is a difference. (And no, we aren't splitting hairs here. It's MCBs )

Second, your statement is not at all true for the environment's MCB; you aren't otherwise accessing that and using 21.49 would be easier there. Your statement is somewhat true for the process's MCB, but note that if you would use 21.49 for freeing that instead, then the MCB splitting operation (which shrinks the process MCB and insures its MCB letter is 'M') would be the only write access to the process MCB. This way, you would also use the formally correct and "monitor-able" DOS function for freeing the MCB (think 21.25 vs. direct access).

(Also note that with entire process relocation, you never directly read from or write to the initial process MCB. But I guess I provided enough shameless plugs for my method in past posts already.)

> Would you like if I uploaded a new package for every little change I do ?

No*, but you said that what you uploaded is one week old already. Why didn't you upload a more recent development version?

* On the other hand, a public svn (or git, hg, ...) repository would allow access to every little change without much inconvenience.

> So long... (pun intended of course)

May the splitting of MCBs continue.

---
l

> The problem with overloading any function is that there's no main authority
> where you can reserve function numbers.

Agreed. This is where my cute (modesty apart) "overloading" is in order.

> In what way would they be disturbed more if you specified cx as additional
> parameter and used it to verify the return values too?

A caller - the BIOS foremost, it is after all the "owner" of that interface -
but any caller (hello, Bret?) might assume no register except AL and the Carry flag will be changed across the int 15/4Fxx call. In practice BIOSes I've looked at do protect themselves, but I'm doing everyone a favor by not using registers other than the normally affected ones and then using the latter in a manner compatible with what the interface expects. Hence the apparently unnecessary passing of data thru a mem buffer provided by caller.
The detailed method affords additional protections both to me (as KBFR) and to int 15/4F callers, as does the choice of the /release/ scancode for a key that is inexistant anyway on standard as well as exotic keyboards. Enough said, or what will remain for that document I promised ?

>> I think you're mistaken, it's the installer portion that for convenience
>> uses some hard coded offset, not the uninstaller. BICBW.

> I wasn't entirely mistaken, and you were wrong: ...

Quite possibly. I am on and off at this, there are legacy details to be cleared and you probably know my code better than I do ;=) That said, the intended use behind the communications scheme is to provide the uninstaller with a simple and sure way to identify an active resident instance, which it does & w/o having to add bloat to the resident (discardable bloat is OK). The details of what additional data are exchanged is not crucial, nor do I consider a prime design objective that a random version of the uninstaller can remove a different version resident (especially if they are both betas!).

> (Note that hardcoding addresses inside resident in the installer is
> alright: The installer knows that this particular resident must be in use
> because it does the installing itself.)

That is the intent indeed.

>> I don't make it a sovereign goal to be tracer-friendly :=)

> Still leaves the direction and interrupt flag....

TBH I frowned a little upon my own use of RETf ... it appeared safe inasfar as (again speaking from memory rather than reading the code) the RETf 2 in question is hit only if returning from the private API call. I might change the RETf to an IRET, which unfortunately adds the overhead of modifying the stacked version of the flags.

>> I won't insult the reader by reminding that IRET back from DOS to my
>> code restores a cleared interrupt flag.

> Unless DOS or another resident interrupt 21h handler uses retf 2 to return
> from the interrupt call too

Dare they !!! :=)

>> And you agreed there a TSR
>> awoken during int 21/48 would not be well advised to mess with DOS mem
>> allocation.
(...)
>> Indeed they mustn't ! Ergo I think my critsec /is/ playing its role
> against
>> unwanted MCB tampering.

> No. You misunderstood the subject of my agreement there. TSRs are not to
> mess with MCBs (too much) while the InDOS flag in DOS's SDA is non-zero.

> However, resident code (in particular any interrupt 21h handler as shown
> previously) is free to call down into DOS while that flag is zero. When an
> interrupt 21h handler is executed, the InDOS flag (which is usually
> modified by DOS) is still zero. Resident code that was invoked by
> intercepting your interrupt 21h function 48h call can assume that it is
> okay to call the DOS memory allocation functions right now (even without
> checking the InDOS flag*) - for your purposes, and for its own purposes as
> well.

Right, I think can see the whole picture now. I was sort of forgetting to take in consideration cases such as of an "intermediate" int 21 handler, interrupted by hardware while processing KBFR's int 21/48.

Seems I want to do both CLI *and* increment INDOS myself, one or the other alone is deficient, but doing both saves my face, doesn't it ?

>> Even if we weren't caring for criticality - we have to access those MCBs in
>> the course of installation anyway, as it stands it is a gain rather than a
>> cost to emulate DOS functions 49 and 4A.

> Second, your statement is not at all true for the environment's MCB; you
> aren't otherwise accessing that and using 21.49 would be easier there.

The gain then is in execution time :=)

> No*, but you said that what you uploaded is one week old already. Why
> didn't you upload a more recent development version?

> * On the other hand, a public svn (or git, hg, ...) repository would allow
> access to every little change without much inconvenience.

I have /really/ less time and more other, tangible problems in front of me than you may believe.

> May the splitting of MCBs continue.

And some hair too, despite contrary allegations haha

> (hello, Bret?)

There are indeed many, many additional comments and observations I could, and maybe even should, make. But I won't.

> This is where my cute (modesty apart) "overloading" is in order.

Your overloading is problematic independently of whether you overload functions or subfunctions.

> The detailed method affords additional protections both to me (as KBFR) and
> to int 15/4F callers,

It doesn't. A caller might just as well assume that dword[ds:si+4] isn't changed.

In fact, using a pointed-to buffer might increase the risk there: if a caller accidentally does call your special function without knowing about the modifications you make, then if you only modified registers I would assume it is more probable that this modification doesn't affect the caller - as opposed to having modified memory at some random location.

This is obviously very theoretical. If any caller accidentally and unknowingly hits all the exact magic values for your overloaded interfaces, you are in for some trouble.

> as does the choice of the /release/ scancode for a
> key that is inexistant anyway on standard as well as exotic keyboards.

As I said, I'm no expert on the keyboard interface. Therefore I will ignore this for now.

> That said, the
> intended use behind the communications scheme is to provide the uninstaller
> with a simple and sure way to identify an active resident instance,

Then you might as well save some resident space by returning only resident's segment; ie, by hardcoding the address of varjne in the uninstaller too.

> design objective that a random version of the
> uninstaller can remove a different version resident

Disagree.

> (especially if they are both betas!).

Agree, somewhat.

> it appeared safe inasfar as the RETf 2 in question is hit only if
> returning from the private API call.

Bad enough. You'll have to insure not to depend on the direction flag value after the call in your program. (And if theoretically an unaware caller happens to get there, they could be in even more trouble than otherwise.)

> I might change
> the RETf to an IRET, which unfortunately adds the overhead of modifying the
> stacked version of the flags.

Yes, you'll have to access the flags image on the stack then. But do you have to modify the carry flag?

> Right, I think can see the whole picture now. I was sort of forgetting to
> take in consideration cases such as of an "intermediate" int 21 handler,
> interrupted by hardware while processing KBFR's int 21/48.

That's one way it might happen, if the intermediate handler enables interrupts. However it doesn't even have to be interrupted: The intermediate handler is allowed to call the DOS memory allocation functions itself.

> Seems I want to do both CLI *and* increment INDOS myself, one or the other
> alone is deficient, but doing both saves my face, doesn't it ?

No, the intermediate handler is still allowed to call DOS itself, as I mentioned in my previous reply. Because you are calling function 48h, the intermediate handler can assume that calling function 48h and the other memory allocation functions for its own purposes is allowed - even if the InDOS flag is already non-zero.

Furthermore what we are calling an intermediate handler could even be part of DOS itself; a compatible DOS system could easily allocate memory for itself after entering its critical section (incrementing the InDOS flag) but before servicing your call.

> The gain then is in execution time :=)

Then please never use interrupt 21h functions 25h, 35h, 48h, 49h, or 4Ah. You don't want to be inconsequential, now do you

> > * On the other hand, a public svn (or git, hg, ...) repository would
> allow
> > access to every little change without much inconvenience.
>
> I have /really/ less time and more other, tangible problems in front of me
> than you may believe.

That was only an off-topic note, not a demand.

---
l

Then I hope I'm at least covering the TSR basics well enough here. If I say anything you disagree with please notify me of it, here or otherwise.

> Then I hope I'm at least covering the TSR basics well enough here. If I say
> anything you disagree with please notify me of it, here or otherwise.

You are covering the basics very well I think, Christian.

1. the int 154FFA scheme

>> The detailed method affords additional protections both to me (as KBFR)
> and
>> to int 15/4F callers,
>
> It doesn't. A caller might just as well assume that dword[ds:si+4] isn't
> changed.

And rightly so. It is not changed - except if the caller was validated to be KBFR. There are 96 bits of ID, not counting the 8 in AL. Many APIs we depend upon could fail with much much higher probability.

> In fact, using a pointed-to buffer might increase the risk there: if a
> caller accidentally does call your special function without knowing about
> the modifications you make, then if you only modified registers I would
> assume it is more probable that this modification doesn't affect the caller
> - as opposed to having modified memory at some random location.

The buffer is not modified, unless validated (bis). I think modifying registers would not be as safe.

> This is obviously very theoretical. If any caller accidentally and
> unknowingly hits all the exact magic values for your overloaded interfaces,
> you are in for some trouble.

Evidently - but then the probability of a memory malfunction is certainly higher than what you envisage.

>> as does the choice of the /release/ scancode for a
>> key that is inexistant anyway on standard as well as exotic keyboards.
>
> As I said, I'm no expert on the keyboard interface. Therefore I will ignore
> this for now.

However it is a significant point, and also in a lesser measure the fact that
"release" scan codes (of non modal keys) are dropped by the PC BIOS anyway.

As said earlier, if I weren't sure of this scheme, I'd use int 15/8FFA instead. Oh, but I'm sure you would find objections, too...
Well, the switch [i]is(/i] definitely an appeling possibility come to think again. Even though my little subversive int 15/4F trick is defendable, there is no compelling reason to clutch to it.

>> That said, the
>> intended use behind the communications scheme is to provide the
> uninstaller
>> with a simple and sure way to identify an active resident instance,

> Then you might as well save some resident space by returning only
> resident's segment; ie, by hardcoding the address of varjne in the
> uninstaller too.

Isn't coding/designing a series of choices, preferably coherent ones, some important and some less so ? I could pass less data from the resident to the (un)installer, or I could pass more! As you observed, returning this address dynamically could allow for additional compatibility checks, but of course it is non essential that it be passed back thru the buffer.

...
>> might change
>> the RETf to an IRET, which unfortunately adds the overhead of modifying
> the
>> stacked version of the flags.


> Yes, you'll have to access the flags image on the stack then. But do you
> have to modify the carry flag?

Yes, at least it is a good precaution. By resetting Cy we signal any observers, hookers, (even the initial caller if by witchcraft it were other than KBFR) - that the "scancode" has been dealt with and they should henceforth ignore it. Tis part of int 15/4F semantics.


2. DOS call reenterancy etc.

>> Seems I want to do both CLI *and* increment INDOS myself, one or the other
>> alone is deficient, but doing both saves my face, doesn't it ?

> No, the intermediate handler is still allowed to call DOS itself, as I
> mentioned in my previous reply. Because you are calling function 48h, the
> intermediate handler can assume that calling function 48h and the other
> memory allocation functions for its own purposes is allowed - even if the
> InDOS flag is already non-zero.

Oh NO! This is pure nonsense. Function 48 and DOS calls in general are not designed to be reentered, and any sound TSR must be aware of the fact. Any attempt to just reenter memory allocation fncts 48,49,4a,58... in the middle like that would yield horrible errors. By faking INDOS to the intermediate handler we forbid such absurd behavior.

> Furthermore what we are calling an intermediate handler could even be part
> of DOS itself; a compatible DOS system could easily allocate memory for
> itself after entering its critical section (incrementing the InDOS flag)
> but before servicing your call.

I don't buy this new fairy tale either. DOS does not randomly 'allocate memory for itself" after sysinit, and especially not in the middle of servicing user calls. What purpose would it serve? If you mean not the DOS core but an add-on, it is subject to the above reenterancy restrictions

>> The gain then is in execution time :=)
>
> Then please never use interrupt 21h functions 25h, 35h, 48h, 49h, or 4Ah.
> You don't want to be inconsequential, now do you

Just some relief and a pair of smileys

---
Ninho

>> (hello, Bret?)
This was meant rhetoric - as an example of non BIOS int 15/4F callers - rather than a spell on you (thought you were out of this) but
>
> There are indeed many, many additional comments and observations I could,
> and maybe even should, make. But I won't.

Care to tell why ? At least I hope it's /not/ because something I wrote has you offended - if it were the case, I would call it a misunderstanding and regret, sincerely. Anyways out of the many many things you could tell (but won't) what I would find most interesting would be an answer to the question : Bret, can it happen that some one of your TSRs issue an int 15 with AX=4FF4 ? If it can, will it be spontaneous or triggered by an (which?) exterior circumstance ? And again if your program must issue the above call, and it consequently receives (from KBFR) AL=F5 and No Carry (no other modification), is it in any way spoiling your processing ?

---
Ninho

> And again if your program must issue the
> above call, and it consequently receives (from KBFR) AL=F5 and No Carry (no
> other modification), is it in any way spoiling your processing ?

If you insist on this irrelevant exercise, then at least correctly mention that dword[ds:si+4] will be overwritten too.

---
l

> And rightly so. It is not changed - except if the caller was validated to
> be KBFR. There are 96 bits of ID, not counting the 8 in AL. Many APIs we
> depend upon could fail with much much higher probability.

I see only 24 bits: al = F4h and bx = "BK". 8 of those bits have a meaning in the original interface, 16 do not have a meaning there.

> I think modifying registers would not be as safe.

My point was that I think that in the event of unintended matching magic values, modifying registers would be safer than modifying memory. The reason I think so is that values in registers will possibly be discarded (and therefore ignored) by the caller anyhow.

Both memory contents and registers of course may or may not be in use.

> As said earlier, if I weren't sure of this scheme, I'd use int 15/8FFA
> instead. Oh, but I'm sure you would find objections, too...

All such schemes could obviously be in use by another program for other purposes already.

> > Then you might as well save some resident space by returning only
> > resident's segment; ie, by hardcoding the address of varjne in the
> > uninstaller too.

> As you observed, returning this
> address dynamically could allow for additional compatibility checks, but of
> course it is non essential that it be passed back thru the buffer.

What? You're already returning it dynamically. I only pointed out that passing this single address dynamically while hardcoding everything else is inconsequential. You might as well hardcode everything if you aren't properly returning all offsets (or a table with all offsets) from the resident interface.

> > the intermediate handler is allowed to call DOS itself

> > Because you are calling function 48h, ... calling function 48h and
> > the other memory allocation functions ... is allowed - even if the
> > InDOS flag is already non-zero.
>
> Oh NO! This is pure nonsense. Function 48 and DOS calls in general are not
> designed to be reentered, and any sound TSR must be aware of the fact. Any
> attempt to just reenter memory allocation fncts 48,49,4a,58... in the
> middle like that would yield horrible errors. By faking INDOS to the
> intermediate handler we forbid such absurd behavior.

This is pure nonsense. That DOS isn't designed to be re-entered is specifically the reason that any intermediate handler that intercepted a function 48h call can assume that the InDOS flag currently must really contain zero (even if it doesn't actually). After all, that is what DOS does itself: It assumes that InDOS is zero, ie that it isn't being re-entered.

The responsibility for checking the InDOS flag is only defined in this scheme for resident TSRs invoked "asynchronously", ie by hardware interrupts. Other than that, all reentrancy issues are dealt with by disallowing code called from within DOS to access DOS itself. (For example, an interrupt 13h handler isn't generally allowed to call DOS.)

> DOS does not randomly allocate memory for itself after sysinit,
> and especially not while it is servicing user calls.

Ah, is that defined in the official DOS standard? Obviously it isn't, since such a standard doesn't exist. And the implementation details of existing systems, such as any given version of MS-DOS, are not definitively precluding the possibility of such a DOS implementation. (Assuming one is not defining "compatible" as "same binary as MS-DOS", which would be a definition that would preclude one from discussing TSR implementation details with me - because I wouldn't bother with them.)

In any case, my point about intermediate handlers that (when intercepting properly inreentrant DOS functions) call into DOS without checking the InDOS flag is independent of whether you consider DOS static or extensible.

> Why would it /ever/ need to?

Be imaginative.

> And if you mean not the DOS core
> but an add-on, it is subject to the above reenterancy restrictions

Except when it isn't.

---
l

>> be KBFR. There are 96 bits of ID, not counting the 8 in AL. Many APIs we
>> depend upon could fail with much much higher probability.

> I see only 24 bits: al = F4h and bx = "BK". 8 of those bits have a meaning
> in the original interface, 16 do not have a meaning there.

96 - possibly more - counting preset parts of the buffer meant to be checked by the resident. Indeed not implemented in the published test version. My bad. 24 bits really are less than I would deem acceptable.

Switching to int 15/8F will allow the relaxing of constraints on the use of registers, make sanity checks simpler too and might be a minor but net gain in code size even. Definitely a winner.

>> As said earlier, if I weren't sure of this scheme, I'd use int 15/8FFA
>> instead. Oh, but I'm sure you would find objections, too...

> All such schemes could obviously be in use by another program for other
> purposes already.

Collisions can be designed against, accounting for probabilities. The advantage of int 15/8F, or any other int 15 subfunction for which no previous use is known, is evident still.

> This is pure nonsense. That DOS isn't designed to be re-entered is
> specifically the reason that any intermediate handler that intercepted a
> function 48h call can assume that the InDOS flag currently must really
> contain zero (even if it doesn't actually).

You would repeat this under oath ?

> After all, that is what DOS
> does itself: It assumes that InDOS is zero, ie that it isn't being
> re-entered.

And this, too ? I mean - limiting ourselves to MSDOS for specifics - it's been my understanding that DOS /increments/ InDOS upon entry and /decrements/ it back on exit (with internal nesting possibly), nowhere does it /assume/ InDOS=0 (but it does /test/ for inDOS =/= 0 on occasions).


> The responsibility for checking the InDOS flag is only defined in this
> scheme for resident TSRs invoked "asynchronously", ie by hardware
> interrupts.

Agreed

>Other than that, all reentrancy issues are dealt with by
> disallowing code called from within DOS to access DOS itself.

Sorry I don't know how to parse that.

>> DOS does not randomly allocate memory for itself after sysinit,
>> and especially not while it is servicing user calls.

> Ah, is that defined in the official DOS standard? Obviously it isn't, since
> such a standard doesn't exist.

Agreed, formally at least. There is more or less a de facto standard, the limits of which are somewhat fuzzy... But you are wandering too far in your supputations. I want to code for MS-DOS first in any case, and I may consider limited adaptations to small deviances in other DOSes.

> not defining "compatible" as "same binary as MS-DOS",

of course not. Be serious please

> In any case, my point about intermediate handlers that (when intercepting
> properly inreentrant DOS functions) call into DOS without checking the
> InDOS flag is independent of whether you consider DOS static or
> extensible.

>> Why would it /ever/ need to?
> Be imaginative.

>> And if you mean not the DOS core
>> but an add-on, it is subject to the above reenterancy restrictions

> Except when it isn't.

All of which would be very dangerous, without even too much imagination.
The way DOS is architected - not just an implementation - pretty much precludes or at least constrains bizarro scenarios like some of which you offered as examples. IMHO and I am challenging you to show an actual int 21 hooker which does the kind of shenanigans (did I get the word right?) you've been advocating repeatedly, like call DOS memory management functions or mess with MCBs directly inside of a user initiated int 21/48 etc, call, and still withstands testing successfully.

---
Ninho

>> And again if your program must issue the
>> above call, and it consequently receives (from KBFR) AL=F5 and No Carry
> (no
>> other modification), is it in any way spoiling your processing ?
>

> If you insist on this irrelevant exercise, then at least correctly mention
> that dword[ds:si+4] will be overwritten too.


No it won't (unless magic values are provided, which my question - to Bret - assumed his program won't issue°). The problem with arguing with you dear CM is that you insisted on discussing this API despite agreing to wait for the docs -and you are drawing false conclusions from excellent, but insufficient, analysis.

Please CM, Bret may or may not answer as far as what his TSRs do (what I'm curious to kearn) - but surely you can't answer in his place can you? and most certainly you shouldn't have chimed in with a misleading remark.

° BX in the version you have, and more data intended - but it doesn't matter here.

---
Ninho

> Care to tell why ? At least I hope it's /not/ because something I wrote has
> you offended - if it were the case, I would call it a misunderstanding and
> regret, sincerely.

Well, since you've asked... No, I'm not offended. I actually find the hypocrisy amusing. You're trying to do the EXACT same thing you told me I couldn't do -- call INT 15.4F from outside an INT 09 handler.

Just as an aside, I hope you realize that doing this (and the fact that MS does it and some of my programs do it) means that your INT 15.4F handler must be fully re-entrant. It may already be without you realizing it (I haven't looked). But, if the user types a key on the keyboard at the same time INT 15.4F is being called from outside INT 09, and it's not fully re-entrant, you could have a big problem.

> Anyways out of the many many things you could tell (but won't) what I would
> find most interesting would be an answer to the question : Bret, can it
> happen that some one of your TSRs issue an int 15 with AX=4FF4 ? If it can,
> will it be spontaneous or triggered by an (which?) exterior circumstance ?
> And again if your program must issue the above call, and it consequently
> receives (from KBFR) AL=F5 and No Carry (no other modification), is it in
> any way spoiling your processing ?

I don't think any of my programs will directly issue an unprovoked INT 15.4FF4, but I have several of them could simulate a scancode of F4h, which could cause INT 15.4FF4 to be issued by the INT 09 handler or by one of my programs. It's impossible to say exactly what would/could happen in such a situation. I will admit that the simulation of an F4h scancode is unlikely, but certainly not impossible.

Also FWIW, the way the keyboard simulation works, the INT 09 handler code may not actually be called by the generation of an IRQ 1. It may actually be called simply as a "subroutine" from another IRQ handler (usually INT 70 / IRQ 8). My programs temporarily disable the real keyboard while they are doing this, so re-entrancy in either the INT 09 or INT 15.4F handlers shouldn't be an issue from that perspective.

The user can tell my "virtual keyboard" programs to simulate almost any scancode they want to send, even if it couldn't come from a "real" keyboard. MOUSKEYS will simulate scancodes based on mouse movements and buttons, JOYKEYS does the same for joystick movements and buttons, and USBKEYB of course does it for USB keyboard buttons. My SCANCODE program will do it based on various kinds of input, including absolute or relative timing, specific text on the screen, and other types of input, including even other scancodes (whether from the real keyboard or another "virtual keyboard" program, including another copy of SCANCODE). At least in the case of SCANCODE, it is actually very useful sometimes to simulate scancodes that actually can't come from the real keyboard.

I will also say this. Overloading interrupts is OK, as long as you are careful about how and why and when you do it. I've done it several times myself. My USB API overloads INT 14h, my joystick programs overload the joystick BIOS interface (INT 15.84), USBKEYB (and in the future all of the other virtual keyboard programs) overloads INT 16h, etc.

The only time you should do this, though, is when there is not an existing, suitable API that does what you need it to do. In addition, you need to be very careful about which INT's and functions and subfunctions you overload. INT 15.4F is simply not one you should be messing with, since it is effectively no more than a "subroutine" for an IRQ handler. I don't think I even need to tell you how bad of an idea it is to overload an IRQ handler, whether directly or indirectly. I haven't actually looked at what you're doing, but I suspect INT 2Dh (AMIS) is probably more than sufficient for what you're trying to accomplish, even though I know it requires more memory than you want it to.

> you are drawing false conclusions from excellent, but insufficient, analysis.

In what way are they false given the provided source code?

> but surely you can't answer in his place can you?

But surely you didn't misunderstand what I wrote that much did you?

> Just as an aside, I hope you realize that doing this (and the fact that MS
> does it and some of my programs do it) means that your INT 15.4F handler
> must be fully re-entrant. It may already be without you realizing it (I
> haven't looked). But, if the user types a key on the keyboard at the same
> time INT 15.4F is being called from outside INT 09, and it's not fully
> re-entrant, you could have a big problem.

More precisely, only the 15.4FF4 handler needs to be reentrant since that's the one that the main KBFR program will call in an attempt to discover existing installed copies of it.

Note that I intentionally didn't say "your 15.4FF4" handler (which would just so happen to be reentrant, as (I think) your whole 15.4F handler is) but "the 15.4FF4" handler. There's a distinction to be made: At the time that you call 15.4FF4, you require all current 15.4F(F4) handlers to be reentrant. Even if there are handlers (intermediate or final) other than your own.

> I suspect INT 2Dh (AMIS) is probably more than sufficient for what
> you're trying to accomplish, even though I know it requires more
> memory than you want it to.

To be precise, 71 bytes plus indirect jumps made necessary by the Int15 IISP headers. An only partially compliant interface could get by with as few as 37 bytes.

---
l

> > This is pure nonsense. That DOS isn't designed to be re-entered is
> > specifically the reason that any intermediate handler that intercepted a
> > function 48h call can assume that the InDOS flag currently must really
> > contain zero (even if it doesn't actually).
>
> You would repeat this under oath ?

No, obviously I wouldn't. But, yes, I do believe it to be true.

> > After all, that is what DOS
> > does itself: It assumes that InDOS is zero, ie that it isn't being
> > re-entered.
>
> And this, too ? I mean - limiting ourselves to MSDOS for specifics - it's
> been my understanding that DOS /increments/ InDOS upon entry and
> /decrements/ it back on exit (with internal nesting possibly), nowhere does
> it /assume/ InDOS=0 (but it does /test/ for inDOS =/= 0 on occasions).

No, in that way DOS doesn't assume that the InDOS flag ever contains the value zero. However, it doesn't check the InDOS flag before incrementing it.

In that context, it can be said that DOS and other handlers of inreentrant interrupt 21h functions conceptually assume the InDOS flag to be zero because they are about to enter DOS: _they cannot be inside DOS yet_. If they in fact are inside DOS already (which means actually in a DOS call or the InDOS flag was faked up) it is the caller's responsibility to have handled conflicts.

> > Other than that, all reentrancy issues are dealt with by
> > disallowing code called from within DOS to access DOS itself.
>
> Sorry I don't know how to parse that.

Code that can be called from DOS isn't allowed to simply call DOS, since it could be re-entering DOS then.

> > In any case, my point about intermediate handlers that (when
> intercepting
> > properly inreentrant DOS functions) call into DOS without checking the
> > InDOS flag is independent of whether you consider DOS static or
> > extensible.
>
> All of which would be very dangerous, without even too much imagination.
> The way DOS is architected - not just an implementation - pretty much
> precludes or at least constrains bizarro scenarios like some of which you
> offered as examples. IMHO and I am challenging you to show an actual int 21
> hooker which does the kind of shenanigans (did I get the word right?)
> you've been advocating repeatedly, like call DOS memory management
> functions or mess with MCBs directly inside of a user initiated int 21/48
> etc, call, and still withstands testing successfully.

I can't provide one that specifically intercepts the memory allocation functions right now (without writing it myself), but you can for example take a look at DOSLFN's source code: it hooks interrupt 21h and extensively works with the InDOS flag (incrementing it by itself additionally, to avoid being re-entered itself) yet it doesn't check the InDOS flag before calling down into DOS.

For one particular example, any 21.71 call can cause DOSLFN to (try to) reload its Unicode translation table, using 21.3D, .3F, .42, and .3E, if the current DOS code page has changed since DOSLFN last had been called.

---
l

> > you are drawing false conclusions from excellent, but insufficient,
> analysis.
>
> In what way are they false given the provided source code?
>
> > but surely you can't answer in his place can you?
>
> But surely you didn't misunderstand what I wrote that much did you?

It's starting to be boring. Please re-read my previous reply, cool down and you'll see the light. You wrote
> If you insist on this irrelevant exercise, then at least correctly mention
> that dword[ds:si+4] will be overwritten too.

which (ignoring the tone as I always will try to) is not right : I'm not so fool as to blindly trash someone else's memory. Overwriting would occurr only if magic values (BX=4B42 is in the code I published, and as I said repeatedly there would be more ) were set by the caller, presumably KBFR.

---
Ninho

> Overwriting would occur only if magic values

Yes. The 16 bits of bx are undefined in 15.4F's specification and you did not specify any further requirements for them in this exercise.

> ... in the code I published, and as I said repeatedly there would be more

I'll judge that would-be interface when concrete documentation for it or an implementation of it is available. In the meantime, it is not unreasonable to assume that we are talking about the current interface unless you explicitly phrase your exercise differently.

---
l

> For one particular example, any 21.71 call can cause DOSLFN to (try to)
> reload its Unicode translation table, using 21.3D, .3F, .42, and .3E, if
> the current DOS code page has changed since DOSLFN last had been called.

It is to presume that DOSLFN does its thing safely, with proper attention to DOS internal stacks and so on. Not /all/ such recursion is bad, otherwise we wouldn't have TSRs (and DOS itself recurrs). At 1st glance the case of LFN functions carefully calling down into short name file functions does not compare, in terms of renterancy, with trying to call memory allocation functions from the interior of functions of the same kind. The latter is not even meaningful.

----- Announcement :

I'll be "off" the stage for some unpredictable time for reasons which I've alluded to earlier, health foremost compounded by a load of material problems which will be surmounted, God helping and inch'Allah! just trying to make it clear that my cutting off the site (and probably Internet) is not me being p...d off or discouraged, and that I /hope/ I'll be back :=)

Meanwhile I don't expect, but I certainly wouldn't object to, someone perfecting or forking, whatever, KBFR.

When I'm back at it myself, I guess I shall :

- ditch the use of int 15/4FF4, which besides it has been met with mixed applause, safe overloading of that particular call leads to somewhat baroque and overall costly design and implementation choices.
- keep my loading scheme as is - not only is is working in practice, also thought 'attacks' have not succeeded in breaking it in the least (safe in dream).
- complete auto-highloading, make it the default.
- and the remaining fixing of odds and ends. Yes including error checking where needed ;:)

- when that's done, perhaps port back to make a matching KBGR (provided there are volunteer beta testers)

So long then, going to check the rest of the forum and then shutting myself down - wish I could just hibernate...

---
Ninho

>> Care to tell why ? At least I hope it's /not/ because something I wrot has
>> you offended

> Well, since you've asked... No, I'm not offended.

Godd, I'm actually relieved...

> I actually find the hypocrisy amusing. You're trying to do the EXACT same thing you told me I couldn't do -- call INT 15.4F from outside an INT 09 handler.

Not so much hypocritical than amusing indeed, it was very counscious and I think it was the reason i mentionned your name in a couple of msgs where I announced "subverting" 15.4F. As to whether it's "the EXACT same" use as your doing I can't pronounce.

You'll be pleased to learn I have renounced this somewhat foolish use of 15.4F (also contradicting by my own conceptions) - not so much because it can't be more than reasonably safe, than because securing the concept and implementation inevitably leads to "baroque" as I called them methods, and ends up costing more in code surface. I couldn't
>
> Just as an aside, I hope you realize that doing this (and the fact that MS
> does it and some of my programs do it) means that your INT 15.4F handler
> must be fully re-entrant. It may already be without you realizing it (I
> haven't looked). But, if the user types a key on the keyboard at the same
> time INT 15.4F is being called from outside INT 09, and it's not fully
> re-entrant, you could have a big problem.

Ah! this is interesting, but more fuel for my and pretty everyone's assumption that int 15/4F should (must?) be /issued/ from the int 9 handler (whether in ROM or otherwise). The case of MS or other software issuing 15/4F53 for the sole purpose of preparing to reboot being set apart, and not sure if even it is assured against reentry in theory (although good enough in practice).

TBH I didn't design the int 15 entry with reentrancy in mind, so there is a high chance that it is not. I would feel concerned (to a point) were my use of 15/4F not already passé :=)


> I don't think any of my programs will directly issue an unprovoked INT
> 15.4FF4, but I have several of them could simulate a scancode of F4h, which
> could cause INT 15.4FF4 to be issued by the INT 09 handler or by one of my
> programs. It's impossible to say exactly what would/could happen in such a
> situation. I will admit that the simulation of an F4h scancode is
> unlikely, but certainly not impossible.
....
Thank you, your explanation is great stuff and amply more than I expected to receive. I'm shortening this not out of disinterest but because by coincidence I'm retiring from the scene for a hopefully short period. I'm saving your explanations about your use of the keyboard ecosystem for later deep reading. I'm sure others readers will find them instructive too...

---
Ninho

> It is to presume that DOSLFN does its thing safely, with proper attention
> to DOS internal stacks and so on.

Yes.

> Not /all/ such recursion is bad, otherwise we wouldn't have TSRs
> (and DOS itself recurrs).

What DOSLFN does isn't recursion as far as DOS is concerned. Recursion occurs in MS-DOS only with functions which execute on the user stack frame (USF) entirely, such as 21.62. (Those are marked in RBIL as "not [using] any of the DOS-internal stacks", and it is correctly noted that these functions "may thus be called at any time, even during another INT 21h call".) There's some specific exceptions but generally only functions executing on the USF may be called when DOS is really re-entered.

> At 1st glance the
> case of LFN functions carefully calling down into short name file functions
> does not compare, in terms of renterancy, with trying to call memory
> allocation functions from the interior of functions of the same kind. The
> latter is not even meaningful.

You must have misunderstood me. What I proposed is the same. DOSLFN does this:

i21:
  cmp ah, 71h
  jne .next
  ...
  mov ax, 3D00h
  int 21h
  ...
  mov ah, 3Eh
  int 21h
  ...

(Note that DOSLFN does actually use the int 21h opcodes as shown above - therefore re-entering intermediate handlers before itself, as well as itself partially. That isn't relevant here though.)

Clearly, DOS is never re-entered.*

What I proposed is conceptually the same, just with memory allocation functions instead of file access functions:

i21:
  cmp ah, 48h
  jne .next
  ...
  mov ah, 48h
  pushf
  push cs
  call .next
  ...

Again, DOS is never re-entered.*

* To be precise, it isn't re-entered more often than the intercepted call would re-enter DOS anyway. Assuming DOS wouldn't be re-entered by the user-issued 21.48 call, the intermediate-handler-issued 21.48 calls aren't re-entering DOS either.

> - keep my loading scheme as is - not only is is working in practice, also
> thought 'attacks' have not succeeded in breaking it in the least (safe in
> dream).

Yes, it is safe. As stated, the current critical section is already unnecessary. (Notwithstanding the (im)possibility of always optimal placement, which has nothing to do with safety.)



> I'll be "off" the stage for some unpredictable time for reasons which I've
> alluded to earlier, health foremost

Get well soon!

> compounded by a load of material problems which will be surmounted,

Ah.

---
l

Taking a break out of retirement...

> You must have misunderstood me. What I proposed is the same.

> Clearly, DOS is never re-entered.*

> What I proposed is conceptually the same, just with memory allocation
> functions instead of file access functions:
>
> i21:
> cmp ah, 48h
> jne .next
> ...
> mov ah, 48h
> pushf
> push cs
> call .next
> ...
>

OK now what you'd in mind is clear. Some confusion came from the use of the term TSR, that in general I take to mean a (HW)interrupt activated TSR. What you have here are soft int 21 hookers (of course they are also TSRs in the general sense).

Then my response can be stated clearly too :

1. the "hooker" MUST NOT do its own (private, hidden) 21/48 call BEFORE effecting the user's (more foreground). That would be IMO a bug (because it changes the actual result expected by the foreground program), and I would never knowingly use any kind of monitoring program that did such things.

Besides there is little reason you would want to program the "hooker" in that rude manner. Presumably the mythical utility will use its hidden memory chunk to record or annotate the foregounder's memory operation, it had better wait till DOS returns.

2. IF otoh, "hooker" passed the unchanged user's request FIRST, and did its own allocation when DOS returns, it's ACCEPTABLE (and in particular it's OK with KBFR's strategy).

3. However, much better would be for your "hooker" to do its underground memory reservations, not from conventional memory which is a precious resource and should not leak or move or vanish mysteriously. Using scratch memory from XMS or EMS or similar looks right. This is no more 1981 !

>> - keep my loading scheme as is - not only is is working in practice,
> also
>> thought 'attacks' have not succeeded in breaking it in the least (safe
> in
>> dream).

> Yes, it is safe. As stated, the current critical section is already
> unnecessary. (Notwithstanding the (im)possibility of always optimal
> placement, which has nothing to do with safety.)

Optimal placement is granted unless backgrounders don't behave. If not, so much the worse! (or use the alternate method I've outlined, basically do "function 48" by hand still under CLI)

> Get well soon!

Doing my best...

---
Ninho

> Some confusion came from the use of the term TSR, that in general I take
> to mean a (HW)interrupt activated TSR. What you have here are soft int 21
> hookers (of course they are also TSRs in the general sense).

Ah, I see. I usually note it specifically if I am talking about "asynchronously" hardware interrupt activated, or "pop up" TSRs. The term TSR otherwise means TSR in the general sense to me.

(Though actually, "Terminate and Stay Resident" isn't even that good a specifier for some of the programs I would describe as TSRs.)

> 1. the "hooker" MUST NOT do its own (private, hidden) 21/48 call BEFORE
> effecting the user's (more foreground). That would be IMO a bug (because it
> changes the actual result expected by the foreground program)

I disagree.

The caller shouldn't expect anything in that way.* After all, calling interrupt 21h means to expect that interrupts might be enabled at some point. And that means that a TSR which is activated by a hardware interrupt could interrupt the currently processed interrupt 21h call and allocate memory for itself before the interrupt 21h call reached DOS's function 48h handler.

Whether the intermediate allocation is in fact issued by an interrupting TSR or by one which is intercepting the function 48h call is a hidden detail that doesn't change the caller's situation.

* If the caller did depend on such expectations to be true, that would in my opinion be a bug (because such dependency makes the scheme so fragile it obviously can be broken by resident software or incompatibilities). Fortunately, your installer doesn't generally depend on that.

> Besides there is little reason you would want to program the "hooker" in
> that rude manner. Presumably the mythical utility will use its hidden
> memory chunk to record or annotate the foregounder's memory operation, it
> had better wait till DOS returns.

Why? If its allocation is just as permanent as that of the caller, then it doesn't matter (memory-layout-wise) whether the intermediate handler does its own or the caller's allocation first.

If the intermediate handler's own allocation can indeed be said to be permanent in some way, then it is (from the handler's point of view) advantageous to do its own allocation first - because the caller _might_ want to allocate that memory only temporarily. And if that was the case, then the resulting memory layout would possibly not be optimal if the handler allocated (temporarily used) memory for the caller first and then allocated (permanently used) memory for itself afterwards.

> 3. However, much better would be for your "hooker" to do its underground
> memory reservations, not from conventional memory which is a precious
> resource and should not leak or move or vanish mysteriously. Using scratch
> memory from XMS or EMS or similar looks right. This is no more 1981 !

Then we'll be discussing whether your XMS or EMS memory allocation scheme is optimal instead

> > Notwithstanding the (im)possibility of always optimal placement,
>
> Optimal placement is granted unless backgrounders don't behave. If not, so
> much the worse!

I would rephrase this as: Optimal placement is achieved if no resident software interferes with temporary allocations during one particular time frame. If not, then oh well the placement isn't optimal but that isn't the end of the world or anything.

As replacing 21.48 entirely is undesirable, that's (almost*) the best method there is.

* To illustrate, there's better methods available that, though not replacing 21.48, are nonetheless overly elaborate:

For example, one could _not_ free the process's environment and the memory used by the old PSP initially, then allocate all available memory as fast as possible, then decide which location would be best for the TSR - and relocate the process and TSR image as often as necessary to get the latter there. Afterwards, all memory not used by the TSR is of course to be freed.

A less extreme method would not free the environment and PSP memory initially as well; it would instead resize both of those blocks to the largest possible size and then try to allocate a third block (with the exact size required by the TSR). It would then decide which location inside the allocated blocks would be the best for the TSR. Relocation and freeing of the unused memory later would behave like in the previous method.

The only advantage these two extreme methods have over our current methods is that the time frame during which temporary allocations of other resident software can disrupt optimal placement would be shortened - it never goes away. (If in doubt, imagine again that resident software might temporarily allocate some memory even before the TSR's installer process is loaded.)

---
l

>> 1. the "hooker" MUST NOT do its own (private, hidden) 21/48 call BEFORE
>> effecting the user's (more foreground). That would be IMO a bug (because
> it changes the actual result expected by the foreground program)
>
> I disagree.

Then I agree to disagree. Or, I would replace the term "bug" by "bad design", at the very least.

> The caller shouldn't expect anything in that way.* After all, calling
> interrupt 21h means to expect that interrupts might be enabled at some
> point. And that means that a TSR which is activated by a hardware interrupt
> could interrupt the currently processed interrupt 21h call and allocate
> memory for itself before the interrupt 21h call reached DOS's function 48h
> handler.

This is the same point. In general because a DOS TSR (in the general acception) is in a position to break the expected "semantics" of exterior DOS calls does not imply that it should indulge itself into doing so, especially when it's gratuitous.

I admit there are disputable limits. Take the specifics of this DOS "request memory block" : you would say, function 48 caller asks for a certain size block, so if it gets one, who's to complain ? But in fact caller expects /more/ than just any properly sized block, he has the right to expect DOS to allocate that block following a specified algorithm [this is an important difference from XMS/EMS], besides his right is confirmed by the ability to choose between several allocation "strategies". Clearly a TSR which follows scenario number 1 is "cheating" our foreground prog.


>> Besides there is little reason you would want to program the "hooker" in
>> that rude manner. Presumably the mythical utility will use its hidden
>> memory chunk to record or annotate the foregounder's memory operation,
> it
>> had better wait till DOS returns.

> Why? If its allocation is just as permanent as that of the caller, then it
> doesn't matter (memory-layout-wise) whether the intermediate handler does
> its own or the caller's allocation first.

It does matter, not size-wise :=) but position-wise it does.

> If the intermediate handler's own allocation can indeed be said to be
> permanent in some way, then it is (from the handler's point of view)
> advantageous to do its own allocation first - because the caller _might_
> want to allocate that memory only temporarily. And if that was the case,
> then the resulting memory layout would possibly not be optimal if the
> handler allocated (temporarily used) memory for the caller first and then
> allocated (permanently used) memory for itself afterwards.

I think the TSR has no right nor reason to "speculate" thusly

>> 3. However, much better would be for your "hooker" to do its underground
>> memory reservations, not from conventional memory which is a precious
>> resource and should not leak or move or vanish mysteriously. Using
> scratch
>> memory from XMS or EMS or similar looks right. This is no more 1981 !
>
> Then we'll be discussing whether your XMS or EMS memory allocation scheme
> is optimal instead

Not comparable : EMS or XMS blocks real location is opaque in principle, there is nothing to "optimise" (from the client's PoV; of course the EMM or XMM is allowed and encouraged to do its own optimisation and move blocks around.)


>>> Notwithstanding the (im)possibility of always optimal placement,
>> Optimal placement is granted unless backgrounders don't behave. If not,
>> so much the worse!

> I would rephrase this as: Optimal placement is achieved if no resident
> software interferes with temporary allocations during one particular time
> frame. If not, then oh well the placement isn't optimal but that isn't the
> end of the world or anything.

If you will, with the understanding that interference during the...time frame should be disallowed.

> As replacing 21.48 entirely is undesirable, that's (almost*) the best
> method there is.

Agreed. And the risk of interferences of the kind we've been making precise, whether you call them bugs or not, is small in the real world (I dare hope it is). Still for when things go better I do not exclude making KBFR walk the (low mem only) MCBs. It's not conceptually complicated (no need to process the general case) and might give us somewhat of a warm feeling.

---
Ninho

Ugh, forgot to save my half-written post elsewhere before I closed my browser. I'll reconstruct this from memory, so complain if you don't understand anything.

> This is the same point. In general because a DOS TSR (in the general
> acception) is in a position to break the expected "semantics" of exterior
> DOS calls does not imply that it should indulge itself into doing so,
> especially when it's gratuitous.

You can't say that about both interceptor- and interrupter-type TSRs. The interceptor arguably should behave in a way so as not to break the semantics you expect, but if the interrupter interrupts the currently processed 21.48 call at the right time, then it doesn't know that your 21.48 call is currently being serviced. So it can't avoid breaking the semantics you expect in that case.

> you would say, function 48 caller asks for a
> certain size block, so if it gets one, who's to complain ?

Basically yes, even though that's not my main point.

However, as noted, the interrupter conceptually could have interrupted your program earlier and could have allocated memory way before you issued your 21.48 request to the topmost interrupt 21h handler. (In practice, you can keep the interrupter out for some time by disabling interrupts in your own code. But by calling interrupt 21h, you accept that interrupts may be enabled elsewhere - and that therefore the interrupter might gain control.)

> But [the caller] has the right to expect DOS to allocate that block
> following a specified algorithm

Still disagreeing.

> besides his right is confirmed by the ability to choose between several
> allocation "strategies".

I don't think the strategies are that important. For example, disabling the UMB link (21.5803.bx=0) overrides all area specifications in the current strategy and searches in the lower memory area instead. (Yes, even if one of the 40h strategies (Upper memory area only) was set.)

> Clearly a TSR which follows scenario number 1 is "cheating"
> our foreground prog.

Again, even though arguably the interceptor should behave so as to fulfil your expectations, the interrupter can't know it is interrupting your program after your program called interrupt 21h to issue a 21.48 request.

> It does matter, not size-wise :=) but position-wise it does.

No, I meant it doesn't matter optimality-wise either.

> I think the TSR has no right nor reason to "speculate" thusly

The (intercepting) TSR must speculate one way or another. Either it issues its own request before the caller's, or after the caller's.

As I explained, if the interceptor deems its own allocation permanent, then it is advantageous to issue its own allocation request first. It isn't advantageous to issue it afterwards.

So if the interceptor needs to allocate memory permanently and you disallow it to issue its own allocation request first, you'll often (every time there's a temporary allocation made using 21.48) force the interceptor to use the worse method.

I thought of another reason why it is advantageous for the interceptor to issue its own request first. Actually, a classic reason for issuing any interceptor's own requests to the intercepted service before issuing the caller's request. Examine this interceptor:

i21:
    cmp ah, 48h
    je .handle
.chain:
    jmp far [cs:.next]

.handle:
    ...
    push ax
    push bx
    ...
    mov ah, 48h
    mov bx, ...
    pushf
    push cs
    call .chain            ; call for this handler's own request
    ...
    pop bx
    pop ax
    ...
    jmp short .chain       ; chain for caller's request

Now examine this one, modified to issue the caller's request first as you demand:

i21:
    cmp ah, 48h
    je .handle
.chain:
    jmp far [cs:.next]

.handle:
    ...
    pushf
    push cs
    call .chain            ; call for caller's request
    push bp
    mov bp, sp
    rcr byte [bp+6], 1     ; flip
    rol byte [bp+6], 1     ; flop
    pop bp
    ...
    push ax
    push bx
    ...
    mov ah, 48h
    mov bx, ...
    pushf
    push cs
    call .chain            ; call for this handler's own request
    ...
    pop bx
    pop ax
    ...
    iret

As you see, in this case the intermediate handler can never simply chain to DOS (as it needs to post-process the results of its own requests). Because of that, and this order of requests (where the caller's request can't be the last one) it also needs to correctly pass the Carry Flag from DOS to the original caller.

> If you will, with the understanding that interference during the...time
> frame should be disallowed.

"Should", but it's not possible to.

---
l