DOS ain't dead

Regarding UPXINFO I have a few questions about Free Pascal. Maybe Laaca or Marco could help me.

1) How to generate a map file? "-Xm" produces no map file for me. Although no error is shown.
2) How to achieve minimum size of executables? I'm using "-O3 -Os -XX" now. Unfortunately FPC's help screen doesn't show default settings as TPC does.
3) Is it possible to put all these options into my source code? For example documentation says "-XX" equals to "{$SMARTLINK}", but that seems to be wrong. Maybe "-CX" is meant?! Also text-mode IDE (FP.EXE) doesn't insert all options, when I press Ctrl-O twice.

---
Forum admin

> Regarding UPXINFO I have a few questions about Free Pascal. Maybe Laaca or
> Marco could help me.
>
> 1) How to generate a map file? "-Xm" produces no map file for me. Although
> no error is shown.

I'll look into it. That feature is newer than the renewed interest in the go32v2, so maybe it was never implemented.

However FPC allows for a lot of tweaking of the linker, e.g. passing GNU ld linker params using -k will get you there:

fpc -k"-Map mymapfile.map" helloworld

(works for me WITH the doscompiler, but under XP)

If you want to have an idea what happens under the hood, compile with -s and view the generated files (script.res ppas.bat link.res)

Note that I have never used a map file with FPC ever. I usually use -gl that will print full tracebacks with file-lineinfo on crashes. (which, if you recompile the RTL with -gl can go through the entire RTL)

It's different if you want to trace back bugreports with small bins though.

> 2) How to achieve minimum size of executables? I'm using "-O3 -Os -XX"
> now. Unfortunately FPC's help screen doesn't show default settings as TPC
> does.

-CX to compile units, -XX to compile/link the final file. The optimization stuff is really peanuts. (IIRC it works a bit better in the development versions because Florian wants to target embedded CPUs, AVR Atmel in the near future, but in the 2.2 series it is peanuts)

If you want to strip the result use -XXs, but some non Unix versions of ld do this badly, so there I usually strip with the provided separate strip.exe binary.

> 3) Is it possible to put all these options into my source code? For
> example documentation says "-XX" equals to "{$SMARTLINK}", but that seems
> to be wrong. Maybe "-CX" is meant?!

The original idea was -XX is "link smartlink" and -CX is "compile smartlink". This because you might not want to smartlink for speed in development builds, but don't want to recompile everything. (in the beginning both -CX and -XX were very slow. This was later sped up by eliminating AS).

Note that, like TP, most $'s are not global, but for the compiled unit (or mainprogram) only. So then you'd have to put them in an inc and include that anywhere.

> Also text-mode IDE (FP.EXE) doesn't
> insert all options, when I press Ctrl-O twice.

Some of the newer functions were never finished. (e.g. there is no way to make code folds persistent), and a lot of infrastructure was never updated for them.

Please file a bug in mantis, preferably with a small note what you try and what you expect. While the IDE development is not die-hard, occasionally people single out a bug and fix it.

(you might want to try yourself even, since the IDE is just a (slightly big) FPC application.

I myself am thinking to try to add CHM support to the IDE. Or at least limited enough CHM to read the autogenerated FPC chm with the unit data. That would solve the 20000 files on disc issue.

> I myself am thinking to try to add CHM support to the IDE. Or at least
> limited enough CHM to read the autogenerated FPC chm with the unit data.
> That would solve the 20000 files on disc issue.

YES! It would be great.

---
DOS-u-akbar!

> > I myself am thinking to try to add CHM support to the IDE. Or at least
> > limited enough CHM to read the autogenerated FPC chm with the unit
> data.
> > That would solve the 20000 files on disc issue.
>
> YES! It would be great.

Laaca, some FPC loose ends:
1) did you notice the "symbolic" package at the end of the first page of the 2.2.2 installer? That's the expression evaluator I talked about before.
2) Have you seen this post?: http://www.bttr-software.de/forum/forum_entry.php?id=4816&page=1&category=0&order=time

(if RR (or anybody else) is playing with the FPC IDE, it is also for them)

> However FPC allows for a lot of tweaking of the linker, e.g. passing GNU
> ld linker params using -k will get you there:
> fpc -k"-Map mymapfile.map" helloworld
> (works for me WITH the doscompiler, but under XP)

Works for me too. Thanks!

> > 2) How to achieve minimum size of executables? I'm using "-O3 -Os -XX"
> > now. Unfortunately FPC's help screen doesn't show default settings as
> TPC
> > does.
>
> -CX to compile units, -XX to compile/link the final file. The optimization
> stuff is really peanuts. (IIRC it works a bit better in the development
> versions because Florian wants to target embedded CPUs, AVR Atmel in the
> near future, but in the 2.2 series it is peanuts)
>
> If you want to strip the result use -XXs, but some non Unix versions of ld
> do this badly, so there I usually strip with the provided separate
> strip.exe binary.

I see, but it still doesn't answer my initial question.

> The original idea was -XX is "link smartlink" and -CX is "compile
> smartlink". This because you might not want to smartlink for speed in
> development builds, but don't want to recompile everything. (in the
> beginning both -CX and -XX were very slow. This was later sped up by
> eliminating AS).

But then that's a bug in FPC's documentation! Chart.pdf shows "XX = $SMARTLINK = Use smartlinking." Btw: Did you receive my mails from last Friday?

> Note that, like TP, most $'s are not global, but for the compiled unit (or
> mainprogram) only. So then you'd have to put them in an inc and include
> that anywhere.

That would be OK, but the problem is that not all command-line options have a source-code equivalent. So I have to use a mixture of both. That's what I dislike.

---
Forum admin

> > > 2) How to achieve minimum size of executables? I'm using "-O3 -Os
> > > -XX" > now. Unfortunately FPC's help screen doesn't show default settings as TPC does.
> > I see, but it still doesn't answer my initial question.

Well, I only replied what is needed on top of what you specified. If you strip and compile all code with -CX and link with -XX, most of the important stuff is there. In theory you could maybe tighten it up by increasing the minimal processor (some conditional moves here and there), but I'd guess that's all sub 100 bytes on even a small exe.

> > The original idea was -XX is "link smartlink" and -CX is "compile
> > smartlink". This because you might not want to smartlink for speed in
> > development builds, but don't want to recompile everything. (in the
> > beginning both -CX and -XX were very slow. This was later sped up by
> > eliminating AS).
>
> But then that's a bug in FPC's documentation! Chart.pdf shows "XX =
> $SMARTLINK = Use smartlinking."

Well, the problem is that they both are. The compiler-linker separation needs two switches instead of one though.

> Btw: Did you receive my mails from last Friday?

Yes. As you can see, the ligature problem is solved. I still have to look into the other issues.

> > Note that, like TP, most $'s are not global, but for the compiled unit
> (or
> > mainprogram) only. So then you'd have to put them in an inc and include
> > that anywhere.

> That would be OK, but the problem is that not all command-line options
> have a source-code equivalent. So I have to use a mixture of both. That's
> what I dislike.

As far as I know, this was never considered a design requirement. At least I can't really remember anybody keeping those in sync. (mostly also, because a global inc file is the only way). A directive was only implemented if there was a need.

Best to either discus it on the mailling list, or file a bug (preferably which include an analysis of which params have no corresponding directives).

> > But then that's a bug in FPC's documentation! Chart.pdf shows "XX =
> > $SMARTLINK = Use smartlinking."
>
> Well, the problem is that they both are. The compiler-linker separation
> needs two switches instead of one though.

I see. But why does FPC's documentation state that command-line option "-XX" equals to source code directive "$SMARTLINK". (That's how I understand it.) If I put "$SMARTLINK" to my program resulting EXE is not smartlinked! It gets only smartlinked, if I specify "-XX" on command line.

Can you tell me what part in FP writes the go32stub to an EXE file? Or is it LD.EXE? I found "go32v2stub" in "ogcoff.pas", but when I write this constant to a temporary file, it still differs from the stub of UPXDUMP.

Apropos: I can't unpack LD.EXE (or other binaries you borrow from DJGPP), because FP still uses the DJGPP 2.03-current chain, where Binutils were compressed by a "crazy" UPX build. (Blame Gordon Schumacher of Seagate. ) Juan Manuel Guerrero later released new binaries (no compression)for the 2.04-beta chain. Better to switch over, I think.

---
Forum admin

> > > But then that's a bug in FPC's documentation! Chart.pdf shows "XX =
> > > $SMARTLINK = Use smartlinking."
> >
> > Well, the problem is that they both are. The compiler-linker separation
> > needs two switches instead of one though.
>
> I see. But why does FPC's documentation state that command-line option
> "-XX" equals to source code directive "$SMARTLINK". (That's how I
> understand it.) If I put "$SMARTLINK" to my program resulting EXE is
> not smartlinked! It gets only smartlinked, if I specify "-XX" on
> command line.

I don't get that from the docs:
http://www.freepascal.org/docs-html/prog/progsu102.html#x110-1100001.2.26

first line:

"A unit that is compiled in the {$SMARTLINK ON} state will be compiled in such a way that it can be used for smartlinking."

> Can you tell me what part in FP writes the go32stub to an EXE file? Or is
> it LD.EXE? I found "go32v2stub" in "ogcoff.pas", but when I write this
> constant to a temporary file, it still differs from the stub of UPXDUMP.

As far as I can see when linking wiht -s, and objdumping the .o of the mainprogram, I'd say FPC has nothing to do with it.

That routine looks like the coff assembler. IOW, it could be that the header is the header for each .o ?

> Apropos: I can't unpack LD.EXE (or other binaries you borrow from DJGPP),
> because FP still uses the DJGPP 2.03-current chain, where Binutils were
> compressed by a "crazy" UPX build. (Blame Gordon Schumacher of Seagate.
> ) Juan Manuel Guerrero later released new binaries (no compression)for
> the 2.04-beta chain. Better to switch over, I think.

Depends. For such a change, some testing needs to be done by people that actually understand dos/dpmi issues. I don't. Filing a bug about it can help avoid it being forgotten, and act as a kind of forum to discuss it and exchange epxeriences

> > Apropos: I can't unpack LD.EXE (or other binaries you borrow from
> DJGPP),
> > because FP still uses the DJGPP 2.03-current chain, where Binutils were
> > compressed by a "crazy" UPX build. (Blame Gordon Schumacher of Seagate.
> > ) Juan Manuel Guerrero later released new binaries (no
> compression)for
> > the 2.04-beta chain. Better to switch over, I think.
>
> Depends. For such a change, some testing needs to be done by people that
> actually understand dos/dpmi issues. I don't. Filing a bug about it can
> help avoid it being forgotten, and act as a kind of forum to discuss it
> and exchange epxeriences

Gordon is MIA, haven't seen him around online lately, and his old (year-old) e-mail address just bounces now. Besides, his compiles (/beta/, not /current/) were 686+ only! So I think (from memory) that FPC is doing the right thing by sticking to /current/. Not sure why it doesn't unpack, though, but you (rr) and I have both reported some of that to UPX authors previously as a potential bug. As long as AS assembles and LD links correctly, I wouldn't majorly worry about it.

P.S. Juan Manuel Guerrero had issues with a linker script re: building latest TexInfo, so he reverted his reversion of BinUtils 2.17 (686+ again, meh). Better to e-mail him regarding what issues he had, even if you do decide to rebuild your own.

> > I myself am thinking to try to add CHM support to the IDE
> YES! It would be great

COOL

Anything to read CHM's except original IE already around ? IIRC NO

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

> So I think (from memory) that FPC is doing the right thing by sticking
> to /current/.

Now that I know all facts, I agree with you.

> Not sure why it doesn't unpack, though, but you (rr) and I have both
> reported some of that to UPX authors previously as a potential bug.

I'll look into this again, because I'm just too curious.

---
Forum admin

> > I see. But why does FPC's documentation state that command-line option
> > "-XX" equals to source code directive "$SMARTLINK". (That's how I
> > understand it.) If I put "$SMARTLINK" to my program resulting EXE is
> > not smartlinked! It gets only smartlinked, if I specify "-XX" on
> > command line.
>
> I don't get that from the docs:
> http://www.freepascal.org/docs-html/prog/progsu102.html#x110-1100001.2.26
>
> first line:
>
> "A unit that is compiled in the {$SMARTLINK ON} state will be compiled in
> such a way that it can be used for smartlinking."

I was talking about (broken) http://www.freepascal.org/docs-html/chart/chart.html (or Chart.pdf).

> > Can you tell me what part in FP writes the go32stub to an EXE file? Or
> is
> > it LD.EXE? I found "go32v2stub" in "ogcoff.pas", but when I write this
> > constant to a temporary file, it still differs from the stub of
> UPXDUMP.
>
> As far as I can see when linking wiht -s, and objdumping the .o of the
> mainprogram, I'd say FPC has nothing to do with it.

I found it in Binutils' "src/bfd/go32stub.h" now.

> That routine looks like the coff assembler. IOW, it could be that the
> header is the header for each .o ?



> Depends. For such a change, some testing needs to be done by people that
> actually understand dos/dpmi issues. I don't. Filing a bug about it can
> help avoid it being forgotten, and act as a kind of forum to discuss it
> and exchange epxeriences

DJGPP 2.04 is very stable and works much better with current hardware or on Windows.

---
Forum admin

> Anything to read CHM's except original IE already around ? IIRC NO

Yes, there is. Just Google or look at Wikipedia.

---
Forum admin

> I found it in Binutils' "src/bfd/go32stub.h" now.

I don't really use FPC (don't know Pascal!), but DJGPP always seems to use STUBIFY.EXE (e.g. try "gcc -v" and you'll see it).

> DJGPP 2.04 is very stable and works much better with current hardware or
> on Windows.

Yes, but the issues are only with Win 2k, XP, etc. because of some NTVDM bugs that MS never fixed. CWS recently pointed me to what may be a simple patch for 2.03 and earlier buggy .EXEs (untested by me). And 2.03p2 (patchlevel 2) has the fixes backported, and FYI, 2.03p2 is still called the "stable" release (and still pointed to by ZIP Picker) since 2.04 "beta" never was fully finished. So both are acceptable, IMO, and the DJGPP maintainers do still try to "port" stuff to both. (That's what I heard was one of the reasons for the hold up for releasing the DJGPP port of GCC 4.3.x, no easily-made 2.03 binary.) Also, earlier versions (2.01 etc.) had more bugs re: LFNs, etc. And 2.02 got the new (non-BSD, more memory-conservative) malloc(). 2.04 has better symlink support (i.e. not just for .EXEs anymore) as well as better >2 GB file support, etc.

> > I found it in Binutils' "src/bfd/go32stub.h" now.
>
> I don't really use FPC (don't know Pascal!), but DJGPP always seems to use
> STUBIFY.EXE (e.g. try "gcc -v" and you'll see it).

That's correct, but FPC doesn't. So it probably comes from LD.

---
Forum admin

> > > I myself am thinking to try to add CHM support to the IDE
> > YES! It would be great
>
> COOL
>
> Anything to read CHM's except original IE already around ? IIRC NO

FPC has a chm lib since 2.2.2. Afaik they've generated .chms on Mac OS X with it. (packages/chm)

However it is not in mainstream use yet, and till now it is mostly the writer that has been stressed.

> > Anything to read CHM's except original IE already around ? IIRC NO
>
> Yes, there is. Just Google or look at Wikipedia.

The trouble is that CHM is roughly compressed html. So to faithfully reproduce any chm, you need to emulate nearly a full IE, with quirks mode, x86 activeX objects etc.

But for this kind of stuff that is less important. For now we limit ourselves to reading our own output.

> The trouble is that CHM is roughly compressed html. So to faithfully
> reproduce any chm, you need to emulate nearly a full IE, with quirks mode,
> x86 activeX objects etc.

There is also *.html.gz format. It's in the HTML spec, should be supported by compliant (all current?) browsers.

> > The trouble is that CHM is roughly compressed html. So to faithfully
> > reproduce any chm, you need to emulate nearly a full IE, with quirks
> mode,
> > x86 activeX objects etc.
>
> There is also *.html.gz format. It's in the HTML spec, should be supported
> by compliant (all current?) browsers.

(Are you sure? Afaik HTTP2 allows for GZ compression, but .gz on HTML sounds like an Unix extension, not something a standard would describe, since .gz is meaning less on a standard windows install. Do you have a reference?)

Well, the crucial part is the combination of archive functionality with compression, and an preexisting viewer (This because the slack space of tens of thousands of little html files is as important as the compression).

Also .chm has the ability to embed internal files, which are important for additional tables for e.g. context sensitive help etc.

All browsers will be internal anyway, since starting up and fully instrumenting the many commonly used browsers is next to impossible.

> > There is also *.html.gz format. It's in the HTML spec, should be
> supported by compliant (all current?) browsers.

> (Are you sure? Afaik HTTP2 allows for GZ compression, but .gz on HTML
> sounds like an Unix extension, not something a standard would describe,
> since .gz is meaning less on a standard windows install. Do you have a
> reference?)

Such files are found on Unix servers, and get used in the usual two ways: They can be read online like any other .HTML file, or downloaded. A compliant browser decompresses for reading. Opera does it for sure, using zlib.

> Well, the crucial part is the combination of archive functionality with
> compression, and an preexisting viewer (This because the slack space of
> tens of thousands of little html files is as important as the
> compression).

By preexisting viewer, do you mean one that the user already has? Or one that you don't have to write, but that you can just throw into the download package? Either way, I don't see any problem, when you put it in those terms. HTML files can be separately gzipped, or tarred first in the normal way. Images can remain as separate files from the original HTML. Viewer should be no problem - doesn't everybody have a browser?

The slack space problem goes away if raw HTML files are tarred before gzipping (or bzipping). An untar/decompressor can be in the download package with little cost in total size (like, *.tar.gz or *.tar.bz2, plus 7zip.exe inside a *.zip download file) - parallel to the old common practice of including a *.zip + unzipper on diskettes.

My point is, there are purely mechanical solutions that do not require programming other than revising scripts for packing files and uploading to servers.

> Also .chm has the ability to embed internal files, which are important for
> additional tables for e.g. context sensitive help etc.

Perhaps .ps or .pdf, with batch files to run Ghostscript or other free reader?

> All browsers will be internal anyway, since starting up and fully
> instrumenting the many commonly used browsers is next to impossible.

Yes, browsers are annoying, with so many variations and degrees of compliance with the HTML specs. But I see that as another reason to go with .ps or .pdf, and perhaps supply a reader on the FP servers.

> > (Are you sure? Afaik HTTP2 allows for GZ compression, but .gz on HTML
> > sounds like an Unix extension, not something a standard would describe,
> > since .gz is meaning less on a standard windows install. Do you have a
> > reference?)
>
> Such files are found on Unix servers, and get used in the usual two ways:
> They can be read online like any other .HTML file, or downloaded. A
> compliant browser decompresses for reading. Opera does it for sure, using
> zlib.

You didn't answer my real question. Is it really a standard, or do some browsers simply do this (since they have zlib support internal for http2 anyway)?

> > Well, the crucial part is the combination of archive functionality with
> > compression, and an preexisting viewer (This because the slack space of
> > tens of thousands of little html files is as important as the
> > compression).
>
> By preexisting viewer, do you mean one that the user already has? Or one
> that you don't have to write, but that you can just throw into the
> download package?

No, more that there are several basic embeddable html viewers, while finding e.g. a RTF widget is hard (for winhelp .hlp). E.g. IPRO.

> Either way, I don't see any problem, when you put it in
> those terms. HTML files can be separately gzipped, or tarred first in the
> normal way. Images can remain as separate files from the original HTML.
> Viewer should be no problem - doesn't everybody have a browser?

Yes. But we are not viewing one simple page here, but an hyperlinked archive that must be available on a simple keypress, and with an index to multiple entrypoints, and with minimal memory use.

And an external program leads to installation/detection and instrumentation problems. E.g. for browsers to force them to change pages, to trap hyperlinks so you can lead them to the next page in the ZIP, etc is _NOT_ standarized.

So I can extract an html file from the ZIP and start it up via some OS feature (e.g. shellexecute) in the browser, but what happens if the user clicks on a link in the shown file? How do I redirect that to the next page in the zip?

Also, the automated helpsystem saves space, since the basic controls don't need to be coded on each page.

> The slack space problem goes away if raw HTML files are tarred before
> gzipping (or bzipping).

Yes. IF YOU LEAVE THEM ZIPPED AFTER INSTALL. But then I can't browse them. The zip or tarring itself is no problem, the (de)compressor is in the installer already. It is kind of slow though with tens of thousands of files. (and with 4k clustersize this quickly turns into tens of MBs, while it is is a 6MB chm)

> My point is, there are purely mechanical solutions that do not require
> programming other than revising scripts for packing files and uploading to
> servers.

I afraid I totally don't get what point you want to make here. Either you are reacting out of reflex against .chm (Microsoft paranoia?), or you totally lost me.

Sure, I can zip html files. But is that equal to a indexed helpfile system? Can I use a mainstream browser to browse such zip as hypertext? NO.

Is there a standard helpfile system on *nix ? :no.

> > Also .chm has the ability to embed internal files, which are important
> for
> > additional tables for e.g. context sensitive help etc.
>
> Perhaps .ps or .pdf, with batch files to run Ghostscript or other free
> reader?

There already is PS/PDF. Beautifully typeset using LaTeX. Even published as a book (though in German). If you have a decent way to turn it into an helpsystem, let me know.

> > All browsers will be internal anyway, since starting up and fully
> > instrumenting the many commonly used browsers is next to impossible.
>
> Yes, browsers are annoying, with so many variations and degrees of
> compliance with the HTML specs. But I see that as another reason to go
> with .ps or .pdf, and perhaps supply a reader on the FP servers.

Any external solution is a maintenance nightmare. I want to display an help text, not debug why the new version of viewer package X version B breaks on OS Y version Z. Specially since I have no interest in package X itself in the first place, except displaying the bloody helptext.

Also using external tools (for the final install, not in crafting the binaries of the install) precludes alternate functionality based on the helpfiles like displaying help in tooltips.

Free Pascal goes pretty far in eliminating external tools, for maintenance and control reasons btw. E.g. on Windows we have elmininated GNU AS and LD, and on the more common (x86,x86_64) ELF targets also AS. Just to avoid the horrible maintenance nightmare, and Unix centric thinking that haunts Free Software on non Unix targets. Unfortunately, there is no substitute for GDB.

> You didn't answer my real question. Is it really a standard, or do some
> browsers simply do this (since they have zlib support internal for http2
> anyway)?

Browsers _should_ be able do it because, a) compression is an http specification (not for Unix only), b) there is a gzip specification. zlib works because it is essentially part of gzip.
http://www.http-compression.com/
http://www.gzip.org/zlib/rfc-gzip.html

> > The slack space problem goes away if raw HTML files are tarred before
> > gzipping (or bzipping).
>
> Yes. IF YOU LEAVE THEM ZIPPED AFTER INSTALL. But then I can't browse
> them. The zip or tarring itself is no problem, the (de)compressor is in
> the installer already. It is kind of slow though with tens of thousands of
> files. (and with 4k clustersize this quickly turns into tens of MBs, while
> it is is a 6MB chm)

I typed too fast and didn't separate threads adequately. Tarring of course is useful in building the download package. A tar file can contain html.gz files just as well as anything else, and then gzipped or bzipped or 7-zipped for transmission. In use, html.gz files are as readable as any other html files.

> > My point is, there are purely mechanical solutions that do not require
> > programming other than revising scripts for packing files and uploading to
> > servers.
>
> I afraid I totally don't get what point you want to make here. Either you
> are reacting out of reflex against .chm (Microsoft paranoia?), or you
> totally lost me.

I was focusing on download package size, because I think reducing download time is more important than disk space after download. Really, what's the big deal about a few MB on today's disks?. And maybe I'm MS-paranoid, so what?

> Sure, I can zip html files. But is that equal to a indexed helpfile
> system? Can I use a mainstream browser to browse such zip as hypertext?
> NO.
>
> Is there a standard helpfile system on *nix ? :no.
>
> > > Also .chm has the ability to embed internal files, which are important for
> > > additional tables for e.g. context sensitive help etc.
>
> > Perhaps .ps or .pdf, with batch files to run Ghostscript or other free
> > reader?
>
> There already is PS/PDF. Beautifully typeset using LaTeX. Even published
> as a book (though in German). If you have a decent way to turn it into an
> helpsystem, let me know.

PDF can have hyperlinks and indexes, right?

> > Yes, browsers are annoying, with so many variations and degrees of
> > compliance with the HTML specs. But I see that as another reason to go
> > with .ps or .pdf, and perhaps supply a reader on the FP servers.
>
> Any external solution is a maintenance nightmare. I want to display an
> help text, not debug why the new version of viewer package X version B
> breaks on OS Y version Z. Specially since I have no interest in package X
> itself in the first place, except displaying the bloody helptext.

> Also using external tools (for the final install, not in crafting the
> binaries of the install) precludes alternate functionality based on the
> helpfiles like displaying help in tooltips.
>
> Free Pascal goes pretty far in eliminating external tools, for maintenance
> and control reasons btw.

Understood. So here is yet another option: Rather than a tightly integrated CHM system, still separate the data files from the viewer. You keep control, need to maintain only your own code, and can issue data or viewer updates in small packages. Also, the user can customize data files (add personal notes, change colors for ergonomic reasons...). It's a tradeoff - some loss of speed (how much really?) and some increase in disk space, but with the gain of greater overall flexibility (my preference).

> E.g. on Windows we have elmininated GNU AS and LD,
> and on the more common (x86,x86_64) ELF targets also AS. Just to avoid the
> horrible maintenance nightmare, and Unix centric thinking that haunts Free
> Software on non Unix targets. Unfortunately, there is no substitute for
> GDB.

I don't see why Unix centric is worse than Windows centric, especially considering that the Linux user base is growing at the expense of Windows.

> In use, html.gz files are as readable as any
> other html files.

I just tried with IE6, and it doesn't open them.

> I was focusing on download package size, because I think reducing download
> time is more important than disk space after download. Really, what's the
> big deal about a few MB on today's disks?. And maybe I'm MS-paranoid, so
> what?

Well, try to enter a file with 100000 files and you know it is more than just size.

> > There already is PS/PDF. Beautifully typeset using LaTeX. Even
> published
> > as a book (though in German). If you have a decent way to turn it into
> an
> > helpsystem, let me know.
>
> PDF can have hyperlinks and indexes, right?

Yes. And they have those, even since before hyperref and pdf(la)tex were parts of the standard distro's (think 1997). Actually the PDF docs are considered authorative, and the HTML ones just a derivative.

That makes it an hyperrefed document though, not an helpsystem. PDF is even worse, since acroread is cumbersome to install (must download it separately, on *nix need to include non-free repositories etc)

> > Free Pascal goes pretty far in eliminating external tools, for
> maintenance
> > and control reasons btw.
>
> Understood. So here is yet another option: Rather than a tightly
> integrated CHM system, still separate the data files from the viewer.

Maybe I should take a step back and describe some of the requirements:

- The main point is that we want to have an helpsystem. Not just a bunch of hapazardly packaged files on disk.
- Also we prefer to use something that at least at some systems is native. That limits it to OS X help and Windows .HLP and .CHM. OS X is an unknown factor, for .hlp and .chm heaps of third party helpcompilers, decryptors, converts etc exist.
- The amount of additional work must be minimal. While the number of free helper utils was bigger for .hlp than for .chm a few years back, CHM was chosen because the html format is simply better known and there is a chance of recycling free third party html viewer widgets as core of the helpviewer on e.g. non Windows systems).
- Format must allow embedding third party streams, which is important for indexes of context-sensitive help systems. Both .hlp and .chm are essentially filesystems-in-a-file.
- Preferably system that uses one file per library, but allows inter-file links. The experience with loose HTML files has been sobering, and there as a "never that again" attitude. Renaming heaps of helpfiles to 8.3 syntax while keeping links intact is cumbersome.
- It must be usable in some form from the textmode IDE preferably even under Dos(what was this forum about again? . IOW, there must be native access to the format. You can't simply start an external helper there. Native access also allows limited reformatting to fit the textmode screen.

> You> keep control, need to maintain only your own code, and can issue data > or viewer updates in small packages.

Ah, but keep in mind that the .chms are not manmade. They are machine generated, and considered throwaway! Editing is not required, since both the master documentation files, as the tools to generate the .chms are multi-platform and open.

> Also, the user can customize data files
> (add personal notes, change colors for ergonomic reasons...). It's a
> tradeoff - some loss of speed (how much really?)

The problem is more in the installation, distribution size, and keeping 8.3 and case sensitivity problems from ruining releases. Help popup speed is less of a problem, except for building indexes and so.

Also moving or compressing an existing FPC installation (for backup or moving purposes) gets cumbersome since the filemanager/compressor spends ages in the huge directories searching for little files.

> and some increase in disk
> space, but with the gain of greater overall flexibility (my preference).

I rather spend that space in providing the help(de)compiler in the pkg, for the e.g. occasional teacher that wants to annotate/fix some minor issue directly in the help, and not wait for a new release.

(split because of size)

> > Just to avoid
> > the horrible maintenance nightmare, and Unix centric thinking that haunts
> > Free Software on non Unix targets. Unfortunately, there is no substitute
> > for GDB.
>
> I don't see why Unix centric is worse than Windows centric,

I never said I was Windows centric. OTOH I don't want to dismiss this entirely, so here are my main gripes:

Our focus is more on the ability to maintain an external package independantly. This is hard nearly everything except Windows. Even the Vista transition is a picknick compared with an OS/X major version or packaging one release for the heaps of fragmentated little Linux distro's.
All of which with their own packagemanagers, archive formats, guidelines and policies; Linux users always describe themselves as technofreaks, but IMHO they are rather bureaucrats that even put the former communist bureaucracy to shame.

Aside from that I think that Unix drives its originally sound "little independant tools" policy too far. For larger packages it becomes a framework that is slow, fragile, and wrought with complex dependancies.

> especially considering that the Linux user base is growing at the expense of Windows.

Peanuts. OS X users are a relatively larger Unix usergroup than Linux
But more importantly, Windows is still number one, despite Linux being in the "quick grower" spot for over ten years now.

Oh, and I'm an old BSDer originally btw, dabbling with Unix at home from before Linux existed, so please don't call be Windows sensitive just because I dare to criticise things that are IMHO wrong in Unix land.

> > Anything to read CHM's except original IE already around ? IIRC NO
> Yes, there is. Just Google or look at Wikipedia.

ERROR: file not found oops ... I don't have Linux, that's probably the problem

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

> > > Anything to read CHM's except original IE already around ? IIRC NO
> > Yes, there is. Just Google or look at Wikipedia.
>
> ERROR: file not found oops ... I don't have Linux, that's
> probably the problem

Sorry, but your only problem is, that you're always crying only instead of creating working solutions for DOS users.

---
Forum admin

> > > Anything to read CHM's except original IE already around ? IIRC NO
> > Yes, there is. Just Google or look at Wikipedia.
>
> ERROR: file not found oops ... I don't have Linux, that's
> probably the problem

http://www.nongnu.org/chmspec/latest/

> http://www.nongnu.org/chmspec/latest/

COOL I was not aware of this. CHM reading in DOS would be nice to have (otherwise I wouldn't have asked), OTOH I don't need it that badly ... and other things have more priority IMHO.

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

> > I myself am thinking to try to add CHM support to the IDE. Or at least
> > limited enough CHM to read the autogenerated FPC chm with the unit
> data.
> > That would solve the 20000 files on disc issue.
>
> YES! It would be great.

I committed this last night. Initially the IDE reads CHMs now. Moreover it was pretty unintrusive, so it can be merged to the stable branch without peril.

> committed this last night. Initially the IDE reads CHMs now. Moreover it
> was pretty unintrusive, so it can be merged to the stable branch

Well, some impressions of FPC 2.2.2:

+++ Ways better than 2.0.4, installer, IDE and compiler do mostly work now

- Still complains about LFN
- Bloat problem (see below)
- UPX license violation (???)
- Watcom and PIC support (???)

About bloat: there are 2 subproblems, output bloat (seems there are ways to reduce it, see DUPLICIT thread), and own bloat. Own bloat is incredibly easy to fix, I just for fun:

- Extracted all ZIP's at both levels
- Tried to un-UPX all EXE's, bad luck, many of them use UPX 2.93 (intended for testing only, not public redistribution) and don't uncompress
- Deleted redundant languages (isn't English sufficient for a "base" package), non-DOS specific examples, and, of course, UPX
- Brewed a new solid 7-ZIP archive with dict size 4 or 8 MiB (so install won't need more memory than using)

Results:

Original   : 17 MiB
dict 4 MiB :  9 MiB
dict 8 MiB :  7 MiB

So the bloat can be reduced by factor 2 by just dumb rearrangements

My suggestion for next version:

- Don't use or include UPX, keep all EXE's not UPX'ed
- Make all inner ZIP's uncompressed AKA Store
- Use 7-ZIP solid LZMA to archive the package

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

FPC 2.2.2 doesn't complain about non LFN environment more. It was fixed from 2.2.0a to 2.2.2

---
DOS-u-akbar!

> FPC 2.2.2 doesn't complain about LFN
> was fixed from 2.2.0a to 2.2.2

Meant the installer, not compiler core.

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

marcov wrote (in other thread):

> How does the textmode FPC ide fare?

Average, see below. Since you are deliberately participating in my text editor crashing challenge , hope you won't get too angry if I report some bugs

> Do you have that file online?

Sure: WOW (2.3 MiB)

> Note that the FPC textmode IDE has a limit on 256 char lines. Less of a
> problem with sourcecode, but with other files it can be limiting.

IMHO doesn't qualify a fault, as least for a text editor.

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

So the results:

- Pretty feature-rich, see shot above

- Doesn't have the mouse-cursor-ghosting-bug known from FD EDIT 0.9 , at least as long as there is no 25 vs 50 problem, see below

- Loading my 15 MiB file is a bit slow, there is no progress indicator and no abort, but still acceptable, minor thing, finally loads correctly.

- Editing this file seems to work, well responsive.

- Critical BUG : pressing CTL-C in the hope of copy into clipboard triggers an internal error and aborting IMHO CTL-C CTL-V and CTL-X should be supported and work (they are not advertised in current version ...). C&P using the alternative ways seems to work.

- Can't drag the scrollbars, but there is "GoTo Line" , so minor thing.

- Too long lines: no crash, lines got force split into pieces per 255 bytes at most, minor problem: instead of reporting the amount of evil lines, it reports the same thing twice, one time as "line -1"

- S&R with single confirm seems to work well

- S&R with "replace all" has problems. Maybe it's somewhat "correct" , but very slow (hog new memory and move complete text for every occurrence ?), strange cursor activity, no progress indicator, and no abort As a minimal P.I. the line number field active while editing could be used, the optimal solution would be 2-pass and a S&R progress box like:

S&R - count pass
Processing line 1'534 from total 987'223
Found 10'442 occurrences so far
Old size 15'753'643 , changed size so far 18'444'210
[abort]

S&R - replace pass
Processing line 1'534 from total 987'223
Replaced 10'442 from 123'777 occurrences so far
Old size 15'753'643 , new final size 23'466'277
[abort]

S&R - done
Replaced 123'777 occurrences
Old size 15'753'643 , new final size 23'466'277
[OK]

- 25 vs 50 lines: it seems to store the height into (at least) one of those numberous CFG/INI/.../??? files, and starting it next time in the "other" height doesn't work (no mouse, at least ALT-X intact). Deleting those files helps. Also there in no internal 25 vs 50 switching support (works well in INFOPAD, in FD EDIT it's present but doesn't work)

- Calculator: starting with 7 and clicking "x^2" ... after overflowing 64-bit, it switches to float (probably intentional), but few clicks later when float range is over as well the punishment is considerably harder

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

For copying text into clipboard press CTRL-ins
paste text: Shift ins
delete selected text: CTRL del

I like these combination more than windowish CTRL-C or CTRL-V because it works with russian keyboard loaded too.
(CTRL-C doen't work with russian keyboard, because it makes completely different ASCII code than programer thought. It is a problem of most windows programs too, by the way)

---
DOS-u-akbar!

> For copying text into clipboard press CTRL-ins
> paste text: Shift ins
> delete selected text: CTRL del

While that works, but I usually use ctrl-k codes. iirc ctrl-k g allows to get block from other window.

I'll reply to the other points later, am going away from work now.

Btw, also see

http://www.bttr-software.de/forum/forum_entry.php?id=4816&page=2&category=0&order=time

It's about the (mostly undocumented, and only halfway usable) code folding.

CHM support is slowly starting to work btw

> For copying text into clipboard press CTRL-ins
> paste text: Shift ins
> delete selected text: CTRL del
>
> I like these combination more than windowish CTRL-C or CTRL-V because it
> works with russian keyboard loaded too.
> (CTRL-C doen't work with russian keyboard, because it makes completely
> different ASCII code than programer thought. It is a problem of most
> windows programs too, by the way)

CTRL-C is a problem too because it is "terminate program" on nearly all commandline systems.

Laaca wrote:

> For copying text into clipboard press CTRL-ins
> paste text: Shift ins
> delete selected text: CTRL del

Sufficiently easy to find in edit menu and seems to work

> I like these combination more than windowish CTRL-C or CTRL-V
> because it works with russian keyboard loaded too.
> (CTRL-C doen't work with russian keyboard

The unkillable "argument" with Russian/German/French/Portugues/...blah... keyboards

> CTRL-C is a problem too because it is "terminate program" on nearly all
> commandline systems.

I am used to CTL-C and it works in FreeDOS EDIT, INFOPAD, KINESICS and FASMD. Not supporting (ignoring) CTL-C is a flaw only (but pretty important, at least for me personally), but crashing on CTL-C is a bug of course.

> reply to the other points later

of mine ?

> CHM support is slowly starting to work btw

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

> > I like these combination more than windowish CTRL-C or CTRL-V
> > because it works with russian keyboard loaded too.
> > (CTRL-C doen't work with russian keyboard
>
> The unkillable "argument" with Russian/German/French/Portugues/...blah...
> keyboards

What do you mean?

---
Forum admin

> > CTRL-C is a problem too because it is "terminate program" on nearly all
> > commandline systems.
>
> I am used to CTL-C and it works in FreeDOS EDIT, INFOPAD, KINESICS and
> FASMD.

Probably all date from after real dos times, by users that don't understand Dos.

I'm not the ultimate doshead, but I KNOW what ctrl-C means in dos, and it is not copy, that's truly Windows.

But if it is REALLY that important to you go to "options ->environment->mouse options" and stop whining.

> > reply to the other points later
>
> of mine ?

Yes, given time.

rr wrote:

> > The unkillable "argument" with
> What do you mean?

Doped keyboard explains everything. Among many others in this forum and other forums, IIRC the main reason why Japheth can't use FASM is his German keyboard

> Probably all date from after real dos times

irreal DOS ...

> by users that don't understand Dos

wow ...

> I'm not the ultimate doshead
> but I KNOW what ctrl-C means in dos, and it
> is not copy, that's truly Windows.

So what is it ?

> But if it is REALLY that important to you go to "options
> ->environment->mouse options"

There is nothing (see shot)



If I select M$, the edit menu gets adjusted and CTL-X and CTL-V start working. So the aforementioned "flaw" doesn't exist in FP IDE DOS, rather very good design

> and stop whining.

Regrettably the BUG is still in - CTL-C crashes in both CUA-91 and M$ mode. Except, of course, (since I KNOW what ctrl-C means in dos ) CTL-C indeed means "Cause Crash"

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

> Except, of course, (since I KNOW what ctrl-C means in dos
> ) CTL-C indeed means "Cause Crash"

If the program really crashes the system (and doesn't just terminate) when aborted by Ctrl-C, it should really hook (at least) Int23 and _ignore_ the Ctrl-C. Or the installed Int23 handler should unhook all hooked interrupts (other than 22, 23 and 24) and return to DOS to terminate without crashing.

---
l

Maybe it could be cured by adding the "djgpp_set_ctrl_c(false)" in the IDE source. However as far I remember in previous FPC version it worked not very reliably. (after few repeats it crashed)

In Blocek I use different method - I hook the keyboard interrupt and filter the CTRL-C combination.

---
DOS-u-akbar!

> Maybe it could be cured by adding the "djgpp_set_ctrl_c(false)" in the IDE
> source. However as far I remember in previous FPC version it worked not
> very reliably. (after few repeats it crashed)

Is that a go32 call? I would say the djgpp lib is only available when GDB is linked, FPC doesn't need the DJGPP lib.

> In Blocek I use different method - I hook the keyboard interrupt and
> filter the CTRL-C combination.

It worked in the past. Either go32v2, tool updates or recent (as in the last 5-7 years) have broken it. Probably needs new investigation.

Procedure "djgpp_set_ctrl_c" is defined in RTL\DPMIEXCP unit for GO32V2 target.

(BTW, what about Giulio Bernardi - does he still work on the DOS version?)

---
DOS-u-akbar!

> Procedure "djgpp_set_ctrl_c" is defined in RTL\DPMIEXCP unit for GO32V2
> target.
>
> (BTW, what about Giulio Bernardi - does he still work on the DOS version?)

Yes and no. Afaik he plans to but he will be very busy till december (or maybe even after) with his traineeship+mini thesis.

A 2.2.4 release is planned for +/- 6 months after 2.2.2, but there is some discussion if we should hold out for 2.4.

I think 2.4 is still far off, but I do see that merging to 2.2.x is getting harder lately. (more conflicts, more sources moved etc), so we can't keep releasing stable releases that really add the bulk of the fixes (as opposed to a few criticial ones) indefinitely.

If you have patches, bugs or suggestions, please enter them into the bugtracker. Many people will see that.

I'm currently playing with the IDE, but most of my plans there are non-dos related. (mostly .chm support and getting proper palette dialog support, backport that from C++ version).

But the IDE is more of a hobby, some other FPC tasks have priority.

> I'm currently playing with the IDE, but most of my plans there are non-dos
> related. (mostly .chm support and getting proper palette dialog support,
> backport that from C++ version).
>
> But the IDE is more of a hobby, some other FPC tasks have priority.

Great! The palette dialog is definitely a must!
Now I have to set the palette in version 1.0.10 and manulaly copy the [colors] section into FP.INI file of 2.2.2 version.

About CHM support: will be possible to load any (not FPC related) CHM file? Is possible to say that DOS world has a first CHM reader?

---
DOS-u-akbar!

> > I'm currently playing with the IDE, but most of my plans there are
> non-dos
> > related. (mostly .chm support and getting proper palette dialog
> support,
> > backport that from C++ version).
> >
> > But the IDE is more of a hobby, some other FPC tasks have priority.
>
> Great! The palette dialog is definitely a must!
> Now I have to set the palette in version 1.0.10 and manulaly copy the
> [colors] section into FP.INI file of 2.2.2 version.

It's a wild idea, so don't hold your breath. Till now I only spent 20min looking at the C++ code and saying, well.. maybe.

> About CHM support: will be possible to load any (not FPC related) CHM
> file? Is possible to say that DOS world has a first CHM reader?

I doubt it is the first. It is not that hard. The problem is not chm but html.

A CHM is simply a compressed archive format with a few records in the header (name of the index etc). In it is HTML and one XML index.

The XML is pretty wellformed and trivial to parse, so not the problem

The HTML is the problem. Loading an arbitrary CHM is the same as making a perfect html browser. So the question is how far to go with that? There can even be activex in there.

So no, I don't plan to advocate this as a generic CHM reader. I'll fix it for FPC as far as I get done, and am willing to apply patches that I can oversee, and that don't hurt the real deal (parsing FPC helpfiles) too much. That's about it.

I hope to have a snapshot ready soon.