DOS ain't dead - Duplicit

Laaca

Czech republic,
11.10.2008, 23:25

Duplicit (Announce)

Hello all!
I wrote a small tool for finding a files with duplicated data on the disk.
download here: http://www.laaca.borec.cz/soubory/duplicit.zip

---
DOS-u-akbar!

DOS386

16.10.2008, 00:59

@ Laaca

BOOOOH - much more duplicates that I could imagine to have

Post reply

> I wrote a small tool for finding a files with duplicated data

:-)

Good things:

- It indeed finds many duplicates if it starts successfully

Less good:

- Crash on startup ... maybe a FP 2.2.2 bug ? I don't see anything evil or suspicious in the DUPLICIT source :confused:

[image]

Also 216 besides 5 is "available" :-(

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

marcov

16.10.2008, 10:48

@ DOS386

BOOOOH - much more duplicates that I could imagine to have

Post reply

> Also 216 besides 5 is "available" :-(

5 File access denied
Permission to access the file is denied. This error might
be caused by one of several reasons:

* Trying to open for writing a file which is read-only, or which is actually a directory.
* File is currently locked or used by another process.
* Trying to create a new file, or directory while a file or directory of the same name already exists.
* Trying to read from a file which was opened in write-only mode.
* Trying to write from a file which was opened in read-only mode.
* Trying to remove a directory or file while it is not possible.
* No permission to access the file or directory.

Read only attributes ?

Laaca

Czech republic,
16.10.2008, 23:15

@ marcov

BOOOOH - much more duplicates that I could imagine to have

Post reply

This error is caused by read-only attribute of DUPLICIT.EXE

I forgot that in default if you open the binary file in pascal it requieres to have the read/write attribute. (variable FileMode is by default set to 2)

To be able to read the read-only files you have to set the FileMode to 0.
I forgot it because in bigger projects I always use my unit Lacrt, which does i automaticaly but here this unit isn't used.

To be short - here is the fixed 1.1 version available:
http://www.laaca.borec.cz/soubory/duplicit.zip

---
DOS-u-akbar!

DOS386

18.10.2008, 02:47

@ Laaca

DUPLICIT overlay: one bug fixed, one left

Post reply

marcov and Laaca wrote:

...

> This error is caused by read-only attribute of DUPLICIT.EXE

Thanks. :-) Confirming.

> To be short - here is the fixed 1.1 version

Funny ... you not only fixed the "R-BUG" , but also reduced bloat from 78 KiB to 64 KiB. Bloat is possible to reduce with FP :-) Confirming, R does no longer crash. :-) Now I also found the "bug" : it reads itself for the purpose of loading text from overlay. :clap: But there is one more bug left in same thing: if I un-UPX , it crashes again, "216" because UPX deletes the overlay :-(

2 more minor things:

#34 1) c:\windaube\ndislog.txt 2) c:\src386\cccore\iexpr.c 3) c:\src386\cccore\compiler.h 4) c:\cwsdpmi.swp

more informative would be:

#34 filesize=0

add sizes of the bunches :-)

And a more verbose summary at the end, like:

Found 173 duplicate groups, total 1'238 redundant files, totally causing 234'567'222 bytes wasted.

For every bunch count (amount-1), so if there are 5 files with same MD5, count 4 as redundant, and take size*4 for waste sum.

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

Laaca

Czech republic,
18.10.2008, 12:07

@ DOS386

DUPLICIT overlay: one bug fixed, one left

Post reply

Good idea, I'll add a summary at the end and will show the filesizes.

But I refuse call the bug that you can't pack and unpack duplicit.exe
It is not bug, it is a feature. I could, of course place the default messages into separate file like duplicit.dat but I want to be it usable only with the single exe file.

---
DOS-u-akbar!

Rugxulo

Usono,
18.10.2008, 21:43

@ Laaca

DUPLICIT overlay: one bug fixed, one left

Post reply

> Good idea, I'll add a summary at the end and will show the filesizes.
>
> But I refuse call the bug that you can't pack and unpack duplicit.exe
> It is not bug, it is a feature. I could, of course place the default
> messages into separate file like duplicit.dat but I want to be it usable
> only with the single exe file.

Just for comparison, here's some similar tools:

fdaz103.zip, A command line program which allows you to easily search for duplicates across your zip files

fdf105.zip, Fast Duplicate Finder. fdf is a command line program which finds duplicate files on your file system. fdf works by comparing the contents of files, instead of their file names. fdf is fast, supports sets, masks and allows you to remove the duplicates as well, GNU license

DOS386

19.10.2008, 01:18

@ Laaca

DUPLICIT overlay: one bug fixed, one left

Post reply

> Good idea, I'll add a summary at the end and will show the filesizes.

:-)

> But I refuse call the bug that you can't pack and unpack duplicit.exe
> It is not bug, it is a feature. I could, of course place the default
> messages into separate file like duplicit.dat but I want to be it usable
> only with the single exe file.

Missed the point. The crash after unpacking is of course a bug. Nevertheless I fully agree that a separate DUPLICIT.DAT file would be bad. The bug is not about placing the text into overlay, but the fact that it crashes if this text is not found. So the proposed solution would be:

- Keep the text in the overlay
- When searching for the text, don't rely on a plain text signature - this would cause a "false positive" finding the text "too early" if not UPX'ed, change at least one byte, for example use "MY%%OVERLAY" as signature, but place "MY()OVERLAY" into the Pascal source and hack "()" into "%%" before comparison
- If text not found, abort with a message "EXE is corrupt"

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

marcov

21.10.2008, 11:17

@ DOS386

DUPLICIT overlay: one bug fixed, one left

Post reply

> - Keep the text in the overlay
> - When searching for the text, don't rely on a plain text signature - this
> would cause a "false positive" finding the text "too early" if not UPX'ed,
> change at least one byte, for example use "MY%%OVERLAY" as signature, but
> place "MY()OVERLAY" into the Pascal source and hack "()" into "%%" before
> comparison
> - If text not found, abort with a message "EXE is corrupt"

- Wait for resourcehandling in the next major FPC version :-)

DOS386

22.10.2008, 15:27

@ marcov

DUPLICIT overlay: one bug fixed, one left

Post reply

> Wait for resourcehandling in the next major FPC version

COOL. IIRC it never worked well with Boreland ... this doesn't mean / shouldn't mean too much for FP, however.

The issue of DUPLICIT is trivial: just make sure not to crash if the text isn't found, and avoid exact plain text as search template.

---
This is a LOGITECH mouse driver, but some software expect here
the following string:*** This is Copyright 1983 Microsoft ***

Berlin, Germany,
20.10.2008, 14:45

@ Laaca

BOOOOH - much more duplicates that I could imagine to have

Post reply

> To be short - here is the fixed 1.1 version available:
> http://www.laaca.borec.cz/soubory/duplicit.zip

APPINFO\COMPUTE.LSM still has the wrong file name. ;-)

---
Forum admin