Moving to GitHub, slowly

The software on this pages will slowly be moved to GitHub https://github.com/hilbix/. The CVS repository will be migrated to GIT as well, so the history will be preserved, a bit. See FAQ.

Scylla and Charybdis, md5chk - Tools

The tools are developed under Linux with ESR's paradigm release early, release often in mind.
So you can consider this beta software, or alpha, or pre-alpha, or even worse ;)

Have a look in the download directory for all downloads.
As always here, all you get is the source. No binaries here.

md5chk 0.7.0-20111017-143949

Create md5sums for easy shell usage

⇒ ⇒ ⇒ The development shifted to another location. All future versions will be published at https://github.com/hilbix/md5chk ⇐ ⇐ ⇐

Sometimes you want to pipe a list of files to md5sum. This then looks like

find . -type f -print0 |
xargs --null md5sum -b -- |
while read a b
do ...

The problem is that the $b then starts with a * and you don't know what happens if the filename contains nonstarndard characters like LF. Some md5sum dialects just create garbage, others escape the MD5 sum.

md5chk doesn't do this and was designed from ground up for bash usage. It guarantees that the md5 sum always is "clean" and the filename always is properly escaped for shell reads. Note that there is no way for shell reads to express the newline character nor spaces at the end of an argument. So an unescape via ANSI strings must be done, sorry.

find . -type f -print0 |
md5chk -n |
while read -r md5 name
do
eval name="\$'$name'"
...

Note that the "eval" line does the escape entirely runs in the shell and thus is very fast.

Another common usage is "create md5 string from a file":

md5="`md5chk -iq "$name"`"; echo $?

Now you can create an md5sum from argument directly, too:

md5="`md5chk -dq "$string"`"

History:

version 0.7.0-20111017-143949

download (260655 bytes) sig

Bugfix release and options -k added

Now large files (more than 2 GB) can be processed, one of my machines exposed that bug to me.

Checksumming a large file with options -l and -m I observed the problem, that long parts of NUL blocks always have an equal sum. So the interleaving still does not protect against wrong ordering.

The complete sum will detect this, however this leaves a big puzzle on how to assemble all the pieces such that they are ok again. This now is solved with the new option -k.

By mixing in the block number into the MD5 hash, the hashes become unique, such that - with interleaving - you can sort all block such that they fit where they belong. (It's still a nontrivial task, but can be done.)

Option -k is followed by Option -p.

version 0.6.0-20111015-091410

download (260108 bytes) sig

Options -c, -l and -m added, install path now /usr/local/bin/

If you have a very big fiile the MD5 sum can be broken up into parts, say each 10K one MD5 sum is taken. If set to 0 means all the full file.

Output is then like FIRST+SECOND+LAST=COMPLETE

With option -l the -m is overlapping, that is each output MD5 covers two blocks instead of one.

Output then is like FIRST-FIRSTSECOND-SECONDLAST+LAST=COMPLETE

Also new is option -c, with this you can do an MD5 of data which goes throgh a pipe.

version 0.5.0-20081102-031402

download (227604 bytes) sig

Added option -p (and --shit)

Option -p initalizes the MD5 algorithm with some arbitrary string. This is to modify the md5 algorithm, to change calculated values. For example 'md5chk -p a -d b c' gives the same as 'md5chk -d ab ac', but the prefix is not shown in the output.

The new integration option --shit is prepared, but not yet operational. (Sorry for the name, but I needed a silly option name which will not interfere with another option which already is in use.) Before you ask: SHIT will provide a built in co-processing method like shell pipes. However it will skip many fork() calls which are very slow platforms like CygWin. It also will work over network sockets.

version 0.4.1-20080528-015214

download (217644 bytes) sig

Bugfix, DEL now is an escaped character, too.

Options in help output sorted, too

version 0.4.0-20080207-025526

(210153 bytes archive)

Minor bug removed and option -z added

char can be unsigned on some platforms. However I doubt this code will it ever make to such platforms ever ;)

Option z is to write NUL terminated lines (companion to reading NUL terminated lines).

[view more history]

[view complete history]

License and Disclaimer

All you can see here is free software according to the GNU GPL.
Copyright (C)2000-2011 by Valentin Hilbig
Note that the software comes with absolutely no warranty of any kind.
You use the software at your own risk.
Valentin Hilbig cannot be hold responsible for any unintended damage,
lost data or malfunction of the software you can find here.

Last modified: 2011-09-12 by Valentin Hilbig [ Imprint / Impressum ]