version 0.3.15-20061003-231645

download (212859 bytes)

This version now has two new preliminary scripts:

1. There now is a restore script bin/md5restore.sh

2. To setup networking there now is bin/sc-setup.sh

Little bit else changed. The scripts are not completely ready yet!

The restore is not capable to restore the metadata of a file. And this script is not thought to restore directories. Also be aware of the fact that sparse files are not yet backed up.

You need scylla-charybdis compiled to use sc-setup.sh

version 0.3.14-20050306-002847

download (132389 bytes)

Some medium restructuring in some central routines done. Multiple file store added (my backup archive became full).

This new feature works, but is nearly untested (as always).

Additional readonly directories named outN where N is a number starting from 0 are searched for existing backed up files, too, like the out directory. This way you can (manually) move old data from out/ into another directory to extend the harddrive space for backup.

Files which are considered new (active data) are copied back into the main file store.

version 0.3.13-20050220-164446

download (124530 bytes)

Wildcard ignores added. As always, this new feature is not much tested.

Ignores, which are listed in a file, now can start with a ? (the ? is skipped) which enables wildcard matching. Wildcards are:

• Allquantor (*), Existquantor (?) and variants ([...]):
• First character ^ inverts content
• First matching character can be anything, so []] matches ] and [[] matches [
• a-b matches a to b including a and b (a<=b)
• b-a matches a to b excluding a and b (a<b)
• Example: []-]-] matches ] or - (this is a-b and -)

These new type of ignores also solves a problem with "normal" ignores (which are much faster) which are processed at startup time. The "normal" ignores sometimes produce "amazing" ignores on frequently changing files, as inode numbers are matched, so the ignore "hits" a wrong file (the file the inode became at the time the backup process reaches the file).

version 0.3.12-20041005-050410

download (99040 bytes)

"nice", security lack fixed, new sparse files handling, bin/compare.sh

md5backup now automatically nices itself and uses file flushs. Also the backed up files are no more readable by others. It ignores sparse files and the source has been reorganized internally, some routines have moved into tinolib.

There is bin/compare.sh which can be used as a template for a restore script! Just copy the script and replace the MODE=compare line by MODE=restore and be sure to have understood what you do (else you will miss the second safety belt). Also bin/compare.sh can check if the backup really worked ;)

Sparse files are skipped now if they are too big (over 1 MB) and are too sparse (75%). The problem with those is, they have too few data in it. The drawback is, that they are no more backed up until I have added some more efficient sparse file support. Think of following: Create a 200 TB file on a 64 bit filesystem. Add one block of data somewhere in it. Now do the backup. If you are able to process 1 GB/s (which is extremely fast) it still takes over 1 day just to hunt for this silly block of data.

The security thingie is, that the files in the out/ directory were globally readable. That really makes no sense but I am usually alone on my machines, so this did not harm me. Be sure to do
chmod -R o-rwx /backup/md5backup

The nice is a step in my continuing effort to make md5backup less invasive for the system. A backup system shall run in background and shall not use up a lot resources when it runs - md5backup does not reach this goal by far.

The nice seems to help when the filesystem is mounted without the option noatime and the harddrive is somewhat slow (as harddrives are, YKWIM). The frequent directory inode flushes of the backup process can hinder other processes from IO. Without the nice, md5backup gets a too high scheduling priority as it always runs as root.

What I would like is to (Posixly correct) scan the directory tree without 'accessing' it and to only use "background IO" (this is when the harddrive else is idle). Did not find a method for this yet, AFAICS (if it is not already present) there should be a process capability to do so, so the process can request from the kernel if it is allowed to scan the directories without leaving a trace (which the kernel grants or not) and to become a "nulltask" for a resource, this is, it only runs if the resource is not used by other processes.

(This text should go into a Wiki, but currently I do not have one.)

version 0.3.11-20040930-013306

download (81692 bytes)

Bugfix for sc-loop.sh: It simply did not work, oops ;)

sc-backup.sh is broken anyways. Following three scripts should run independently from each other:

• The backup process backing up MySQL (sc-mysql.sh)
• The backup process backing up files (dobackup.sh)
• The network process, transporting files (sc-move.sh)

However sc-backup.sh (and therefor sc-loop.sh) calls them one after another. This way a network starvation slows down the backup cycle extremely. Bad design as it is, keep that in mind. I now "improved" sc-backup.sh a little bit such that the loop does not completely stop when the network is down (but it can take ages), so the left over data is hopefully transferred at the next cycle. Leave improvemnts of sc-backup.sh and/or sc-loop.sh for the future.

However you can always invent your own scripts or run the three scripts noted above from cron, of course.

License and Disclaimer

All you can see here is free software according to the GNU GPL.
Copyright (C)2000-2011 by Valentin Hilbig
Note that the software comes with absolutely no warranty of any kind.
You use the software at your own risk.
Valentin Hilbig cannot be hold responsible for any unintended damage,
lost data or malfunction of the software you can find here.