Moving to GitHub, slowly

The software on this pages will slowly be moved to GitHub https://github.com/hilbix/. The CVS repository will be migrated to GIT as well, so the history will be preserved, a bit. See FAQ.

Scylla and Charybdis, processes for data backup, transfer and transformation

Welcome. This page describes in short form what you find here.

The software I write is mainly about data backup, transfer, transformation and associated tasks like system adminstration.

Further reading

• Know How Programming where I save some often used code snippets

Latest update

• 2008-12-27 Added signature files for many (but not all) downloads

Quick overview

The main page you are probably interested in are the "Tools (new)". This page lists the tools from which might interest you.

These tools are packaged with "make dist" from my tinolib. Then they are automagically deployed here.

Scylla + Charybdis

There is the old homepage of Scylla and Charybdis "Scylla+Charybdis old home" which is a real chaos. Perhaps you better skip it and just proceed to the downloads. However this page is where everything started and I never came around to fix this page.

Additional tools

And there are some elder tools which are listed under an old tools page "Tools (old)". This page was manually maintained and thus is awful, clumsy and painful. However I still not came around to move these to the new format.

Please note that there are interesting things in it like xml2sql-v which is a tool to insert the contents of any XML file into a SQL database for postprocessing the XML structure with SQL. I use this frequently to post-process XML files to fetch the contents into the database. So this is a common adaptor for tools which are able to write XML to import the contents into a database. You then can access the xml structure through select statements (well, often you need some pivot transformations as xml2sql-v cannot interpret the contents, only the XML structure, but a good ETL tool shall provide this to you). xml2sql-v was not designed for the EAI case where single XML requests must be handled efficiently (your application must handle this) but for the data load case, where you have gigantic XML files, this is a tool of relief.

Central theme

All my tools here have one thing in common and that is data processing. It's not all too uncommon for things in Internet to be about that. Some of the tools are only associated with data processing, as they are related to system maintainance, which you need to do the data processing.

The common problems with data processing are:

• You have data on one location and want it to copy it somewhere else.
• You have data in one form and want to have it in another form.
• You have an enourmous amount of data and want to process this somehow.
• Things are failing unpredictably.

The main obstacles on the way to gain the results are incompatibilities and interrupts of services and plain bugs. So the processes shall be defined as follows:

• They shall be restartable, so they shall automatically continue from where they left off.
• They shall never give unexpected results, so if they signal success there must not be any error (with a the highest degree of probability you can have in this universe).
• They shall behave predictably, so even in the worst case you can think of they shall not do any additional harm.
• They shall be graceful in case some unpredictable happens.

Example for md5backup:

• It writes the data to the local drive. So even if the network goes down the backup is done. It is not saved, though, but the restore point is present.
• The files are written under their MD5 sum. So even if md5backup ceases to exist, you still can easily tell if the contents of the files are unharmed.
• charybdis is used for networking of md5backup (in future md5backup shall get it's own networking, but not now). If it's interrupted in the middle it restarts the transfer from where it left off.
• scylla (the opposit of charybdis) is not able to delete files. So even if you happen to stumble over the passwords you cannot harm the remote data which already is safe. All you can do is create another version which can easily be reverted to the correct files.

Example of ptybuffer:

• Often you want to automatically background tasks. Sometimes they need a tty!
• screen is not entirely helpful in this respect, as it's more a tool for interactive use.
• expect is not entirely helpful, as this does not allow you to take over manual control when needed.
• ptybuffer is just a very simple tool. It does not try to do anything clever. It just starts the command on a TTY and keeps a history of the output. You can then connect to prybuffer and see this history. And you can send input to the TTY, of course.

Development grade code

All tools presented here have development grade code. This means, they are in constant development. And the releases are early, that is, I do release them here before I try them myself thouroughly. (Normally I do a release and install it somewhere. So it can take some days until I notice flaws.)

So you must expect bugs if I made a mistake. That's why there are around all those old versions. If you have trouble you can revert to them.

Also I do not provide binaries. I do not have the time to do so. Currently there is nobody else supporting me with providing binaries for all platforms. So you must compile it yourself. This can trigger problems as I don't try the compile on different platforms. I just compile it where I need it. So I will fix things as soon as I see prooblems, but this might take it's time.

Platforms I want to support are:

• All variants of Linux, notably the newer ones
• CygWin, except for things which cannot be done on this platform
• Partly any other flavour of Unix, this includes MacOS-X
• And even other OSses like Windows. There will be some Windows only tools in future, too!

However for today this only means if you have problems with it, please try to patch it yourself and perhaps send the patches to me. You don't need to explain things a lot. If you are short of time just do

	patch=patch.tgz; tar cfz - . | uuencode "$patch" | mail -s "$patch" webmaster@scylla-charybdis.com

and let me sort things out (but don't expect a quick answer).

Notes

This all here shall sometime combine with many of my other pages into a central CMS. Currently (June 2006) the layout of this pages are the most advanced design (table-less design, fully HTML compliant with no Tidy warnings, with Accessibility and Breadcrumbs). They are lacking central CSS support, colors and further layout, of course.

All this has to come with a CMS, where the task of layouting is no more this cumbersome like today, where central formats can be applied to pages more easily. However no CMS out there fulfills my demands. The only one which comes near my basic needs is Bamboo, this is:

• No database. Period. So if the database is broken, the CMS still works, only additional functions like Search may be affected.
• No database. Period. So login and editing must not need a database.
• No database. Period. But pages must keep their editing history like in WiKis. So CVS or similar (no SVN, as this is web based itself) shall be used.
• No database. Period. Did I mention that a CMS must not need a database?
• And it shall support page transitioning. That is: Tar old pages. Untar them in some directory. Smile as you now can edit the pages (with history) and fix the most broken things.
• Multi Domain support. That is, when moving pages into the CMS old links must no break if you pointer the domain to the CMS. It must be able to serve different content of the same page under different domains without help of the web server virtual hosting (it shall support this, too, but it's too clumsy to need it always). So starting a new domain is just pointer it to the CMS and here you go. (Note that my webservers do automatical virtual hosting, that is "mkdir domain" creates the web presence. This shall be supported by the CMS.)
• Editing and Viewing are separate. That is: http://domain/ does the view, https://domain/ does the edit. But not forced to be this way, so you can do inline-editing and you can do it through a completely different domain (in case the pages need https themself).
• That are only the basic major needs. There are a lot of details I did not mention.

Well, it takes time to create such a thing myself. You will find it here as soon as I start working on it. Project name is tinocms, of course.