Words of Wisdom

1. Release early, release often! (Source: Eric S. Raymond, The Cathedral and the Bazaar, 1997)
2. Ask "why" until you have all the answers.
3. Program to the API. APIs are hard to change. design them properly upfront.
4. If you don't have the time do do it properly now, what makes you think you will have the time later to go back and fix it? (Source attributed to Basket Ball coach John Wooden)
5. Hope is not a Strategy (Source: Google SRE Books)
6. No surprises! (Source: Goldman Sachs)
7. The key to making programs fast is to make them do practically nothing. (Mike Haertel)
8. Markets can remain irrational longer than you can remain solvent
9. You can't manage time. You can manage how you allocate it.
10. Frustration is your inability to make change happen (Gitte Klitgaard)
11. Everybody dies but not everybody lives.
12. Fail to plan - plan to fail
13. All models are wrong, but some are useful (Source: George E.P. Box)
14. There are only 02 hard things in computer science: cache invalidation, naming things and off by 1 errors.

Run Cyrus IMAPD mailserver from a Docker container with the mailbox data on an external volume

The Objective:

Run an Imap server on the local Linux machine in a way that it is easy to move from one computer to the next.

The use case: I want to run the imap server on my desktop most of the time but when I go away I want to take it along with me on my laptop.

The first solution was to set up a headless virtual machine with VirtualBox and install the cyrus imap server into the virtual machine. This works and the .vdi disk images can be moved from one computer to the next. The downside is that the disk image is around 19GB in size which takes hours to copy. Also running the VM on the Laptop reduces memory and degrades performance.

The Docker solution:

Build a docker container from the latest OpenSuse image and install cyrus imapd into it. Since containers "forget" all changes when the are shut down we use a VOLUME to persist the database and mail data on the host filesystem. The host directory with the cyrus data can then by rsynced to the new machine and the container can be started there. The mail client finds the impad server on localhost:143.

The Dockerfile:

FROM opensuse:42.1

ENV mailboxuser richi
ENV mailboxpassword password

MAINTAINER Richard Eigenmann 

USER root

# add the packages needed for the cyrus server and some to work with the shell
RUN zypper --non-interactive in \
  cyrus-imapd \
  cyradm \
  cyrus-sasl-saslauthd \
  cyrus-sasl-digestmd5 \
  cyrus-sasl-crammd5 \
  sudo less \
  telnet;

# set up the saslauthd accounts (complication: the host name changes all the time!)
# -u cyrus ensures the account is set up for the hostname cyrus
# cyrus is the account we need to run the cyradm commands
RUN echo ${mailboxpassword} | saslpasswd2 -p -u cyrus -c ${mailboxuser}
RUN echo "password" | saslpasswd2 -p -u cyrus -c cyrus
RUN chgrp mail /etc/sasldb2
RUN chsh -s /bin/bash cyrus


# Set up the mailboxes by starting the cyrus imap daemon, calling up cyradm
# and running the create mailbox commands.

# Step 1: set up a sasl password valid under the build hostname (no -u param).
# Since sasl cares about the hostname the validation doesn't work on the above
# passwords with the -u cyrus hostname.

RUN echo "password" | saslpasswd2 -p -c cyrus

# Step 2: We can't use here-documents in docker so we create the instructions
# that cyradm needs to execute in a text file

RUN echo -e "createmailbox user.${mailboxuser}\ncreatemailbox user.${mailboxuser}.Archive\nexit" > /createmailbox.commands

# Step 3: Start the daemon and in the same build container run the cyradm command
# (note the ; \  at the end of the line!)

RUN /sbin/startproc -p /var/run/cyrus-master.pid /usr/lib/cyrus/bin/master -d; \
sudo -u cyrus -i cyradm --user cyrus -w password localhost < /createmailbox.commands; \
mv /createmailbox.commands /createmailbox.commands.completed;


# create a file startup.sh in the root directory
RUN echo -e "#!/bin/bash\n"\
"if [ -e /var/dostart.semaphore ]; then\n"\
"chown -R cyrus:mail /var/spool/imap /var/lib/imap\n"\
"/usr/lib/cyrus/bin/master -d\n"\
"sleep .6\n"\
"ps u --user cyrus\n"\
"fi"\
> /startup.sh; \
chmod +x /startup.sh


# start the cyrus server and a shell
CMD  /startup.sh; /bin/bash

Running the server:

Build the container:

docker build -t richi/cyrus-docker:latest .

Do these steps to set up the mail server and the host directory: Build the container:

# on the host server
mkdir /absolute/path/to/the/exported/directory/var
docker run -it --rm --hostname cyrus -v /absolute/path/to/the/exported/directory/var:/mnt richi/cyrus-docker:latest

# inside the container 
cp -r /var/* /mnt
touch /mnt/dostart.semaphore

All subsequent runs:

docker run -it --rm --hostname cyrus -p 143:143 -v /absolute/path/to/the/exported/directory/var:/var --log-driver=journald richi/cyrus-docker:latest

Testing:

telnet localhost 143

#should result in output like this:

Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE LOGINDISABLED AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR] cyrus Cyrus IMAP v2.4.18 server ready

Discussion:

Setting up the basic container and adding the cyrus software is straight forward.

Setting up the mailbox user account with the password and creating the mailbox structure is tricky: cyrus uses saslauthd to check the passwords of the users logging in. Saslauthd has some sort of anti-tamper mechanism that leverages the hostname in the validation. Since the Docker build process changes the hostname at every step this gets problematic. The saslpasswd2 -u cyrus statements set the passwords for the user account and the cyrus admin account for the hostname cyrus (the -u).

To set up a mailbox account cyrus requires the daemon to be running. The user cyrus then needs to run the cyradm command with the instructions to create the mailbox. Here documents don't seem to be supported inside Dockerfiles so we first create a script file "createmailbox.commands". We then use sudo to promote to the cyrus account and then pipe in the instructions from the script file.

This creates a Docker container that can start up and knows the user, his password and has the basic mailbox structure. You can point your mail client at this imap server and things will work fine until you restart the container. The container will forget all changes when it is shut down. Since cyrus impad stores all state in the /var directory a solution is to export the var directory to the host filesystem so that it can be easily transported to other computers as well as backed up. The -v parameter in the docker run command does just this.

The syntax of the -v parameter is the absolute (!) path of the directory on the left gets mounted to the directory on the right of the colon. Annoyingly, if you just use the -v bind-mount parameter the previous contents of the /var directory in the container are hidden and you just see the empty /var directory from the host filesystem. There doesn't appear to be a way to bind-mount the host directory so that all the obfuscated directories and files from the container "shine through" and all new writes go to the bind-mounted directory.

Therefore we must copy all the content in the container's /var to the host directory first. The way I suggest doing this is to start the container and bind-mound the host's directory to /mnt in the container. Then a cp -r can copy all content from /var to the new directory. After shutting down the container and starting it up with the directory mounted to /var we are back to the original view.

But not quite: The important directories for cyrus, /var/lib/imap and /var/spool/imap, used to be owned by cyrus:mail but are owned by root after the volume mount. Since the server feels it can't read the mailbox database if it is root owned we need to correct this before the startup. I have thus created a startup.sh script that fixed the ownership of the mounted host directory and then starts the daemon. To keep everything in one Dockerfile I create the startup stript with an echo statement right inside the Dockerfile.

To facilitate rsyncing from one host to the other I suggest chown -R user:users on the host directory. Docker runs as root and will create all new files as root owned files but can perfectly well read and write to user owned files. Userspace synchronisation tools will find it much easier to deal with user owned files, however.

Don't call it "new"! But calling it "old" is OK.

Have you ever seen a directory with files names like this?

Which is the current one and which are older working copies?

The super organised person would name the files like this:

For the rest, let me recommend calling the files "old" as you have no trouble picking out the latest version here:















Of course that means you need to save the changes in 2 steps. You will have to save the new version of the document under a temporary name close the document, go to the explorer and rename 2 files. The effort is worth it!

Best advice I ever got

The best advice I ever got came from Renée Watkins. She gave me hard time over some software I had written to book FX trades. She kept asking me for detail upon detail and I just didn't know all the answers. Eventually she recommended that I ask WHY?  It took this to heart and it has helped me no end! If you don't know why something is supposed to work this way or that then whatever you code will not fit the expectations of your user.

It also ties in with another favourite from work: "No surprises ". If you ask enough probing questions then you will understand the problem being solved and will avoid many unpleasant surprises.

Backups

OK, you say, I get it, we should back up our data! And then you make a half hearted attempt and move on. But deep down you know about MTBF, the Mean Time Between Failures. The one thing we can say for certain about mechanical systems (such as your Hard Disk) is: IT WILL FAIL. The MTBF might give you the confidence that "my hard disk is likely to go on for another 3 years" and I sure hope it does. And when it does fail, often its not completely dead and you can get much of your data off it...

So my suggestion is to keep your data fully replicated on multiple devices. In order to do this easily I find it best to have one single directory underneath which everything of importance goes. (Think about it: When your disk blows in 3 years your computer is old and you will replace it with the shiniest new one that your budget allows. It will have a new version of Windows on it with new versions of the applications you use (and icons all looking different and in unusual places.) You don't need a backup of the Operating System and the Programs; you just need a backup of your data. [Yes, a list of the programs you use will be helpful! Perhaps you should go off and create just such a list in Evernote right now?]

I suggest you have one directory on the root of the filesystem (say c:\) with the name of the person. Example:  c:\Tom

Then you need to consider what kind of data you have. Some of this will be insensitive such as eBooks, mp3s, movies whilst you may feel other data is somewhat more private in nature like your salary slips, tax filings, accounts or contracts. You can grant and revoke permissions at a directory level so I suggest you create the insensitive directories directly under the main directory and create a Private directory for the more sensitive stuff. I.e.:

c:\Tom\Mp3
c:\Tom\Pictures
c:\Tom\Movies
c:\Tom\ToDo
c:\Tom\Private
c:\Tom\Private\Taxes
c:\Tom\Private\Contracts
c:\Tom\Private\Contracts\HealthInsurance
c:\Tom\Private\Accounts

You need to decide where the pictures should go. You probably want to share them with friends and family so they would more likely go into the main directory than the "Private" directory. (If aunt Mathilda is sitting next to you do you really want to be clicking around in the "Private" directory?)

I find it very useful to have a "ToDo" folder. This is supposed to be empty but will take all temporary stuff that you haven't filed properly yet.

The goal is to have all your data somewhere under c:\Tom and nothing on your Desktop, nothing in c:\documents and settings\local user\My Pictures\ and other crazy locations. There is an added advantage to this because some programmers seem to think that they can freely create junk files in your "Documents and Settings" folder. You have no idea what these files are and don't know if you can delete them without breaking anything. By having your data in your own structure they can freely use those locations and you will just walk away from that pile of junk when you upgrade to your next computer.

Now you are ready to do something about your backups! In the simplest form you just copy the entire c:\Tom folder to an external Hard Disk. Buy a large one and call the copy something like "\Tom-Backup-2018-03-01" and the next one "\Tom-Backup-2018-04-01". This allows you to go back to an old backup if you discover a file was corrupted or you accidentally lost half the text of your thesis some time in March.

I own multiple computers and like to have the whole directory replicated to each machine (in the belief that not all disks will fail at the same time). The problem you get into is that between copies different files will be modified on each of the machines. You need clever software to figure out what files were modified on which machine so that the latest version can be copied over. My favourite software for this is Unison File Synchronizer. It works really fast on huge directories between two Linux machines and works well (but slower) when comparing two directories (one local, one remote) on Windows.

For backups I recommend Box Backup. This backup software looks for changes on the filesystem and encrypts the changes and uploads them to the backup server. By searching for the changes it doesn't have to upload all 100MB of the file, just the parts that actually changed. Because it stores the changes it can reconstruct a file from several changes back. Because it encrypts the data on the client the person running the server can't decrypt the data. It runs in the background and figures out what to do completely on it's own. The client comes for Linux and Windows whilst the server needs to run Linux. The downside is that it is difficult to set up (especially the bit with the cryptography keys). Also most home users are throttled on the uplink of their internet connection which makes backups very slow. At worst the Internet will seem slow because the page requests have to queue up behind large backup packets.

Update on 19 May 2013: http://freefilesync.sourceforge.net/ looks like an interesting alternative to Unison for directory synchronisation.

Getting Things Done (GTD)

It must have been around the year 2000 when the bank I work for paid for David Allen to come to Switzerland and do a day's training session with us. It changed my life!

I came back to my desk and looked at it differently: For each paper I asked myself what is this? Does it have a right to live on the surface of my desk? Will I ever look at this again? Will I ever be asked to produce this for someone? By the end of the day the surface was empty.

After going through the drawers of my desk and repeating the same process at home I attacked my inbox. "Inbox Zero" is a great concept. In order to track all the things that need doing I had to get organized. Back in 2000 I was using the fantastic Lotus Organizer. Sadly IBM wasted 2bn to buy out Lotus and failed to retain the Organizer development team (if they even were at Lotus any longer). The tool just aged without any enhancements. I eventually had to switch to the Outlook Tasks at work. I set up categories: "1 Must do" (with the number 1 so it sorts on top) "2 Should do", "3 Might do", "5 Waiting for" and "9 Done". I have realised that I get frustrated and feel overwhelmed when there are more than 3 items in the "1 Must do" category.

One cool thing about categories is that a task can be attached to more than one category. So I have set up people categories such as "5  @John" for the "waiting for" items that I need to discuss with John when I have a slot with him.

For my private tasks I use Remember The Milk. A great product, all in the cloud with an attractive Android companion app. Todoist also seems to be a great product.

Recently I have started to use Evernote. At first it seemed a bit strange but I love it now. There were lots of little files and information lying around in a "stuff" folder. I was never too happy with that. OK, it wasn't in paper form any more (I scan everything) so the weight, storage space and flammable hazard were no longer an issue but it just isn't easy to search. With Evernote you upload the images, pdf files quotations etc. and attach tags to each note. Evernote even parses the text in the pdfs which really helps with searching. My notes are building up nicely and I find it hugely helpful to access this from any computer. Often I find I remember there was that snippet of information I filed but it's at home on the desktop computer in the "stuff" folder so "I'll look it up and email it to you when I get home".  What's also really cool is the Android app integration with the camera. See something interesting in the newspaper, at a presentation, trade show etc.? Just snap it into Evernote and then file it or read up on it  later!

There is a really cool video series on how to use Evernote called The Secret Weapon.

But, does it work?
At work my colleagues certainly know about David Allen's theory (they even made an on-line learning module with him). Looking around me many make a valiant effort at having a clean desk. I am, however, the only one with a Zero inbox.

I find David Allen's 2 Minute Rule ("If it takes less than 2 Minutes to do then do it immediately and skip the ToDo list") helpful. I can't handle a large list of "Must Do" items. This has to be cleared down to less than 4 items.

Recently the project I'm working on got really busy and I just could not keep up with the speed at which the mails came flying in. I had to set up a "ReadLater" folder and could only really achieve "Inbox Zero" by filtering the things that deserved attention into the ReadLater folder and getting rid of the rest right there. I then had to close down Outlook and do some real work every now and then before opening it up, sorting the next 58 unread emails, picking a next topic and shutting Outlook down to deal with that.

For me I think it generally works and there is no way back.

Amazing stuff made from paper: Peter Callesen

Check out the stuff Peter makes from paper:

https://www.petercallesen.com/works