Shifting gears

After almost 2 years with the OpenQuake project I will be joining Rackspace as a technical cloud advocate on 01-Nov-2012.

This is novel and exciting in many ways as I will have the opportunity to pursue long standing interests and passions (cloud computing, scalable and robust IT architectures, open source, strategic thinking, reaching out to technical audiences etc.) as part of my *day* job.

I am looking forward to working with the good folks at Rackspace, the cloud community at large and anybody interested in putting cloud technology to good use!

See you around!

EuroPython talk info

The slides from the EuroPython talk (python & amqp) I held this morning are here. I’ll post a link to the video when it becomes available.

There are two things I wanted to mention but did not get to:

1. python-celery: if you are looking to partition and distribute computations do take a look at it. We are using it in the OpenQuake project and are very happy with it.
2. RabbitMQ in Action: in case you are using RabbitMQ or plan to do so get this book. I started reading it recently and derived a lot of value from it.

How to dual-boot a stable/experimental system with minimal breakage

For a couple of years now I have been using a scheme that allows me to dual-boot a stable system (for work) and an experimental system (for fun) with minimal breakage.
Recent reports of people who upgraded their linux machines and ended up with a broken system prompted me to share it.

The idea is to divide your hard disk into at least 7 partitions

    root/usr system 1            12GB
    root/usr system 2            12GB
    /var partition system 1       6GB
    /var partition system 2       6GB
    shared /tmp partition         4GB
    shared swap partition         2GB
    shared home partition       100GB

Just in case you are wondering about the small partition sizes: I am using a 160 GB SSD. It was the best hardware investment in a long time and really makes a difference.
If you are using e.g. a 320/500GB hard disk feel free to double the partition sizes (and/or triple the size of the home partition).

When installing a new linux now only two partitions dedicated to that particular installation are needed:

• a root/usr partition
• a /var partition

All the others (/tmp, swap, and /home) are shared. This works particularly well when the two installed systems are reasonably similar (e.g. Ubuntu 10.10 and 11.04). What you can do with the set-up described above is a full/proper installation of the desired system as opposed to an upgrade.

Please note that backing up data you cannot afford to lose is a standard procedure before you tinker with your system (e.g. prior to OS installations and/or upgrades).

Sometimes the experimental system is so unstable that I use another technique: a chroot/schroot combination.

There was e.g. a period during which an installed Ubuntu 11.04 was “unusable” (for me) but I needed to run it for a number of reasons.
I resorted to running Ubuntu 10.10 as my main work system and having an 11.04 chroot. Entering the latter via the schroot utility made for a fairly seamless experience.

I hope this helps

SSDs are the way to go!

I bought an intel X25-M SSD last week and it does make a *big* difference! It is faster, develops less noise and heat and the battery lasts longer.

I am using it with a lenovo thinkpad t410 laptop running Ubuntu 10.10 and it’s just great!

For what it’s worth I am running a pretty recent kernel in conjunction with the Ubuntu maverick userland. Not sure how well the normal 2.6.35 kernel supports SSDs.

Anyway, SSDs are the way to go

What is the best way to reset a file in a git topic branch?

Sometimes, when reviewing topic branches, I like to reset a file (to whatever it was in the master branch) and play around with it.

I figured out how to do that (see below) but it’s a bit clunky. Please take a look and comment if you know of a better way.

Here goes the example: first a repository is initialised and a file is added to it.

$ mkdir -p gitreset

$ cd gitreset/

$ git init .
Initialized empty Git repository in /home/muharem/tmp/gitreset/.git/

$ cat > a
This is file a, rev. 1
^D

$ git add a

$ git commit -a -m "initial commit"
[master (root-commit) 3e74747] initial commit
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 a

Next a topic branch is created and the file is modified in the former.

$ git checkout -b topic-branch master 
Switched to a new branch 'topic-branch'

$ cat > a
This is file a, rev. 2
^D

$ git diff
diff --git a/a b/a
index 595c3aa..7eb0dca 100644
--- a/a
+++ b/a
@@ -1 +1 @@
-This is file a, rev. 1
+This is file a, rev. 2

$ git commit -a -m "change to file a"
[topic-branch 300108e] change to file a
 1 files changed, 1 insertions(+), 1 deletions(-)

Now I would like to reset the file to whatever it was in the master branch.

$ git reset master a
Unstaged changes after reset:
M	a

$ cat a
This is file a, rev. 2

$ git diff
diff --git a/a b/a
index 595c3aa..7eb0dca 100644
--- a/a
+++ b/a
@@ -1 +1 @@
-This is file a, rev. 1
+This is file a, rev. 2

$ git diff --staged
diff --git a/a b/a
index 7eb0dca..595c3aa 100644
--- a/a
+++ b/a
@@ -1 +1 @@
-This is file a, rev. 2
+This is file a, rev. 1

It appears the file was reset but the revision of interest is in the staging area. To get that revision into the working tree I need to do additional work.

$ git diff --staged | patch -p1
patching file a

$ cat a
This is file a, rev. 1

Is there a way to have the changes resulting from git reset in the working tree straightaway?

OpenQuake is hiring in Zürich and in Pavia

The OpenQuake project is looking to hire two Python developers (one in Zürich/Switzerland, the other in Pavia/Italy).

We are a global and public project, do our development in accordance with agile principles and all our code is open.

Please see [1] and [2] below for more details on what we do.

In case you are interested, please send me an email (muharem SPAM-SUCKS linux.com) with your date of availability, your CV as well as some (python) code samples.

[1] http://www.globalquakemodel.org
[2] http://www.openquake.org

The world has changed

Whether for the better or the worse is left as an exercise to the reader but it definitely has changed.

The other day I was whining diffusely about the breakage of my new lenovo thinkpad t410. A day later I get a response. And a useful one at that!

Think about it! When did we ever have this before? Random people from a different continent taking note of one’s utterances and sharing their knowledge?

Just in case you missed it, the world has changed. And in this particular case I quite like it

Vim mappings for the win

I mostly work in source code hierarchies where for a given source file X.source the location of the file with the unit tests is tests/test_X.source and more often than not I need to do edit the unit tests after having opened the actual source file.

Being the geek that I am I *obviously* need to come up with some sort of optimisation or shortcut even if it takes 10x as long as stupidly typing in ":e tests/test_X.source" all the time

Thankfully, the solution in vim turns out to be quite straightforward. The following mapping (conveniently added to your $HOME/.vimrc file) will open the unit test file when you type %%

nnoremap %% :e =escape(expand("%:h")."/tests/test_".expand("%:t"), "")^M

Please note that the last bit ("^M") is just one character (the Enter key) that you can get by typing ^V followed by the Enter key.

I guess what I should really do is write a configurable vim plugin that opens arbitrary files/locations based on the current buffer/location. Oh well, so much to do and so little time

Smartphones are game changing technology!

I received my Motorola Milestone android smartphone last Friday and have now first hand experience of the utility of such a device. It is a

• navigation device
• great “communicator” (micro-blogging, email, irc, instant messaging etc.)
• e-reader (rss feeds, electronic books in a variety of formats)
• mobile media player
• voip/sip device

all neatly packaged in a gadget that fits in your pocket. Oh, and yes, you can also use it place or receive phone calls
So, just in case you don’t have a smartphone yet: get one! It’s a game changer.

Recent example: I took my mum to airport this morning (30 minute drive). After seeing her off I checked for new audio-casts (using Google listen) and saw one that interested me (Hadoop with Philip Zeyliger).
It was downloaded in no time and I could listen to it while driving back home which was a great use of my time.

Here’s some additional software I installed over the last couple of days:

• twidroid for twitter and identica
• NewsRob a Google reader client
• Google listen for audio-casts etc.
• Astro file manager not used that much yet
• Meebo IM for instant messaging

Any other cool stuff out there that I missed?

Handy bash function for merge conflicts

I have been merging packages from debian unstable to ubuntu karmic lately and every so often I need to look at a merge conflict.

Sometimes it’s not so obvious why a conflict occurred at all. Here’s an example conflict from the libsepol package:

  1 <<<<<<< libsepol-2.0.32-1ubuntu1 (ubuntu)
  3     /* Get the scope info for this boolean to see if this is the declaration,
  4      * if so set the state */
  5     scope = hashtab_search(state->cur->policy->p_bools_scope.table, id);
  6     if (!scope)
  7         return SEPOL_ERR;
  8     if (scope->scope == SCOPE_DECL)
  9         base_bool->state = booldatum->state;
 10 
 11 =======
 13     /* Get the scope info for this boolean to see if this is the declaration,
 14      * if so set the state */
 15     scope = hashtab_search(state->cur->policy->p_bools_scope.table, id);
 16     if (!scope)
 17         return SEPOL_ERR;
 18     if (scope->scope == SCOPE_DECL)
 19         base_bool->state = booldatum->state;
 20 
 21 >>>>>>> libsepol-2.0.36-1 (debian)

In order to avoid staring at the screen until my eyes start to bleed I came up with the following bash function:

  1 function dsec() {
  2     xclip -selection clipboard -o > /tmp/patch-seg-to-diff.txt
  3     rm -f /tmp/ubuntu-snippet.txt /tmp/debian-snippet.txt
  4     ed /tmp/patch-seg-to-diff.txt > /dev/null 2>&1 <<-EOUBUNTU
  5         1 d
  6         /^===/,$ d
  7         wq /tmp/ubuntu-snippet.txt
  8 EOUBUNTU
  9     ed /tmp/patch-seg-to-diff.txt > /dev/null 2>&1 <<-EODEBIAN
 10         1,/^===/d
 11         $ d
 12         wq /tmp/debian-snippet.txt
 13 EODEBIAN
 14     echo 'diff -Nru /tmp/ubuntu-snippet.txt /tmp/debian-snippet.txt | colordiff | less'
 15     diff -Nru /tmp/ubuntu-snippet.txt /tmp/debian-snippet.txt | colordiff | less
 16 }

I can now use my favourite editor to copy the conflicting code to the clipboard. When invoked on the command line the dsec function reads the clipboard content and chops it apart (using the ed utility) into a debian and an ubuntu snippet respectively. Subsequently it invokes the diff utility on the two snippets showing me how they differ.

 1 --- /tmp/ubuntu-snippet.txt 2009-06-26 12:42:04.000000000 +0200
 2 +++ /tmp/debian-snippet.txt 2009-06-26 12:42:04.000000000 +0200
 3 @@ -1,5 +1,5 @@
 4     /* Get the scope info for this boolean to see if this is the declaration,
 5 -    * if so set the state */
 6 +    * if so set the state */
 7     scope = hashtab_search(state->cur->policy->p_bools_scope.table, id);
 8     if (!scope)
 9         return SEPOL_ERR;

Hmm .. this looks like a white space issue. Since the dsec function printed the actual diff command to the terminal, I can copy and paste that to the command line and add the -b parameter so that the diff utility ignores changes in the amount of white space and ..

 1 $ diff -Nrub /tmp/ubuntu-snippet.txt /tmp/debian-snippet.txt
 2 $ echo $?
 3 0

.. voila! the diff output this time around is empty.

In order for this to work the utilities used must be installed, so e.g. on a ubuntu system run:

sudo apt-get install xclip colordiff

And .. don't forget to have fun