I have cleaned up the documentation for the scriptutil module which is available on the web now. If you happen to run ubuntu you can also install it as a package straight from my PPA.

Please have a look at this tutorial in case you’re interested in scriptutil usage examples.

Enjoy!

I have to admit that I didn’t fully understand the apfelmus example code. Nevertheless, I made an effort to address both of his criticisms:

1. The merge() function does not use an accumulator argument any more and is indeed much simpler now.
2. The recursive mergesort_ function now does not use the haskell list length operator any more. Instead, the length of the list to be sorted is passed down the recursive chain.

Please see the optimised mergesort implementation for details.

These improvements reduced the RAM utilisation and improved the run-time performance by another 20% respectively.

Nothing to scoff at, eh?

After my (naive) mergesort implementation from yesterday used around 730 MB of RAM to sort a (26 MB) file containing approx. 400,000 strings I consulted the good folks on the #haskell IRC channel.

Their advice was to use byte strings as opposed to normal strings since the former perform much better.

I tried that and observed that the RAM utilisation and run-time went down by approximately 85% !

The reduced RAM utilisation was attributed to the more efficient byte strings and the improved run-time performance to the reduced garbage collection overhead respectively.

The difference between the source files (mergesort.hs, Scaffolding.hs) is minimal and switching over to byte strings was facilitated by the fact that they expose the same interface as normal strings.

Nice!

I have been reading about haskell for a while now and felt that it’s time to “get my hands dirty” by (re-)implementing some of the well known algorithms in computer science. For my first exercise I chose merge sort.

The resulting code (sans command line parameter handling, auxiliary I/O functions etc.) is quite neat and was written in little time. Haskell is very expressive and its attractiveness immediately obvious when formulating pseudo-mathematical problem solutions.

  1 module Main(main) where
  2 
  3 import qualified IO
  4 import System(getArgs)
  5 import Monad(mapM_)
  6 import qualified Scaffolding
  7 
  8 mergesort l =
  9     if (length l) <= 1 then l – The list is already sorted.
 10     else – Split the list into two halves and sort these.
 11         merge (mergesort lpart) (mergesort rpart) []
 12         where llen_half = (length l) `div` 2
 13               (lpart, rpart) = splitAt llen_half l
 14 
 15 – Case #1: the right list is empty; just append the left list to the
 16 – accumulator. The latter is reversed because we were prepending
 17 – to it in case #3.
 18 merge lpart [] acc = (reverse acc) ++ lpart
 19 
 20 – Case #2: the left list is empty; just append the right list to the
 21 – (reversed) accumulator list.
 22 merge [] rpart acc = (reverse acc) ++ rpart
 23 
 24 – Case #3: neither of the left/right lists to be merged is empty;
 25 – prepend the lesser head element to the accumulator list.
 26 merge (l:ls) (r:rs) acc =
 27     if l <= r then merge ls (r:rs) (l:acc)
 28     else merge (l:ls) rs (r:acc)
 29 
 30 – The main method, handles command line arguments, input and output.
 31 main = do
 32     args <- getArgs
 33     fileh <- case head args of
 34                 – The text to be sorted is to be read from stdin.
 35                 “-” -> do return (Just IO.stdin)
 36                 – The text to be sorted is to be read from the file
 37                 – specified.
 38                 “-f” -> do Scaffolding.openFile (head (tail args))
 39                 – Just sort the command line parameters.
 40                 _ -> do let sorted_args = mergesort args
 41                         IO.putStrLn (“mergesort(” ++ show args ++
 42                                      “) = “ ++ show sorted_args)
 43                         return Nothing
 44     case fileh of
 45         – Read text from file handle, sort it and print to stdout.
 46         Just fileh -> do text_to_sort <- Scaffolding.readLines fileh []
 47                          mapM_ IO.putStrLn (mergesort text_to_sort)
 48         – We failed to open the file specified (if any).
 49         Nothing -> do return ()

I also liked the fact that the haskell standard library is rich and nicely documented.

Just to get an idea of how my naive mergesort implementation performs I created a file listing all the files on my Dell D630 laptop running Hardy Heron (please ignore the recursive nature of this problem ).

 1 u804: haskell $ ghc –version
 2 The Glorious Glasgow Haskell Compilation System, version 6.8.2
 3 u804: haskell $ ls -l all-files.txt
 4 -rw-r–r– 1 mhr mhr 26245382 2008-05-31 17:18 all-files.txt
 5 u804: haskell $ wc -l all-files.txt
 6 399754 all-files.txt
 7 u804: haskell $ uname -a
 8 Linux u804 2.6.24-18-generic #1 SMP Wed May 28 20:27:26 UTC 2008 i686 GNU/Linux

I then let loose the compiled haskell binary on that file (all-files.txt) and compared how it performed against /usr/bin/sort.

  1 u804: haskell $ time ./mergesort -f all-files.txt >/dev/null
  2 
  3 real    0m12.994s
  4 user    0m12.349s
  5 sys 0m0.632s
  6 u804: haskell $ time sort all-files.txt >/dev/null
  7 
  8 real    0m10.675s
  9 user    0m10.613s
 10 sys 0m0.052s
 11 u804: haskell $ time sort all-files.txt >/dev/null

The difference in performance was moderate (22%) given that I did not tune the haskell implementation in any way.

However, I gasped when I observed the RAM utilisation while the haskell program was running: it used 726 MB of RAM!

Compare this to 34 MB that were used by the standard sort program.

I guess it’s time to get acquainted with ghc -prof and friends

Last but not least I’d like to point to a great haskell resource, the Real World Haskell beta book.

Introduction

After a long break I picked up the Erlang book again and my appetite for writing some erlang code was soon kindled.

A small Python component I produced at work seemed like a good candidate for my (sequential) erlang exercises. It is a fairly simple component that removes user/password data embedded in URLs.

Just so you know where I am coming from:

• my main/favourite programming language is Python
• my exercises are mainly about sequential, non-distributed and non-telecoms-related problems whereas erlang’s main strength and appeal lies in the area of parallel/distributed telecoms/networking systems
• I have played with erlang a little bit before (ring benchmark, XML parsing) and liked it in general although IMHO it lacks severely when it comes to the availability and quality of standard library components.

Now that my particular set of preconceptions is clear and in the open, let’s look at the stuff below

File processing with erlang’s regexp module

The initial implementation of the URL filter in erlang used its regexp library.

  1 -module(regex).
  2 -export([main/1]).
  3 
  4 isalphanum(C) when C > 47, C < 58; C > 64, C < 91; C > 96, C < 123 -> true;
  5 isalphanum(_) -> false.
  6 
  7 %% Generate a temporary file name of length N
  8 genname(0, L) -> L;
  9 genname(N, L) ->
 10     R = random:uniform(123),
 11     case isalphanum(R) of
 12         true -> genname(N-1, [R|L]);
 13         false -> genname(N, L)
 14     end.
 15 
 16 %% Returns a randomly generated temporary file path where the basename is
 17 %% of length N
 18 mktemppath(Prefix, N) -> Prefix ++ “/” ++ genname(N, []).
 19 

Please note how I had to implement functionality absent from the standard library above.

 20 %% Removes passwords embedded in URLs from a log file.
 21 scrub_file(Tmpdir, F) ->
 22     %% make a temporary directory if it does not exist yet.
 23     case file:make_dir(Tmpdir) of
 24         ok -> ok;
 25         {error,eexist} -> ok;
 26         _ -> exit({error, failed_to_make_tmpdir})
 27     end,
 28 
 29     %% Move the original file out of the way.
 30     T = mktemppath(Tmpdir, 16),
 31     case file:rename(F, T) of
 32         ok -> ok;
 33         _ -> exit({error, failed_to_move_file})
 34     end,
 35 
 36     %% Now open it for reading.
 37     {_, In} = file:open([T], read),
 38     %% Open the original path for writing.
 39     {_, Out} = file:open([F], write),
 40 
 41     %% Call the function that will scrub the lines.
 42     scrub_lines(In, Out),
 43 
 44     %% Close the file handles and return the path to the original file.
 45     file:close(Out),
 46     file:close(In),
 47     T.
 48 

The code that scrubs the URLs is below, the scrub_lines() function is tail recursive.

 49 %% This is where the log file is actually read linewise and where
 50 %% the scrubbing function is invoked for lines that contain URLs.
 51 scrub_lines(In, Out) ->
 52     L = io:get_line(In, ”),
 53     case L of
 54         eof -> ok;
 55         _ ->
 56             %% Does the line contain URLs?
 57             case string:str(L, “://”) of
 58                 0 -> io:format(Out, “~s“, [L]);
 59                 _ ->
 60                     case regexp:gsub(L, “://[^:]+:[^@]+@”, “://”) of
 61                         {ok, S, _} -> io:format(Out, “~s“, [S]);
 62                         {R, S, _} -> io:format(“Failed: {~p,~p}”, [R,S])
 63                     end
 64             end,
 65             %% Continue with next line.
 66             scrub_lines(In, Out)
 67     end.
 68 
 69 %% Main function.
 70 main([A]) ->
 71     {A1,A2,A3} = now(),
 72     random:seed(A1, A2, A3),
 73 
 74     %% A single argument (the name of the file to be scrubbed) is expected.
 75     F = atom_to_list(A),
 76     T = scrub_file(“tmp”, F),
 77 
 78     %% The scrubbed file content will be written to a new file that’s
 79     %% in the place of the original file. Where was the latter moved to?
 80     io:format(“~s~n“, [T]),
 81 
 82     init:stop().

The cursory benchmarks performed (on log files of varying size) using the python and the erlang code confirmed other people’s experiences with erlang’s regex performance (but see also this interesting “rebuttal”).

The do-it-yourself construction

Curious to learn whether the performance can be improved by abstaining from regular expressions I came up with an alternative implementation that does not use regexp.

As you can see below the do-it-yourself construction is indeed performing slightly better at the expense of being very specialized and requiring 60% more code.

In conclusion

Every couple of months or so I develop a euphoria towards erlang which is consistently dampened by using the language to tackle problems for which the language admittedly was not designed in first place.

I guess most people start using a language for simple programming exercises first as opposed to building something like a Jabber/XMPP instant messaging server straightaway.

I hate to repeat myself but improving the standard library (by adding common functionality and making sure it performs decently) would do a lot to attract fresh talent to the erlang community and I hear that a certain rate of influx of “fresh blood” is a necessary prerequisite for success.

Ah, and no, you were not supposed to grok the sentence above unless you read it three times

I am using a MacBook Pro for roughly 10 months now and I am generally very happy with it. Nevertheless, I like experimenting and playing with other operating systems, primarily with various linux distributions and with members of the *BSD family.

Running Ubuntu or FreeBSD in a virtual machine (using VMWare’s fusion or Parallels desktop) is a good way to get a first impression of such a system but at some point you will want real hardware. I reached that point with Ubuntu when I took an interest in hard disk encryption (why you should encrypt your hard disk).

It just so happened that around that time Aldi (a German retailer) had a nice 12 inch subnotebook on offer. Aldi was targeting the female clientele (the notebook was even adorned with rhinestones etc.) which resulted in a 100 Euro price markup. The system was still a very good value for the money, however, and I even managed to get my hands on the black model

I booted the system up (it had Windows Vista Home Premium pre-installed) in order to take a backup of the system in case I had to restore the original state (prior to sending in it for service for example).

Vista was atrocious, totally crippling the poor thing. I mean, it was horrible, even the simplest interactions or commands took a ridiculous amount of time to complete.
Anyway, I managed to take a snapshot of the hard disk, wiped it clean and installed Ubuntu-7.10.

After running Vista on the box for a few hours, Ubuntu was a relief it felt very snappy and was generally performing very well. I could do all my development work on the notebook without any problems.

I thought that was very cool I finally had a second box, a “baby laptop”, I could use for my experiments. Having two laptops may sound a little bit excessive unless you tamper frequently with all sorts of operating systems. You can keep one machine in a “stable” condition and do all the proper (money generating) work on it while totally rebuilding the other.

I am a slightly paranoid person and like to have as much security on my machines as I can get. I hence wanted to install Ubuntu with full hard disk encryption but more on that topic in a forthcoming article..

Hello there,

for anybody owning a Mac and having an interest in file systems I just wanted to point to MacFUSE (a FUSE-Compliant File System Implementation Mechanism for Mac OS X) which is another brilliant piece of work done by Amit Singh (of “Mac OS X Internals” fame).

There is also a technical presentation given by Amit on the MacFUSE project (YouTube video).

Introduction

Despite being a great DBMS, PostgreSQL has a few wrinkles that can cause quite a bit of pain

One such wrinkle is the insertion of database rows with Bytea arrays.

If you’re not dealing with PostgreSQL or don’t need to wrangle with Bytea arrays feel free to skip this article. If you are, however, this article may save you a lot of time and frustration.

The code

Since the correct number of backslashes in the code below is important but unlikely to be displayed properly in your web browser please view it syntax highlighted here or in plain format here.

The code below consists of roughly two sections:

• the actual code of interest dealing with the backslash plague and the quoting orgy (prepareByteaString(), lines 49-82)
• a test harness (lines 84-11 merely facilitating the testing of the function of interest

The problem on hand is that the PostgreSQL Array Value Input syntax requires that the Bytea array literal be enclosed in single quotes. Psycopg2 however quotes individual byte strings using single quotes as well.

Please note: a byte string or array corresponds to one Bytea variable. A Bytea array is hence an array of byte arrays or strings.

The resulting Bytea array literals are a mess and cause syntax errors when used in INSERT statements etc. The approach chosen is to “re-quote” the Bytea literals returned by the Psycopg2 Binary() function from single to double quotes, For more detail please see the comments on lines 68-80.

  36 # created: Thu Oct 25 21:35:50 2007
  37 __version__ = “$Id:$”
  38 # $HeadURL $
  39 
  40 import psycopg2 as ps2
  41 
  42 class PGT(object):
  43     “”"Utility functions for using a PostgreSQL database with python“”"
  44 
  45     def __init__(self):
  46         “”"initialiser“”"
  47         # super(NEW_CLASS, self).__init__()
  48 
  49     @staticmethod
  50     def prepareByteaString(byteaSeq):
  51         “”"
  52         Given a sequence of byte arrays this function prepares a properly
  53         quoted string to be used for inserting database rows with Bytea
  54         arrays.
  55 
  56         Given e.g. a table like the following
  57             Create table baex(byteaa Bytea ARRAY[16]);
  58         the resulting string (’bas’) can be used as follows:
  59             cursor.execute(”INSERT INTO baex(byteaa) values(’{%s}’)” % bas))
  60 
  61         Parameters:
  62         - byteaSeq: a sequence of byte arrays each corresponding to a Bytea
  63                     value in the database
  64         Returns:
  65         string: containing all the byte arrays from the ‘byteaSeq’ properly
  66                 quoted for utilisation in an INSERT statement
  67         “”"
  68         # in a first step
  69         #   1. quote all the byte arrays (using the psycopg2 Binary()
  70         #      function)
  71         #   2. strip away the single quotes on the left and right side
  72         #   3. escape any double quote characters with a backslash
  73         # The last step is necessary because we will use the double quote as
  74         # the quote/delimiter for the byte arrays
  75         baSeq = [str(ps2.Binary(ba))[1:-1].replace(r’“‘, r’\”‘) for ba in byteaSeq]
  76         # join the prep’ed byte arrays into a single string
  77         bas = “\”%s\”” % ‘“,”‘.join(baSeq)
  78         # double the number of backslashes (needed because we’re inserting a
  79         # Bytea array as opposed to a single Bytea)
  80         bas = bas.replace(’\\‘, ‘\\\\‘)
  81         # done!
  82         return(bas)

The code below is only executed when you invoke the Python file directly but will not run if you import it.

  84 if __name__ == ‘__main__‘:
  85     ### TEST code ********************************************************
  86     import os, sys
  87     from random import random as rand
  88 
  89     # connect to test database
  90     db = ps2.connect(”dbname=’test’ user=’postgres’“)
  91     cursor = db.cursor()
  92     # create test table
  93     cursor.execute(’Create Table public.baex(id Serial, byteaa Bytea ARRAY[16])‘)

For test purposes I am connecting to my test database (line 90) and creating a test table (baex, line 93).

  95     sys.stdout.flush()
  96     print “\n******************** Bytea data generated: ********************“

Then I generate Bytea data for three database rows and insert it into the database (loop on lines 98-109).

  97     # generate 3 byte array sequences
  98     for rowId in range(1, 4):
  99         byteaSeq = []

Each row is populated with a Bytea array holding of up to seven elements. I am using the random() function to generate the bytes.

 100         # generate 1-7 random byte arrays
 101         for numOfSstrings in range(1, int(rand()*8)):
 102             bytea = ”.join([chr(int(rand()*256)) for x in range(int(rand()*5))])
 103             byteaSeq.append(bytea)
 104         print byteaSeq
 105         # get the INSERT string for the byte array sequence generated
 106         bas = PGT.prepareByteaString(byteaSeq)
 107         # insert the row into the table
 108         cursor.execute(”INSERT INTO public.baex(id, byteaa) VALUES(%s, ‘{%s}’)” \
 109                        % (rowId, bas))
 110     db.commit()
 111     sys.stdout.flush()

Once the data is inserted into the database I run psql to check whether everything worked properly..

 113     print “\n******************** Bytea data inserted: ********************”
 114     sys.stdout.flush()
 115     # show the data inserted
 116     os.system(”psql -d test -U postgres -c ‘SELECT * FROM public.baex’“)

.. and finally drop the test table.

 117     cursor.execute(’DROP TABLE public.baex‘)
 118     db.commit()

A few test runs

I have selected the two test runs below because they show some interesting (edge) cases.

The first one shows that single quotes are dealt with properly (last byte in last byte string of first row and first byte string of the third row)

******************** Bytea data generated: ********************
['R\xb7', '\xd7\xcd\xbb', 'u', "\xa3[\x97'"]
['\xfbT\x84g', '\xfa', '']
["'", '\x8d', '', '\xc5\n5', 'A', '\xa4P*']

******************** Bytea data inserted: ********************
 id |                    byteaa
—-+———————————————–
  1 | {”R\\267″,”\\327\\315\\273″,u,”\\243[\\227'"}
  2 | {"\\373T\\204g","\\372",""}
  3 | {',"\\215","","\\305\125",A,"\\244P*"}
(3 rows)

The second test run demonstrates that double quotes are handled correctly (fist byte of first byte string in first row)

******************** Bytea data generated: ********************
['"\xa5\x13', '\x9a`', '', '\xf4\x98']
[]
['u', '?7', '1\xff', '.\x16\xcf', '\xcbe}', '']

******************** Bytea data inserted: ********************
 id |                   byteaa
—-+——————————————–
  1 | {”\”\\245\23″,”\\232`”,”",”\\364\\230″}
  2 | {”"}
  3 | {u,?7,”1\\377″,”.\26\\317″,”\\313e}”,”"}
(3 rows)

Conclusion

Once you bite the bullet and invest the time to think about the problem, code the function and test it, it’s simple

Introduction

In the last few months I have taken some time to play with a number of dynamic languages. My experiments were mostly in the “web hacks” category e.g. fetching files from the web and extracting data of interest from these. For my most recent hack (get wordpress weblog statistics) I used Ruby.

The task at hand

The task at hand consists of fetching the weblog statistics for my wordpress weblog and displaying them in the terminal window.
This includes the handling of possible redirections to the wordpress.com login page, the parsing of the HTML file to be obtained and the extraction of the various weblog statistics.

The tools used

After briefly surveying the tools and libraries available in Ruby-land I settled for WWW::Mechanize a Ruby implementation of Perl’s venerable WWW-Mechanize CPAN module.

Under the hood WWW::Mechanize uses the Hpricot HTML parser.

The approach

The worpress.com weblog statistics pages have a URL with the following structure:

    http://#{user}.wordpress.com/wp-admin/index.php?page=stats

They contain the following statistics for today and yesterday respectively:

• Referrers: people clicked links from these pages to get to your weblog
• Top posts: these posts on your weblog got the most traffic
• Search engine terms: these are terms people used to find your weblog
• Clicks: your visitors clicked these links on your weblog

Each of the above are structured as follows:

<div class="statsdiv">
<h3><a href="7-day page URL">statistics type</a></h3>
<p>..explanatory text..</p>
<h4>Today</h4>
  <table class="statsDay">
    <tr><th>..</th><th class="views">..</th></tr>
    <tr>
      <td class="label">URL or term</td>
      <td class="views">number of views</td>
    </tr>
    ...
  </table>
<h4>Yesterday</h4>
  <table class="statsDay">
    <tr><th>..</th><th class="views">..</th></tr>
    <tr>
      <td class="label">URL or term</td>
      <td class="views">number of views</td>
    </tr>
    ...
  </table>
</div>

The Ruby code below first finds the <div class="statsdiv"> sub-trees and then extracts today’s data from them.

The code

  1 #!/usr/bin/env ruby
  2 
  3 require 'rubygems'
  4 require 'mechanize'
  5 
  6 HELP_STRING =<<EOS
  7 
  8 Tool for fetching wordpress.com weblog statistics. Usage:
  9 
 10     wls.rb [username] [pwd]
 11 
 12 where 'user' is your wordpress user name and 'pwd' is your
 13 password respectively.
 14 
 15 EOS
 16 
 17 if not ARGV.grep(/-h|–help/).empty?
 18     puts HELP_STRING
 19     exit(0)
 20 end
 21 
 22 # try to access the weblog statistics page
 23 user = 'muharem'
 24 password = nil  # set your password here if you dislike being prompted for it
 25 
 26 if ARGV[0]
 27     user = ARGV[0]
 28 end
 29 if ARGV[1]
 30     password = ARGV[1]
 31 end
 32 
 33 stats_url = "http://#{user}.wordpress.com/wp-admin/index.php?page=stats"
 34 
 35 # instantiate/initialise web agent ..
 36 agent = WWW::Mechanize.new
 37 agent.user_agent_alias = 'Mac Safari'
 38 # .. and get the weblog statistics page
 39 page = agent.get(stats_url)
 40 
 41 # did we get back the login form?
 42 if (page.title.strip.split[-1] == 'Login')
 43     # yes, fill it in and submit it
 44     loginf = page.form('loginform')
 45     loginf.log = user
 46     if not password
 47         print "Enter your wordpress.com password: "
 48         password = $stdin.gets.chomp
 49     end
 50     loginf.pwd = password
 51     agent.submit(loginf, loginf.buttons.first)
 52 end
 53 
 54 # now get the actual weblog statistics page
 55 page = agent.get_file(stats_url)
 56 # parse it!
 57 doc = Hpricot(page)
 58 
 59 # search for the div elements that contain the statistics data
 60 stats_divs = doc.search("//div[@class='statsdiv']")
 61 stats_divs.each do |div|
 62     heading = div.search("h3/a/text()")
 63     # we are only interested in the statistics for today
 64     day = div.search("h4/text()").first
 65     if (heading and day)
 66         heading = "==== #{heading} (#{day.inner_text.downcase}) ====".center(50)
 67         puts "\\n#{heading}\\n"
 68         # find the table with today's statistics data
 69         tab = div.search("table").first
 70         if tab
 71             # extract the statistics data from the <tr> elements
 72             tab.search("tr").each do |tr|
 73                 what = tr.search("td[@class='label']")
 74                 views = tr.search("td[@class='views']")
 75                 whats = what.inner_text.strip()
 76                 if not whats.empty?
 77                     views = views.inner_text.strip()
 78                     printf("%s — %5s\\n", whats.center(45), views)
 79                 end
 80             end
 81         end
 82     end
 83 end
 84 # grab the div with the general (weblog level) statistics data
 85 gbdiv = doc.search("//div[@id='generalblog']")
 86 # find the <p> element with the number of views today
 87 vtoday =  gbdiv.search("p").find { |p| p.inner_text.index('Views today') }
 88 if vtoday
 89     printf("\\n%s\\n\\n", "=> #{vtoday.inner_text.strip} <=".center(45))
 90 else
 91     puts "\\n\\n!! No weblog statistics data found."
 92     puts "   Did you enter a wrong user name and/or password?"
 93 end

Example output

  1            ==== Referrers (today) ====
  2    stumbleupon.com/refer.php?url=http%3A?     —    16
  3    stumbleupon.com/refer.php?url=http%3A?     —     3
  4    planeterlang.org/story.php?title=Erla?     —     2
  5    linuxquestions.org/questions/showthre?     —     2
  6        del.icio.us/jdkimball/stackless        —     1
  7    rodenas.org/blog/2007/08/27/erlang-ri?     —     1
  8    intertwingly.net/blog/2007/08/14/Long?     —     1
  9              ozone.wordpress.com              —     1
 10    programming.reddit.com/search?q=erlan?     —     1
 11 
 12            ==== Top Posts (today) ====
 13           Processing XML in Erlang            —    22
 14   Erlang vs. Stackless python: a first ben    —    18
 15   Python: file find, grep and in-line repl    —     4
 16   Python decorator mini-study (part 1 of 3    —     2
 17   Code refactoring with python's functoo      —     2
 18   Python: find files using Unix shell-styl    —     2
 19   Determine order of execution by (re-)seq    —     2
 20            A first look at Groovy             —     1
 21   Python decorator mini-study (part 2 of 3    —     1
 22    Turn on line numbers while searching in    —     1
 23 
 24       ==== Search Engine Terms (today) ====
 25               erlang benchmark                —     3
 26              stackless vs erlang              —     2
 27          python decorators argument           —     2
 28       source code of an execution path        —     2
 29           python functools partial            —     2
 30                  python grep                  —     2
 31          python string replace 2.4.4          —     1
 32               erlang vs C speed               —     1
 33     erlang command line arguments getopt      —     1
 34    Python + parsing command line arguments    —     1
 35 
 36              ==== Clicks (today) ====
 37         hpcwire.com/hpc/1295541.html          —     2
 38    pragmaticprogrammer.com/titles/jaerla?     —     1
 39    hrnjad.net/src/6/scriptutil.py.html#f?     —     1
 40 
 41             => Views today: 68 <=

Conclusion

I am a total Ruby beginner but have a lot of experience with Python and Perl. It took approximately 2 hours (and frequent look-ups in the pickaxe book) to write the tool above and I enjoyed it

Being the kind of person that stays away from all things over-hyped I ignored Ruby for the last two years or so but I have to say it’s a cool language after all.

Click here to download the code.

Introduction

This is my second stab at Erlang (see the ring benchmark article for my first take on it). This time around I wanted to get a sense of how well Erlang and its libraries support more mundane tasks like e.g. XML parsing.

I am not a big fan of XML but it is the lingua franca of the web and any language that aspires to become “mainstream” has to support it in an efficient manner.

In order to get a feeling for how well Erlang is doing in this respect I am going to repeat my recent XML processing experiments with Groovy but this time using Erlang.

Example

I’ll be doing some basic processing of RSS files. For a full example of what these look like see e.g. the MacBreak’s weekly RSS file. Here’s an excerpt (abridged for the sake of clarity):

  1 <?xml version=“1.0″ encoding=“utf-8″?>
  2 <rss version=“2.0″>
  3   <channel>
  4     <title>MacBreak Weekly</title>
  5     <item>
  6       <title>MacBreak Weekly 53: Bill In A Box</title>
  7       <link>http://www.podtrac.com/twit.cachefly.net/MBW-053.mp3</link>
  8       <pubDate>Wed, 15 Aug 2007 12:58:11 -0700</pubDate>
  9       <enclosure url=“http://www.podtrac.com/twit.cachefly.net/MBW-053.mp3“ />
 10     </item>
 11   </channel>
 12 </rss>

Each of the potentially many <item> tags keeps the data pertaining to a single audiocast episode. What we want to extract is:

• the audiocast episode title (<title> tag)
• the publication date (<pubDate> tag)
• the URL pointing to the MP3 file (<link> tag)

Depending on the publisher the format of the RSS file may vary slightly. The publication date is e.g. sometimes buried in a <dc:date> tag.
Likewise, the MP3 URL is sometimes not contained in a <link> tag but in the url attribute of the <enclosure> tag.

The RSS files I will be using for testing are as follows:

 1 bbox33:audiocasts $ pwd
 2 /Users/mhr/dl/audio/audiocasts
 3 bbox33:audiocasts $ find . -type f -name \*.rss
 4 ./metadata/Cc_zwei.rss
 5 ./metadata/Elrep.rss
 6 ./metadata/Security_now_.rss
 7 ./metadata/Technometria.rss
 8 ./metadata/This_week_in_tech.rss
 9 ./metadata/Windows_weekly.rss

Opening remarks

Unfortunately, the documentation of Erlang’s XML parsing library is pretty scant and — apart from the xmerl User’s Guide — I could not find any tutorials on xmerl on the web.

Again, a language aspiring to widespread adoption should have more material covering these kinds of basics.

There were a few choices as to how to go about the parsing business:

• using xmerl_scan with or without customization functions.
• using xmerl_scan together with xmerl_xpath (an XPath 1.0 implementation).

I chose the first approach because it was better documented. Only after finishing the first cut of the parsing code did I find xmerl_xpath usage examples on a mailing list and played with it.

The code

The Erlang code that finds the RSS files listed above and parses them is as follows:

  1 -module(xml2).
  2 -export([main/1]).
  3 -include_lib(“xmerl/include/xmerl.hrl”).
  4 
  5 parseAll(D) ->
  6     % find all RSS files underneath D
  7     FL = filelib:fold_files(D, “.+.rss$”, true, fun(F, L) -> [F|L] end, []),
  8     [ parse(F) || F <- FL ].
  9 
 10 parse(FName) ->
 11     % parses a single RSS file
 12     {R,_} = xmerl_scan:file(FName),
 13     % extract episode titles, publication dates and MP3 URLs
 14     L = lists:reverse(extract(R, [])),
 15     % print channel title and data for first two episodes
 16     io:format(“~n>> ~p~n“, [element(1,lists:split(3,L))]),
 17     L.
 18 
 19 % handle ‘xmlElement’ tags
 20 extract(R, L) when is_record(R, xmlElement) ->
 21     case R#xmlElement.name of
 22         enclosure ->
 23             if element(1, hd(R#xmlElement.parents)) == item ->
 24                     FFunc = fun(X) -> X#xmlAttribute.name == url end,
 25                     U = hd(lists:filter(FFunc, R#xmlElement.attributes)),
 26                     [ {url, U#xmlAttribute.value} | L ];
 27                 true -> L
 28             end;
 29         channel ->
 30             lists:foldl(fun extract/2, L, R#xmlElement.content);
 31         item ->
 32             ItemData = lists:foldl(fun extract/2, [], R#xmlElement.content),
 33             [ ItemData | L ];
 34         _ -> % for any other XML elements, simply iterate over children
 35             lists:foldl(fun extract/2, L, R#xmlElement.content)
 36     end;
 37 
 38 extract(#xmlText{parents=[{title,_},{channel,2},_], value=V}, L) ->
 39     [{channel, V}|L]; % extract channel/audiocast title
 40 
 41 extract(#xmlText{parents=[{title,_},{item,_},_,_], value=V}, L) ->
 42     [{title, V}|L]; % extract episode title
 43 
 44 extract(#xmlText{parents=[{link,_},{item,_},_,_], value=V}, L) ->
 45     [{link, V}|L]; % extract episode link
 46 
 47 extract(#xmlText{parents=[{pubDate,_},{item,_},_,_], value=V}, L) ->
 48     [{pubDate, V}|L]; % extract episode publication date (’pubDate’ tag)
 49 
 50 extract(#xmlText{parents=[{'dc:date',_},{item,_},_,_], value=V}, L) ->
 51     [{pubDate, V}|L]; % extract episode publication date (’dc:date’ tag)
 52 
 53 extract(#xmlText{}, L) -> L.  % ignore any other text data
 54 
 55 % ‘main’ function (invoked from shell, receives command line arguments)
 56 main(A) ->
 57     D = atom_to_list(hd(A)),
 58     parseAll(D).

Conclusion

Erlang’s filelib:fold_files() function is cool and a good example of how easy things should be.

On the other hand:

• it took quite a bit of time and effort to write the code above (perhaps due to my lack of experience in functional programming) and it was not fun
• beauty is in the eye of the beholder as they say and the promise of being able to write clean and attractive code is a major reason to pick up a new programming language. Again, maybe it’s just me but I did not find the resulting Erlang code to be particularly attractive.
• the XML parser chokes reproducibly on XML files with non-ASCII character sets (try the code e.g. with the following RSS file (containing german characters))
• The XPath implementation appears to be incomplete: I could not use the | operator in an XPath expression to select several paths for example.

Anyway, that’s just my $ 0.02 on XML parsing with Erlang. I am not an expert by any stretch of the imagination so feel free to point to anything I may have missed.