Code refactoring with python’s functools.partial

In order to get a better feeling for what can be done with functools.partial() I am taking some “real world” python code of mine and refactoring it to use curried functions.

In the example below the construct_getopt_data() function (lines 7-17) takes the data structure shown on lines 44-54 (below) and returns a 2-tuple where

• the first element is a string of option letters (second argument to getopt() (see variable shortflags in the output below)) and
• the second element is a list of strings with the names of long options (third argument to getopt() (see variable longflags below))

The resulting data is thus as follows:

mhr@playground2:~/src/published$ python2.5 partialf.py
shortflags = 'l:ep:a:x:r:o:i:dc'
longflags =
(   'lines=',
    'echo',
    'fyo',
    'pager=',
    'pgr=',
    'algo=',
    'recipient=',
    'decrypt',
    'crypt')

>> test ok

And the code that’s producing the output above looks as follows:

 1  #!/usr/bin/env python
 2  from pprint import PrettyPrinter as PP
 3  from itertools import (imap, repeat)
 4  from functools import partial
 5  from operator import (ge, lt)
 6
 7  def construct_getopt_data(args):
 8      """uses lambdas"""
 9      # single and multi-character flag iterators
10      shortiter = lambda args: argiter(args, lambda s: s <= 2)
11      longiter = lambda args: argiter(args, lambda s: s > 2)
12
13      # single character flags
14      shortfs = imap(formatf, shortiter(args), repeat(':'))
15      # multi-character flags
16      longfs = imap(formatf, longiter(args), repeat('='))
17      return(''.join(shortfs), tuple(longfs))
18

Please note:

• the function above uses the lambda construct (on lines 10-11) to customise the short and long argument iterator respectively.
• the use of itertools.repeat() to supply the fchar parameter to the formatf() function (lines 14 and 16)

The analogous function construct_getopt_data2() (lines 19-29) below performs the same taks but uses curried functions as opposed to lambdas.

19  def construct_getopt_data2(args):
20      """uses functools.partial"""
21      # single and multi-character flag iterators
22      shortiter = partial(argiter, op=partial(ge, 2))
23      longiter = partial(argiter, op=partial(lt, 2))
24
25      # single character flags
26      shortfs = map(partial(formatf, fchar=':'), shortiter(args))
27      # multi-character flags
28      longfs = map(partial(formatf, fchar='='), longiter(args))
29      return(''.join(shortfs), tuple(longfs))
30

It utilises functools.partial()

1. on lines 22-23: to customise the argument iterator by
   1. currying the built-in operator functions operator.ge() (greater or equal) and operator.lt() (less than)
   2. using these curried operator functions to preset the op parameter of the argiter() generator
2. on lines 26 and 28 to preset the fchar parameter of the formatf() function (the ruse with itertools.repeat() is hence not needed any more)

31  def argiter(args, op):
32      """pair short/long flags will their respective types"""
33      for flags, argdata in args.iteritems():
34          for flag in flags:
35              if op(len(flag)): yield (flag, argdata[1])
36

The argiter() generator above facilitates the iteration over the input data structure (lines 44-54) in (single character flag, type) and (multi-character parameter, type) pairs respectively.

37  def formatf((argn, argt), fchar):
38      """format for getopt(),
39      argn is the flag, argt is its type, fchar is one of ':' or '='"""
40      return argt == bool and argn.lstrip('-') or "%s%s" % (argn.lstrip('-'), fchar)
41

The formatf() function above returns the single and multi-character command line flags in the format required by getopt().

42  if __name__ == '__main__':
43      # dictionary with command line args along with their types and defaults
44      args = {
45          ('-a', '-x', '--algo')      :   ('algo', str, None),
46          ('-c', '--crypt')           :   ('crypt', bool, None),
47          ('-d', '--decrypt')         :   ('decrypt', bool, None),
48          ('-e', '--echo', '--fyo')   :   ('echo', bool, None),
49          ('-l', '--lines')           :   ('lines', int, '25'),
50          ('-i', )                    :   ('input', str, None),
51          ('-o', )                    :   ('output', str, None),
52          ('-p', '--pager', '--pgr')  :   ('pager', str, '/usr/bin/less'),
53          ('-r', '--recipient')       :   ('recipient', str, None)
54      }
55      pp = PP(indent=4)
56      sfs1, lfs1 = construct_getopt_data(args)
57      sfs2, lfs2 = construct_getopt_data2(args)
58      print "shortflags =", pp.pformat(sfs1)
59      print "longflags =\\n", pp.pformat(lfs1)
60
61      if (sfs1 == sfs2) and (lfs1 == lfs2): print "\\n>> test ok"
62      else: print "\\n>> test failed"

In conclusion

While functools.partial() is certainly an interesting and cool addition to the python toolchest it would appear that it’s not indispensible for functional style programming.

I would love to see examples or code snippets that are made possible and/or improved greatly by leveraging functools.partial().

First experiments with python’s functools.partial

I started playing with one of the python2.5 features, namely functools.partial() that allows one to effectively “construct variants of existing functions that have some of the parameters filled in” (quote from Functional Programming HOWTO).

In the code below I am trying to preset the second paramater of the operator.lt() built-in function.

 1  #!/usr/bin/env python2.5
 2  """
 3  python 2.5 partial functions experiments (part 1)
 4  """
 5  import operator
 6  import functools
 7
 8  if __name__ == '__main__':
 9      #>>> help(operator.lt)
10      #Help on built-in function lt in module operator:
11      #
12      #lt(...)
13      #    lt(a, b) -- Same as a<b.
14
15      #>>> operator.lt
16      #<built-in function lt>
17
18      less_than_two = functools.partial(operator.lt, b=2)
19      print "1 < 2: ", less_than_two(1)
20      print "2 < 2: ", less_than_two(2)

However, when I run the code I get the following error:

mhr@playground2:~/src/published$ python2.5 partialf3.py
1 < 2:
Traceback (most recent call last):
  File "partialf3.py", line 19, in <module>
    print "1 < 2: ", less_than_two(1)
TypeError: lt() takes no keyword arguments

For some reason python’s built-in functions do not accept keyword arguments. Hence, my attempt to preset the second parameter (but leave the first alone (line 18 above)) does not work.

Now, I understand I can achieve the same effect by using the opposite (greater than) operator (operator.gt) in conjunction with preset positional parameters (see line 18 below).

 1  #!/usr/bin/env python2.5
 2  """
 3  python 2.5 partial functions experiments (part 2)
 4  """
 5  import operator
 6  import functools
 7
 8  if __name__ == '__main__':
 9      #>>> help(operator.gt)
10      #Help on built-in function gt in module operator:
11      #
12      #gt(...)
13      #    gt(a, b) -- Same as a>b.
14
15      #>>> operator.gt
16      #<built-in function gt>
17
18      less_than_two = functools.partial(operator.gt, 2)
19      print "1 < 2: ", less_than_two(1)
20      print "2 < 2: ", less_than_two(2)

This yields the expected behaviour:

mhr@playground2:~/src/published$ python2.5 partialf_2_5.py
1 < 2:  True
2 < 2:  False

The refusal on the part of python’s built-in functions to accept keyword arguments seems odd since it introduces somewhat of an inconsistency (built-in functions differing from “normal” functions in that regard).

I am wondering why this seeming inconsistency was introduced to python, a language that prides itself on a clean design.