Code refactoring with python’s functools.partial

In order to get a better feeling for what can be done with functools.partial() I am taking some “real world” python code of mine and refactoring it to use curried functions.

In the example below the construct_getopt_data() function (lines 7-17) takes the data structure shown on lines 44-54 (below) and returns a 2-tuple where

  • the first element is a string of option letters (second argument to getopt() (see variable shortflags in the output below)) and
  • the second element is a list of strings with the names of long options (third argument to getopt() (see variable longflags below))

The resulting data is thus as follows:

mhr@playground2:~/src/published$ python2.5 partialf.py
shortflags = 'l:ep:a:x:r:o:i:dc'
longflags =
(   'lines=',
    'echo',
    'fyo',
    'pager=',
    'pgr=',
    'algo=',
    'recipient=',
    'decrypt',
    'crypt')

>> test ok

And the code that’s producing the output above looks as follows:

 1  #!/usr/bin/env python
 2  from pprint import PrettyPrinter as PP
 3  from itertools import (imap, repeat)
 4  from functools import partial
 5  from operator import (ge, lt)
 6
 7  def construct_getopt_data(args):
 8      """uses lambdas"""
 9      # single and multi-character flag iterators
10      shortiter = lambda args: argiter(args, lambda s: s <= 2)
11      longiter = lambda args: argiter(args, lambda s: s > 2)
12
13      # single character flags
14      shortfs = imap(formatf, shortiter(args), repeat(':'))
15      # multi-character flags
16      longfs = imap(formatf, longiter(args), repeat('='))
17      return(''.join(shortfs), tuple(longfs))
18

Please note:

The analogous function construct_getopt_data2() (lines 19-29) below performs the same taks but uses curried functions as opposed to lambdas.

19  def construct_getopt_data2(args):
20      """uses functools.partial"""
21      # single and multi-character flag iterators
22      shortiter = partial(argiter, op=partial(ge, 2))
23      longiter = partial(argiter, op=partial(lt, 2))
24
25      # single character flags
26      shortfs = map(partial(formatf, fchar=':'), shortiter(args))
27      # multi-character flags
28      longfs = map(partial(formatf, fchar='='), longiter(args))
29      return(''.join(shortfs), tuple(longfs))
30

It utilises functools.partial()

  1. on lines 22-23: to customise the argument iterator by
    1. currying the built-in operator functions operator.ge() (greater or equal) and operator.lt() (less than)
    2. using these curried operator functions to preset the op parameter of the argiter() generator
  2. on lines 26 and 28 to preset the fchar parameter of the formatf() function (the ruse with itertools.repeat() is hence not needed any more)
31  def argiter(args, op):
32      """pair short/long flags will their respective types"""
33      for flags, argdata in args.iteritems():
34          for flag in flags:
35              if op(len(flag)): yield (flag, argdata[1])
36

The argiter() generator above facilitates the iteration over the input data structure (lines 44-54) in (single character flag, type) and (multi-character parameter, type) pairs respectively.

37  def formatf((argn, argt), fchar):
38      """format for getopt(),
39      argn is the flag, argt is its type, fchar is one of ':' or '='"""
40      return argt == bool and argn.lstrip('-') or "%s%s" % (argn.lstrip('-'), fchar)
41

The formatf() function above returns the single and multi-character command line flags in the format required by getopt().

42  if __name__ == '__main__':
43      # dictionary with command line args along with their types and defaults
44      args = {
45          ('-a', '-x', '--algo')      :   ('algo', str, None),
46          ('-c', '--crypt')           :   ('crypt', bool, None),
47          ('-d', '--decrypt')         :   ('decrypt', bool, None),
48          ('-e', '--echo', '--fyo')   :   ('echo', bool, None),
49          ('-l', '--lines')           :   ('lines', int, '25'),
50          ('-i', )                    :   ('input', str, None),
51          ('-o', )                    :   ('output', str, None),
52          ('-p', '--pager', '--pgr')  :   ('pager', str, '/usr/bin/less'),
53          ('-r', '--recipient')       :   ('recipient', str, None)
54      }
55      pp = PP(indent=4)
56      sfs1, lfs1 = construct_getopt_data(args)
57      sfs2, lfs2 = construct_getopt_data2(args)
58      print "shortflags =", pp.pformat(sfs1)
59      print "longflags =\\n", pp.pformat(lfs1)
60
61      if (sfs1 == sfs2) and (lfs1 == lfs2): print "\\n>> test ok"
62      else: print "\\n>> test failed"

In conclusion

While functools.partial() is certainly an interesting and cool addition to the python toolchest it would appear that it’s not indispensible for functional style programming.

I would love to see examples or code snippets that are made possible and/or improved greatly by leveraging functools.partial().

About these ads

One thought on “Code refactoring with python’s functools.partial

  1. Example that comes to mind is pkgcore’s source, http://dev.gentooexperimental.org/pkgcore-trac/browser (related, a fairly decent trac-bzr plugin based on jelmers work (iirc) is available there for anyone interested). Offhand, tracing pkgcore.util.currying would be wise; we use partial, and pre_curry/post_curry; pre_curry/post_curry are what I describe in a comment in your last blog entry.

    One reason for a duplicate pre_curry, is that partial cannot be bound into any class namespace; in other words, it’s essentially a staticmethod. pre_curry can be bound into the class namespace, thus it can be used to build normal methods (fairly useful).

    http://dev.gentooexperimental.org/pkgcore-trac/browser/pkgcore/ebuild/ebuild_src.py, base._get_attr is one (imo) good example of partial usage.

    The __getattr__ for that class reflects to _get_attr to generate an attr on the fly; we could use properties for it, but doing a memoizing property means that you always have to invoke some native python code for each lookup (if self.foo is not None: return self.foo\nelse generate foo); that adds up however, see comment in other blog entry about python func overhead.

    *Meanwhile*, so _get_attr is poked for attr -> functor; invokes the functor as functor(ebuild_src_instance).

    Now, we could just as easily break it into a 101 functions for it; but not a fan of 2 line functions (hate boilerplate), so went this route, which works fairly nicely.

    Re: indispensible… eh. Keep in mind that ‘bound’ methods, eg,
    class foo:
    __str__ = lambda s:”cheese”

    foo().__str__ # is a bound method

    Meaning, self is automatically ‘curried’ into the invocation; internally it doesn’t work this way (descriptor based, note the ‘bound’ vs ‘unbound’ comments), but at a high level it is a form of currying, and pretty indispensible.

    Required for functional style though? Would argue it is pretty much required due to the fact function prototypes don’t always match what is available at the invocation site; now either you wind up defining different versions of the func, or you wind up defining a shim that mangles the invocation site’s call into the normal expected args.

    partial serves as that shim; can do it other ways, but still need some way to push args/vars down to avoid duplicate function definitions; wrapping the target functor with it’s args, creating a new functor is far cleaner then passing down a (func, args, kwds) into code.

    Reason is simple imo; if you can wrap it into a functor, it behaves as if you were *not* currying args; if you have to use the tuple form to pass data down, that limits where you can pass it since the target code has to know about that passing form, and do an apply of sorts for invocation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s