Category Archives: Decorator

Decorator fix

Posted on by
Reply

Decorator implementation correction

Again, I would like to thank Brian McErlean who pointed out some issues in the Params2attribs decorator code (see his first two comments on part three).

I reworked the implementation of the __call__ method as shown below and it now handles parameter defaults as well as keyword parameters properly.

 1  #!/usr/bin/env python
 2  """
 3  Module with a decorator class for initialiser (__init__()) methods. The
 4  decorator wraps initialiser methods in functions that copy the initialiser
 5  parameter values to attributes of the respective object.
 6  """
 7
 8  # Copyright: (c) 2006 Muharem Hrnjadovic
 9  # created: 16/10/2006 06:49:38
10
11  __version__ = "$Id$"
12  # $HeadURL $
13
14  import inspect
15
16  class Params2attribs(object):
17      """Decorator class for initialiser (__init__()) methods, wraps them
18      in functions that copy the initialiser parameter values to attributes
19      of the respective object."""
20      def __init__(self, exclude_params=None):
21          """what parameters should we exclude from copying (if any)?"""
22          self.exclude_params = exclude_params
23      def __call__(self, f):
24          """define and return the wrapper for the decorated __init__() method"""
25          # introspect f() for parameter data, 'argspec' becomes part of
26          # the wrapper() closure
27          argspec = inspect.getargspec(f)
28          def wrapper(*args, **kwargs):
29              """wrapper function for a decorated __init__ method"""
30              # the first parameter is the initialiser's object reference
31              obj = args[0]
32              # initialise parameter value list
33              pvalues = [None] * (len(argspec[0]) - 1)
34              # first superimpose the default values (if any)
35              if argspec[3]: pvalues[len(pvalues)-len(argspec[3]):] = argspec[3]
36              # now superimpose the positional parameter values
37              pvalues[0:len(args)-1] = args[1:]
38              # combine the initialiser's parameter names with their
39              # respective values (excluding the object reference)
40              argvl = zip(argspec[0][1:], pvalues)
41              # finally, superimpose the keyword parameter values 
42              for k in kwargs: argvl[argspec[0].index(k)-1] = (k, kwargs[k])
43              # copy the initialiser parameters to attributes of its object
44              self.copy2attribs(obj, argvl)
45              # finally, invoke the wrapped initialiser method
46              f(*args, **kwargs)
47          # assign the wrapped method's doc string to the wrapper
48          wrapper.func_doc = f.func_doc
49          # .. and eventually return the wrapper function
50          return wrapper
51      def copy2attribs(self, obj, argvl):
52          """copy the attribute/value pairs in 'argvl' to attributes of object
53             'obj'. Resolve any embedded attribute/value pair lists through
54             recursive calls"""
55          for argn, argv in argvl:
56              # resolve embedded parameters via a recursive call
57              if isinstance(argn, list):
58                  self.copy2attribs(obj, zip(argn, argv))
59              # skip unwanted initialiser parameters
60              elif self.exclude_params and argn in self.exclude_params:
61                  continue
62              else: setattr(obj, argn, argv)

Please note: only lines 32 – 42 (above) changed. The rest of the code is the same. Speaking of code: now the source files are available as well. Please find them here:

Python decorator mini-study (part 3 of 3)

Posted on by
3

Introduction

In part one of this article series I introduced the Params2attribs decorator that copies object initialiser parameters to object attributes.
In part two we took a closer look at the Python decorator mechanics.

This is the final part in which we are going to dissect the Params2attribs decorator.

Overview

From the code below you can glean that the decorator is implemented as a class with three methods:

  • initialiser (lines 20 – 22)
  • the special __call__ method that gets invoked when you treat an object like a function (lines 23 – 42)
  • an auxilliary method (copy2attribs) that’s in charge of the parameter copying business

Before diving into the code, please bear in mind that the Params2attribs::__call__() method gets executed whenever Python loads any code decorated with Params2attribs (e.g. lines 19 – 20 in part one). At that point the wrapper function is only defined and returned to Python as a replacement for the decorated function.

In contrast, the wrapper function is executed whenever the decorated function is called (e.g. line 27 in part one).

If any of this is unclear please refer back to part two where the workings of Python decorators were explained in greater detail.

 1  #!/usr/bin/env python
 2  """
 3  Module with a decorator class for initialiser (__init__()) methods. The
 4  decorator wraps initialiser methods in functions that copy the initialiser
 5  parameter values to attributes of the respective object.
 6  """
 7
 8  # Copyright: (c) 2006 Muharem Hrnjadovic
 9  # created: 16/10/2006 06:49:38
10
11  __version__ = "$Id$"
12  # $HeadURL $
13
14  import inspect
15
16  class Params2attribs(object):
17      """Decorator class for initialiser (__init__()) methods, wraps them
18      in functions that copy the initialiser parameter values to attributes
19      of the respective object."""
20      def __init__(self, exclude_params=None):
21          """what parameters should we exclude from copying (if any)?"""
22          self.exclude_params = exclude_params

In order to see what is going on here I will use the debugger to display the values of various variables of interest.

For example, when the Params2attribs::__call__() method runs, the value of argspec (as returned by the inspect.getargspec() utility (line 27)) is as follows:

argspec = ( ['self', 'arg_1', ['arg_2', 'arg_3'], 'arg_4', 'arg_5'],
            None,
            None,
            None)

What we are after here are the names of the parameters (passed to the decorated function) and they are all contained in the first element of the argspec tuple. Please note how the names of embedded parameters are returned in an embedded list which is quite handy as we will see later.

23      def __call__(self, f):
24          """define and return the wrapper for the decorated __init__() method"""
25          # introspect f() for parameter data, 'argspec' becomes part of
26          # the wrapper() closure
27          argspec = inspect.getargspec(f)

The wrapper

At the point of its definition the wrapper function has access to the argspec tuple due to lexical scoping but argspec will be available to it at execution time as well because it is part of its closure.

When a CoolApproach object is instantiated as shown on line 27 in part one, the wrapper function is called and the value of its args parameter is as follows:

args = ( <__main__.CoolApproach object at 0xb7b47f6c>,
         'aa',
         ('E1', 'E2'),
         22,
         3.3300000000000001)

The first argument is a CoolApproach reference (due to invocation on a CoolApproach object). The others were passed on line 27 in part one.

28          def wrapper(*args, **kwargs):
29              """wrapper function for a decorated __init__ method"""
30              # the first parameter is the initialiser's object reference
31              obj = args[0]
32              # combine the initialiser's parameter names with their
33              # respective values (but ommit the initialiser's object reference)
34              argvl = zip(argspec[0][1:], args[1:])
35              # copy the initialiser parameters to attributes of its object
36              self.copy2attribs(obj, argvl)
37              # finally, invoke the wrapped initialiser method
38              f(*args, **kwargs)
39          # assign the wrapped method's doc string to the wrapper
40          wrapper.func_doc = f.func_doc
41          # .. and eventually return the wrapper function
42          return wrapper

The first step is to combine the parameter names and values in one list, namely argvl (see line 34 above) which then has the following value:

argvl = [ ('arg_1', 'aa'),
          (['arg_2', 'arg_3'], ('E1', 'E2')),
          ('arg_4', 22),
          ('arg_5', 3.3300000000000001)]

Subsequently, the argvl list is passed (along with the object reference) to the copy2attribs helper function which will do the copying for us (line 36 above).
Last but not least, the wrapped function itself is invoked on line 38 (above).

The helper function

Eventually, the code below copies the parameter name/value pairs to object attributes. While doing so, it observes the list of parameters that are not to be copied (see line 52).

43      def copy2attribs(self, obj, argvl):
44          """copy the attribute/value pairs in 'argvl' to attributes of object
45             'obj'. Resolve any embedded attribute/value pair lists through
46             recursive calls"""
47          for argn, argv in argvl:
48              # resolve embedded parameters via a recursive call
49              if isinstance(argn, list):
50                  self.copy2attribs(obj, zip(argn, argv))
51              # skip unwanted initialiser parameters
52              elif self.exclude_params and argn in self.exclude_params:
53                  continue
54              else: setattr(obj, argn, argv)

If you were to run the code in the debugger (highly recommended :-)) and placed a breakpoint inside the copy2attribs function you would see that it is invoked twice.

The argvl value seen for the first call is:

argvl = [ ('arg_1', 'aa'),
          (['arg_2', 'arg_3'], ('E1', 'E2')),
          ('arg_4', 22),
          ('arg_5', 3.3300000000000001)]

The argvl value for the second call is:

argvl = [('arg_2', 'E1'), ('arg_3', 'E2')]

How is this to be understood? The first invocation is from the wrapper function (line 36). The second invocation occurs through a recursive call (line 50) when the top-level copy2attribs invocation processes the embedded list (second element of argvl: (['arg_2', 'arg_3'], ('E1', 'E2'))).

Conclusion

This concludes the Python decorator mini-study. I hope you liked it and would appreciate your comments. Also, I will be making the sources available as soon as I have figured out how I can upload something that is not a picture to WordPress.

Addendum

Please note: the fix for the issues pointed out in the comments as well as the links to the source code can be found here.

Python decorator mini-study (part 2 of 3)

Posted on by
8

Introduction

In the previous part of this article I introduced the Params2attribs decorator that copies object initialiser parameters to object attributes.

In this part I aim to introduce the Python decorator mechanisms needed to drill down into Params2attribs in the forthcoming third part of this series.

Python decorator 101

Python decorators come in two flavours: simple and not so simple :-) The Params2attribs decorator which is the subject of this article series falls into the second category. But, as they say, first things first..

Simple decorators

The simple decorator variety is in essence a function that returns a wrapper function that is to be invoked instead of the wrapped function. Confused? You should be :-)

Let’s have a look at an example. In the code below the wrapper function (lines 13 – 16) merely logs the invocation of the wrapped function.

Just in case you haven’t seen this before: the *args and **kwargs constructs (on lines 13 and 15) are the Python way to pass positional and keyword function parameters in a generic fashion.

 1  #!/usr/bin/env python
 2  """simple decorator example"""
 3
 4  # Copyright: (c) Muharem Hrnjadovic
 5  # created: 17/10/2006 08:25:09
 6
 7  __version__ = "$Id$"
 8  # $HeadURL $
 9
10  from sys import stdout
11
12  def interceptor(f):
13      def wrapper(*args, **kwargs):
14          stdout.write("Hey %s(), you have been wrapped..\\n\\t" % f.__name__)
15          f(*args, **kwargs)
16          stdout.write(".. resistance is futile\\n")
17      return wrapper
18
19  @interceptor
20  def plf(): stdout.write("Help! I am only a poor little function!\\n")
21
22  if __name__ == '__main__':
23      plf()

A few points to note here: the decorator function interceptor()

  • is invoked by Python whenever it loads any code decorated with it (lines 19 – 20)
  • defines (lines 13 – 16) and returns (line 17) another function (wrapper()) that is to be invoked (line 23) instead of the decorated one (plf())

As you may have guessed already, running the code above results in the following output:

mhr@playground2:~/src/published$ python simpledeco.py
Hey plf(), you have been wrapped..
        Help! I am only a poor little function!
.. resistance is futile

Complex decorators

Complex decorators may be implemented as classes or using nested functions. The latter variant was suggested by a reader and is shown after the section on decorator classes.

Decorator classes

When using decorator classes any data beyond the function to be wrapped is passed to the decorator class initialiser (see e.g. the before and after parameters of initialiser interceptor::__init__(), lines 13 – 15).

Furthermore, instances of decorator classes must be callable like functions. Our decorator class is hence implementing the special method __call__ (lines 16 – 21).

 1  #!/usr/bin/env python
 2  """not so simple decorator example"""
 3
 4  # Copyright: (c) Muharem Hrnjadovic
 5  # created: 17/10/2006 08:25:09
 6
 7  __version__ = "$Id$"
 8  # $HeadURL $
 9
10  from sys import stdout
11
12  class interceptor(object):
13      def __init__(self, before, after):
14          self.before = before
15          self.after = after
16      def __call__(self, f):
17          def wrapper(*args, **kwargs):
18              stdout.write(self.before)
19              f(*args, **kwargs)
20              stdout.write(self.after)
21          return wrapper
22
23  @interceptor('>> ', ' <<\\n')
24  def plf(): stdout.write("Help! I am only a poor little function!")
25
26  if __name__ == '__main__':
27      plf()

The code above works approximately in the following fashion:

  1. Python loads the module and encounters the decorated code on lines 23 – 24
  2. at that point it will instantiate an interceptor object and pass the two string parameters to its initialiser
  3. the initialiser just copies the before and after parameters to object attributes for later use (lines 13 – 15)
  4. then the interceptor object (instantiated in the previous step) is called by Python as a function at which point its __call__ method is invoked
  5. __call__ defines (lines 17 – 20) and returns (line 21) the wrapper function
  6. the call to plf() (line 27) invokes the wrapper() function defined and returned in step 5

The output (did you guess it?) is as follows:

mhr@playground2:~/src/published$ python notsosimpledeco.py
>> Help! I am only a poor little function! <<

Nested functions

As a reader pointed out after the first draft of this article was posted (see also the first two comments on this weblog entry) a similar effect can be achieved using nested functions. The nested functions variant of the decorator looks as follows:

 1  #!/usr/bin/env python
 2  """a variant of the not so simple decorator example"""
 3
 4  from sys import stdout
 5
 6  def interceptor(before, after):
 7      def interceptor_inner(f):
 8          def wrapper(*args, **kwargs):
 9              stdout.write(before)
10              f(*args, **kwargs)
11              stdout.write(after)
12          return wrapper
13      return interceptor_inner
14
15  @interceptor('>> ', ' <<'\\n')
16  def plf(): stdout.write("Help! I am only a poor little function!")
17
18  if __name__ == '__main__':
19      plf()

Conclusion

This is the end of part two of the Python decorator mini-study. In the next part of this article (part 3) we’ll finally get to dissect the Params2attribs decorator introduced in part one

Python decorator mini-study (part 1 of 3)

Posted on by
8

Introduction

On quite a few occasions I would find myself coding Python object initialiser methods where I would manually copy the method parameters to object attributes in order to make use of them later (see e.g. TheHardWay::__init__(), lines 13 – 17 below).

This is an unnecessary laborious, error-prone and unpythonic activity :-) In this article I hence present an alternative and more pythonic solution: a function decorator that copies initialiser method parameters to object attributes (see lines 19 – 24 below for an example showing how that decorator is used).

Furthermore, this article shows how to devise Python function decorators that require additional data (beyond the function that is to be decorated) in order to be useful.

Last but not least, the decorator to be shown in (part 3 of) this article is most useful in conjunction with initialiser methods but there is nothing to prevent you from using it with any object method.

Warm-up exercise

Again, the code section below (lines 13 – 17) shows how to copy the parameters to object attributes in a manual fashion.

 1  #!/usr/bin/env python
 2  """Demo code for initialiser method decorator"""
 3
 4  # Copyright: (c) 2006 Muharem Hrnjadovic
 5  # created: 15/10/2006 11:21:01
 6  __version__ = "$Id$"
 7  # $HeadURL $
 8
 9  import pprint as PP, sys
10  from p2adeco import Params2attribs
11
12  class TheHardWay(object):
13      def __init__(self, arg_a, arg_b, arg_c):
14          self.arg_a = arg_a
15          self.arg_b = arg_b
16          self.arg_c = arg_c
17          # .. now do whatever initialisation is required ..

The alternative solution below uses a python function decorator (on line 19) which is given a (possibly empty) tuple of parameter names for which no copying should occur.
In this particular example we don’t want arg_3 to be copied and hence specify it in the tuple passed to the decorator’s object initialiser.

Please note also that the decorator is capable of handling embedded parameter lists (like e.g. parameters arg_2 and arg_3 which are passed to the decorated initialiser via such a list (on line 20)).

18  class CoolApproach(object):
19      @Params2attribs(('arg_3',))
20      def __init__(self, arg_1, (arg_2, arg_3), arg_4, arg_5):
21          # .. at this point all parameters except for 'arg_3' have been
22          # copied to object attributes
23          # .. now do whatever initialisation is required ..
24          print ">> In initialiser, self.arg_1 = '%s'" % self.arg_1

Finally, the block below merely instantiates a CoolApproach object (line 27) in order to demonstrate the described decorator behaviour. Line 28 sorts the attributes of the newly instantiated object for clarity.

25  if __name__ == '__main__':
26      embedded_params = ('E1', 'E2')
27      obj = CoolApproach('aa', embedded_params, 22, 3.33)
28      sorted_attributes = sorted(obj.__dict__.iteritems())
29      sys.stdout.write("\\nObject attributes:\\n%s\\n" %
30                       PP.pformat(sorted_attributes, indent=4, width=60))

When running the code above the following output results:

>> In initialiser, self.arg_1 = 'aa'

Object attributes:
[   ('arg_1', 'aa'),
    ('arg_2', 'E1'),
    ('arg_4', 22),
    ('arg_5', 3.3300000000000001)]

From the output we can see that:

  1. the CoolApproach initialiser method was entered (print statement on line 24)
  2. all the desired initialiser parameters were copied to attributes of the respective object (print statement on lines 29 – 30)

Please note also that arg_3 is not listed as an attribute in the output above indicating that the decorator is paying attention to the parameter exclusion list passed to it.

Conclusion

This is the end of part one, the next part of this article introduces the Python decorator mechanics needed for the final part (dissecting the Params2attribs decorator)