• programming in python • a course for the curious •

Vim & Python

There are a few plugins for writing Python code in Vim. Just recently I started to use jedi-vim. It implements omnicompletion for Python using the jedi module. Both jedi-vim and jedi are authored by David Halter. There are a few things you can configure and they are well documented on the github jedi-vim page. The plugin works for me very smoothly, its nicely designed and has some cool features. I really like the jedi#goto() function which by default is bound to <Leader>g. When you are on a variable, class or function name and you hit <Leader>g it takes you straight to the definition of that object. I've implemented such a function for vim scripts (ftpdev plugin) and it helps a lot working with code. Jedi-vim allows to change the default key bindings, which is also nice since I got used to hitting gd to go to a local declaration. Let me note that it is really well done: it can take you to different modules where the object was defined. If you import an object from a module this might require two steps: first will take you to the import statement and next will go to that module, if you imported whole module it will take you directly to the object definition. The only restriction are the built-in modules which are implemented in C rather than Python, but you will not go around this anyway.

Jedi-vim can also find related names (with <Leader>n) in the current project and populate the quick fix list with the results. Related names are the ones which correspond to the same object. Let say that you import a function from a module in two different places (files): then jedi will find that these two names are related. Then you can easily jump between different position where the name occurs.

Another nice feature is the renaming function. It renames all the related names. I haven't played much with this but it sounds promising: especially if you change your mind with the object name at some point. You can first use the <Leader>n to review where the changes will be made. It seems that if you change the name of an object with the 'as' keyword (from ... import ... as ...) the name just after import, which is the original name of the object is not added to the related names, but I guess this can be fixed ...

Another nice feature to have is folding. There are a few plugin which can fold python code and I found SimplyFold the best solution. It really does the job well. The only problem is that is might be slow on big python files. For me it was not working on my previous laptop (which was a bit slow any way) with 2000 lines of code but with a new fast computer it works just fine. If it is too slow for you, you can always use the trick: ":set fdm=indent". It will fold probably more that you would like to, but it should be fast enough.

I also use the following helper motion commands:

  1. fun! Python_jump(pat, flag, count)
  2. normal! m`
  3. for x in range(a:count)
  4. call search(a:pat, a:flag)
  5. endfor
  6. endfun
  7. nnoremap <silent> <buffer> ]] :<C-U>call Python_jump('^\s*\zs\(def\\|class\)\>', 'W', v:count1)<cr>
  8. nnoremap <silent> <buffer> [[ :<C-U>call Python_jump('^\s*\zs\(def\\|class\)\>', 'bW', v:count1)<cr>
  9. nnoremap <silent> <buffer> ]} :<C-U>call Python_jump('^\(def\\|class\)\>', 'W', v:count1)<cr>
  10. nnoremap <silent> <buffer> [{ :<C-U>call Python_jump('^\(def\\|class\)\>', 'bW', v:count1)<cr>

There is yet one more plugin which is useful. PyInteractive can turn vim command line into Python REPL, which state is persistent (i.e. all objects are left intact when you switch it on and off). It is nice to test code snippets, but also you can run buffers. It has completion so it much nicer that typing :py print(...) all the time.

Python3.3 on Gentoo

Do you want to install Python3.3 on Gentoo. It is plain simple, though you should change a few settings if you want to maintain it the right way and do not have portage complains about missing dependencies. Here are simple steps:

  • Python3.3 is not hard masked any more, but it is kye-word masked. To unmask it add

    =dev-lang/python-3.3.2 ~amd64  # don't forget to use your arch keyword

    to /etc/portage/package.keywords,

  • add python3_3 to PYTHON_TARGETS variable in /etc/portage/make.conf

  • add 3.3 to USE_PYTHON in /etc/poratege/make.conf

  • unmask python_targets_python3_3 and python_single_target_python3_3 use flags in /etc/portage/profile/use.mask by adding the following lines:

  • now you need to rebuild your python library with python_updater and you you are done!

Here you have the Gentoo python-r1 eclass guide. Though it does not mention the fourth step (unmasking the use flags), I had a dependencies problem using emerge -DuNa word which has been solved by this step.

Python generators

This is a simple introduction to generators in Python.

First let me explain what is going on when you do a for loop:

>>> for x in iterable_object :
...     print(x)

First python calls iterable_object.__iter__ method to get an object over which it will iterate. Let me make a diggression how you can check this in the Python source code (yes it is not difficult). First compile a simple loop and check the opcodes:

>>> loop = "for x in iter: pass"
>>> loop_code = compile(loop, '<string>', 'exec')
>>> from dis import dis
>>> dix(loop_code)  # deassemble the python code object into single instructions (opcodes)
                    # each item in the following line is a single opcode which then is translated into
                    # c instructions in cPython.
1           0 SETUP_LOOP              14 (to 17)
            3 LOAD_NAME               0 (iter)
            6 GET_ITER
        >>  7 FOR_ITER                6 (to 16)
            10 STORE_NAME             1 (x)
            13 JUMP_ABSOLUTE          7
        >>  16 POP_BLOCK
        >>  17 LOAD_CONST             0 (None)
            20 RETURN_VALUE

Now do you see the GET_ITER opcode. You can open Python/ceval.c file in the Python source code and you will see that it calls PyObject_GetIter c function, which actually corresponds to the __iter__ method. Ok, but this was a digression, I will come back to this later ...

Python has many iterable objects (i.e. the ones which have __iter__() method. The simplest example is Python2 str and unicode or Python3 bytes and str, but there are also others like: tuple, list and dict. The bound method __iter__ does not accept any arguments and it is supposed to return an iterator object. The iterator object contains __next__ method which returns the x values, unless the iteration has exhousted. In the last case it raises StopIteration exception which is internally cought by the for loop. There is however another way of getting iterator objects using the yield keyword.

>>> def gen(n=5):
...     for x in range(n):  # in Python2 you'd use xrange
...         yield x
>>> for x in gen(3):
...     print(x, end=' ')  # Python3 for Python2 do "from __future__ import print_function"
0 1 2 >>>

The function gen returns a generator object. It has __iter__ method which returns itself, since generator objects can be iterated directly (it already contains __next__ method among other special methods which you will see in a moment). So let us check it out:

>>> g = gen(2)
>>> g is g.__iter__()
>>> g.__next__()
>>> next(g)  # this is a wrapper around ``__next__``
>>> next(g)

But more interestingly generators have a send() method. What it is used for: to send data back to the generator. It accepts a single argument which is then send to the generator. It turns out that the yield is an expression (and when used as an expression should almost always be surrounded by brackets) - and expressions can be evaluated. Let us first begin with a note that .send(None) is equivalent to __next__():

>>> g = gen(2)
>>> g.send(None)
>>> g.send(None)
>>> g.send(None)

First time you call send() method (unless you called __next__() already) you have to pass None as the argument (or use __next__()). So let us check out how it works:

>>> def gen(n=5):
...   for x in range(n):
...     received = (yield x)
...     print('received={}'.format(repr(recieved))
>>> g = gen(3)
>>> next(g)  # or g.send(None)
...          # advances the generator to the first yield statement
...          # and we got what was 'yielded':
>>> # now the generator is at a yield statement so we can send something to it
>>> g.send(object)
recieved=<class 'object'>

Actually two things happen when you use send() method:

  • the (yield x) evaluates to what was passed as an argument of send(),
  • the method __next__() is called: i.e. the generator code runs until next yield statement is met.

There are two useful utility function for generators:

>>> from functools import wraps
>>> def init(func):
...     """decorator which initialises the generator
...     """
...     @wraps(func)
...     def inner(*args, **kwargs):
...         g = func(*args, **kwargs)
...         g.send(None)
...         return g
...     return inner
>>> def consumer(gen, data):
...     """gen is an initialised generator
...     """
...     for d in data:
...         try:
...             gen.send(d)
...         except StopIteration:
...             break

Let us make a simple example:

>>> @init
... def gen(func):
...     x = (yield)
...     while True:
...         x = (yield func(x))

This is simply an implementation of function func as a generator, maybe not very useful (since you can use function by itself), but intriguing:

>>> g = gen(lambda x: x**2)
>>> g.send(2)
>>> g.send(9)

Python3.3 has also yield from which is you can read about here. We also didn't touch the throw method of generators. And there is a difference between yield from and looping a generator when an exception is caught (yield from will print a more readable traceback). And one nice new feature in Python3.3 is that you can mix return and yield in one function block.

For a more serious application of generators see this article.

Command line user interface with Python

There are two options for parsing the command line that you might think off at first: the optparse module (which is obsolte by the way) and the argparse module. But there is a much simpler and more efficient tool: docopt module. It gives you just one function docopt.docopt(), which can make the magic. You just pass a usage message and optionally an argv list. For example let us implement on of the iptables interfaces. The man page of iptables specification contains these lines:

  1. iptables [-t table] {-A|-C|-D} chain rule-specification
  2. rule-specification = [matches...] [target]
  3. match = -m matchname [per-match-options]
  4. target = -j targetname [per-target-options]

I will only ignore per-match-options and per-target-options to keep things simple. The usage string that will work with docopt module in this case is almost a copy of the above specification:
  1. doc="""Usage:
  2. iptables [-t=<table>] (-A|-C|-D) <chain> [-m=<match_name>]... [-j=<target_name>]
  3. Options:
  4. -t=<table> table [default: table]
  5. -m=<match_name> match
  6. -j=<target_name> target
  7. """

The docopt function can parse both `Usage:` and `Options:` part and get the relevant information. The usage message should be compliant with the POSIX standard which most of you are familiar with, since many (though not all) linux man pages are written using it.

Now let us test the result:

  1. >>> from docopt import docopt
  2. >>> cmd = docopt(doc, ['-t', 'nat', '-D', 'INPUT', '-m', 'conntrack', '-j', 'ACCEPT'])
  3. >>> cmd # it is just a dictionary:
  4. {'-A': False,
  5. '-C': False,
  6. '-D': True,
  7. '-j': 'ACCEPT',
  8. '-m': ['conntrack'],
  9. '-t': 'nat',
  10. '<chain>': 'INPUT'}

Let me explain it in more detail: the [ ] means that this part is optional (you can also set the default value), the (-A|-C|-D) means that one of the switches -A, -C, -D is required and the ellipsis ... means that you can repeat the previous option:

  1. >>> cmd = docopt(doc, ['-A', 'CHAIN', '-m', 'MATCH_1', '-m', 'MATCH_2'])
  2. >>> cmd
  3. {'-A': True,
  4. '-C': False,
  5. '-D': False,
  6. '-j': None,
  7. '-m': ['MATCH_1', 'MATCH_2'], # here is a list of all -m values
  8. '-t': 'table',
  9. '<chain>': 'CHAIN'}

The limitation I have found is the parsing of the following git command:

  1. >>> git_cmd = """Usage:
  2. ... git [--version] [--help] [-c <option>=<value>]...
  3. ..."""
  4. >>> docopt(git_cmd, ['-c', 'name=value'])
  5. {"--help": false,
  6. "--version": false,
  7. "-c": true,
  8. "<name>=<value>": "name=value"}

But if we remove the '=' we can get a very similar interface:

  1. >>> git_cmd = """Usage:
  2. ... git [--version] [--help] [-c <name> <value>]...
  3. ..."""
  4. ...
  5. >>> cmd = docopt(git_cmd, ['-c', 'user.name', 'me', '-c', 'user.email', 'me@email.org'])
  6. >>> cmd
  7. {"-c": 2,
  8. "<option>": ["user.name", "user.email"],
  9. "<value>": ["me", "me@email.org"]}

Now you can use href="http://docs.python.org/2/library/functions.html?highlight=zip#zip">zip
to get a dictionary of options:

  1. >>> config_opts = dic(zip(cmd['<name>'], cmd['<value>']))
  2. >>> config_opts
  3. { 'user.name': 'me', 'user.email': 'me@email.org'}

You can find more examples a youtube video from PyCon2012 and the documentation at docopt.org.

Completion for the linux lpr command

Quite often I print from the command line using the command lpr. Unfortunatelly, the bash-completion package does not provide a command line completion for it. Here you can find my bash completion script which supports all of the lpr switches. The script adds also a completion for the lprm command. How to install it? Well, it depends on the Linux distribution. On Gentoo, just put this script in the $HOME/.bash_completion.d/ folder, or if you want to do that system-wise drop it under /etc/bash_completion.d directory (this directory exists in major Linux distribution, while the former one is Gentoo specific).

  1. # lpr(1) and lprm(1) completions by Marcin Szamotulski <mszamot@gmail.com> March 2011
  2. # This script uses two functions from 'base': _known_hosts_real and _user_at_host.
  3. # to complete -H and -U switches of the lpr command (-h and -U of lprm).
  4. _lpr()
  5. {
  6. local cur prev opts comp_opts configfile
  8. cur="${COMP_WORDS[COMP_CWORD]}"
  9. prev="${COMP_WORDS[COMP_CWORD-1]}"
  10. comp_opts="+o default +o filenames"
  11. if [[ ${prev} == "=" ]];then
  12. prev="${COMP_WORDS[COMP_CWORD-2]}"
  13. fi
  14. if [[ ${cur} == "-" ]];then
  15. opts="-E -H -C -T -J -P -U -# -h -l -m -o -p -q -r"
  16. COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
  17. else
  18. case ${prev} in
  19. -o)
  20. opts="scaling= media= sides= Collate= orientation-requested=
  21. job-sheets= job-hold-until= page-ranges= page-set= number-up=
  22. page-border= number-up-layout= outputorder= cpi= lpi= ppi=
  23. columns= page-left= page-right= page-top= page-bottom= position=
  24. natural-scaling= hue= saturation= penwidth= landscape mirror raw
  25. fitplot prettyprint nowrap blackplot";;
  26. sides)
  27. opts="two-sided-long-edge two-sided-short-edge one-sided";;
  28. Collate)
  29. opts="True False";;
  30. job-hold-until)
  31. opts="indefinite day-time night second-shift third-shift
  32. weekend";;
  33. job-sheets)
  34. opts="none classified confidential secret standard topsecret
  35. unclassified";;
  36. media)
  37. opts="Letter Legal A4 COM10 DL Transparency Upper Lower
  38. MultiPurpose LargeCapacity Custom\.";;
  39. number-up-layout)
  40. opts="btlr btrl lrbt lrtb rlbt rltb tblr tbrl";;
  41. outputorder)
  42. opts="normal reverse";;
  43. page-border)
  44. opts="double none double-thick single single-thick";;
  45. page-set)
  46. opts="odd even";;
  47. position)
  48. opts="center top left right top-left top-right bottom bottom-left
  49. bottom-right";;
  50. orientation-requested)
  51. opts="3 4 5 6";;
  52. -P)
  53. opts=`lpstat -a 2>/dev/null | awk '{print $1}'`;;
  54. -i)
  55. opts=`lpstat -R 2>/dev/null | awk '{print $2}' | sed 's/^[a-zA-Z0-9\-]*\-//'`;;
  56. -H)
  57. comp_opts="no compreply"
  58. _known_hosts_real "$cur";;
  59. -U)
  60. comp_opts="no compreply"
  61. _user_at_host;;
  62. -C)
  63. comp_opts="-o filenames";;
  64. -J)
  65. comp_opts="-o filenames";;
  66. -T)
  67. comp_opts="-o filenames";;
  68. -#)
  69. comp_opts="-o filenames";;
  70. *)
  71. opts=""
  72. if [[ ${cur} != "=" ]];then
  73. compopt +o filenames
  74. _filedir
  75. comp_opts="no compreply"
  76. else
  77. return 0
  78. fi;;
  79. esac
  80. fi
  81. [[ ${cur} == "=" ]] && cur=""
  82. if [[ ${comp_opts} != "no compreply" ]];then
  83. if [[ ${comp_opts} != "" ]];then
  84. compopt ${comp_opts}
  85. fi
  86. COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) );
  87. fi
  88. # For options from opt_array do not append a white space after completin
  89. opt_array=("scaling media sides Collate orientation-requested job-sheets
  90. job-hold-until page-ranges page-set number-up page-border number-up-layout
  91. outputorder cpi lpi ppi columns page-left page-right page-top page-bottom
  92. position natural-scaling hue saturation penwidth")
  93. nospace=0
  94. for i in ${COMPREPLY};do
  95. for j in ${opt_array};do
  96. if [[ $i =~ $j ]];then
  97. nospace=1;
  98. fi
  99. done
  100. done
  101. [[ $nospace == 1 ]] && compopt -o nospace
  102. return 0;
  103. }
  104. complete -F _lpr lpr lp
  105. _lprm()
  106. {
  107. local cur prev opts configfile
  108. COMPREPLY=()
  109. cur="${COMP_WORDS[COMP_CWORD]}"
  110. prev="${COMP_WORDS[COMP_CWORD-1]}"
  111. [[ ${prev} == "=" ]] && prev="${COMP_WORDS[COMP_CWORD-2]}"
  112. if [[ ${cur} == "-" ]];then
  113. opts="-E -P -U -h"
  114. COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
  115. else
  116. case ${prev} in
  117. -P)
  118. opts=`lpstat -a 2>/dev/null | awk '{print $1}'`
  119. COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) );;
  120. -h)
  121. _known_hosts_real "$cur";;
  122. -U)
  123. _user_at_host;;
  124. *)
  125. opts=`lpstat -R 2>/dev/null | awk '{print $2}' | sed 's/.*-\([0-9\-]*\)$/\1/'`
  126. COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) );;
  127. esac
  128. fi
  129. return 0;
  130. }
  131. complete -F _lprm lprm

Completion for Python Console

There are several ways how you can make your life easier using the Python console. For example you can switch on code completion: hitting enter you will get completion of your variable, object name, class name, or a built-in function. It also supports the name spaces of imported modules. Here I will guide you how to do that.

First create an environment variable $PYTHONSTARTUP and give it the value '$HOME/.pystartup'. Now add the follwing content to this file:

  1. import atexit
  2. import os
  3. import os.path
  4. import sys
  5. import readline
  6. import rlcompleter
  7. historyPath = os.path.expanduser("~/.pyhistory")
  8. def save_history(historyPath=historyPath):
  9. import readline
  10. readline.write_history_file(historyPath)
  11. if os.path.exists(historyPath):
  12. readline.read_history_file(historyPath)
  13. atexit.register(save_history)
  14. del atexit, readline, save_history, historyPath, rlcompleter

You can enhance the rlcompleter if you want just subclass the rlcompleter.Completer object and rewrite its complete method. For example you can add completion for the import statement. This is a fun thing to do!

You can also use the $PYTHONSTARTUP file to load modules which you often use. For example the above script is not deleting the modules os, os.path and sys. Now you got the Tab completion which is a cool feature of IPython, though the Python console starts a way faster!

Contact us.
Created using , and
we are not associated with Python Software Foundation
© Copyright 2012, 2013 Accorda Institute.