I was naturally very excited, then, when GDB 7 introduced support for extension via Python. This seemed like an obvious step towards a “unified” debugging environment, one where I could step naturally between the languages, set breakpoints, etc. This new GDB feature doesn’t solve the problem directly, but it opens the door for more sophisticated extensions to GDB than were previously practical.

Building on this new GDB feature, recent Python source releases include GDB extensions as a build product. That is, starting at around python-3.2, when you build Python from source one of the build products is a set of GDB extensions that make it possible to do very natural debugging of Python from inside GDB. These extensions allow you, for example, to step up and down the Python call stack, display frame information (locals), and so forth. And this can be done while also navigating the base C call stack. This provides a very intuitive and productive debugging environment for anyone working across the two runtime environment.

edit: It was pointed out on reddit that gdb’s embedded Python interpreter has been around since 2009 and is thus not particularly new.

GDB 7′s Python extension system

What exactly is GDB’s new Python extension system? Broadly speaking, it’s a way to script GDB using Python rather than GDB’s own internal language. From around version 7 onward, GDB embeds a Python interpreter that you can invoke from the prompt and from scripts. For example:

(gdb) python
>import sys

To support writing extensions, you can import the module “gdb” from this embedded interpreter. This module provides access and hooks into GDB so that you can control it via Python scripts. You can read all about this Python extension support in GDB’s documentation.

This new facility has been used to implement, for example, pretty printers for STL containers. But in this article we’re interested in improving the CPython debugging experience, so we’ll cover that in the next section.

Python’s GDB extensions

As previously mentioned, recent Python source releases (from about 3.2 onward) include a new build product: a GDB Python extension that helps you debug Python. What does this mean? The extension encodes knowledge about CPython’s actual C-level structures, function frame implementation, PyObject, and so forth. It hooks this knowledge into GDB so that you can interrogate the CPython runtime at the Python source level rather than the CPython level.

For example, here are some partial stack traces from a break point inside a C extension module. The first one is a raw stack trace showing the guts of CPython execution:

(gdb) bt
 #0 get_value () at some_lib.c:1
 #1 0x00007ffff686348c in ffi_call_unix64 () at /home/abingham/src/Python-3.2/Modules/_ctypes/libffi/src/x86/unix64.S:75
 #2 0x00007ffff6862bc3 in ffi_call (cif=0x7fffffffde40, fn=, rvalue=, avalue=)
 at /home/abingham/src/Python-3.2/Modules/_ctypes/libffi/src/x86/ffi64.c:485
 #3 0x00007ffff68574ec in _call_function_pointer (pProc=, argtuple=, flags=,
 argtypes=, restype=, checker=) at /home/abingham/src/Python-3.2/Modules/_ctypes/callproc.c:808
 #4 _ctypes_callproc (pProc=, argtuple=, flags=, argtypes=,
 restype=, checker=) at /home/abingham/src/Python-3.2/Modules/_ctypes/callproc.c:1151
 #5 0x00007ffff684f823 in PyCFuncPtr_call (self=0x7ffff7e3a600, inargs=0x7ffff7f87050, kwds=)
 at /home/abingham/src/Python-3.2/Modules/_ctypes/_ctypes.c:3766
 #6 0x00000000004db737 in PyObject_Call (func=0x7ffff7e3a600, arg=0x506417, kw=0x4) at Objects/abstract.c:2149
 #7 0x00000000004658d4 in do_call (f=0x987d20, throwflag=) at Python/ceval.c:4095

The next one shows the Python debugging extension getting loaded, followed by a new stack trace. This new trace is still at the C source level, but you’ll notice that many of the CPython calls are annotated with Python source level information (e.g. file names, line numbers, etc.):

 (gdb) python
 >import python3_2_gdb
 >(gdb) bt
 #0 get_value () at some_lib.c:1
 #1 0x00007ffff686348c in ffi_call_unix64 () at /home/abingham/src/Python-3.2/Modules/_ctypes/libffi/src/x86/unix64.S:75
 #2 0x00007ffff6862bc3 in ffi_call (cif=0x7fffffffde40, fn=, rvalue=, avalue=)
 at /home/abingham/src/Python-3.2/Modules/_ctypes/libffi/src/x86/ffi64.c:485
 #3 0x00007ffff68574ec in _call_function_pointer (pProc=, argtuple=, flags=,
 argtypes=, restype=, checker=) at /home/abingham/src/Python-3.2/Modules/_ctypes/callproc.c:808
 #4 _ctypes_callproc (pProc=, argtuple=, flags=, argtypes=,
 restype=, checker=) at /home/abingham/src/Python-3.2/Modules/_ctypes/callproc.c:1151
 #5 0x00007ffff684f823 in PyCFuncPtr_call (self=0x7ffff7e3a600, inargs=(), kwds=) at /home/abingham/src/Python-3.2/Modules/_ctypes/_ctypes.c:3766
 #6 0x00000000004db737 in PyObject_Call (func=, arg=, kw=)
 at Objects/abstract.c:2149
 #7 0x00000000004658d4 in do_call (f=
 Frame 0x987d20, for file bar.py, line 5, in llama (lib=<CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>), throwflag=) at Python/ceval.c:4095
 #8 call_function (f=
 Frame 0x987d20, for file bar.py, line 5, in llama (lib=<CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>), throwflag=) at Python/ceval.c:3898
 #9 PyEval_EvalFrameEx (f=
 Frame 0x987d20, for file bar.py, line 5, in llama (lib=<CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>), throwflag=) at Python/ceval.c:2673

In particular, notice that the “python” command changes the GDB prompt to “>”. This is how you can tell when you’re in python mode. To exit python mode, you need to use Ctrl-D (on windows this might be Ctrl-Z…I haven’t checked.)

This final stack trace shows a Python source level stack trace provided by the extension:

 (gdb) py-bt
 #9 (unable to read python frame information)
 #12 Frame 0x987d20, for file bar.py, line 5, in llama (lib=<CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>)
 print(lib.get_value())
 #15 Frame 0x98a700, for file bar.py, line 9, in baz ()
 llama()

It’s important to recognize that this was all done in the same GDB session, at the same break point, etc. What you can see is that the Python-provided extension gives you a very clear picture of the exact state of the Python stack along with the underlying C stack. If you’ve ever tried debugging C extensions through GDB, you’ll immediately appreciate the clarity this provides.

Other extension features

The CPython GDB extension provides more than just stack traces and frame annotation. You can also use it to navigate the Python stack:

 (gdb) py-up
 #12 Frame 0x987d20, for file bar.py, line 5, in llama (lib=<CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>)
 print(lib.get_value())
 (gdb) py-up
 #15 Frame 0x98a700, for file bar.py, line 9, in baz ()
 llama()
 (gdb) py-down
 #12 Frame 0x987d20, for file bar.py, line 5, in llama (lib=<CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>)
 print(lib.get_value())

examine locals:

(gdb) py-locals

lib = <CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>

list source code:

 (gdb) py-list
 1 import ctypes
 2
 3 def llama():
 4 lib = ctypes.cdll.LoadLibrary('libsome_lib.so')
 5 print(lib.get_value())
 6
 7 def baz():
 8 print('baz')
 9 llama()
 10

and print individual variables:

 (gdb) py-print lib
 local 'lib' = <CDLL(_FuncPtr=, get_value=, _handle=9720816, _name='libsome_lib.so') at remote 0x7ffff6ac53d0>

Limitations

The CPython extension doesn’t (yet?) offer a complete debugging solution. For example, I haven’t found any way to set a break point in Python code. Likewise, you can’t set watches on Python variables. These features may not even be feasible with the GDB architecture. Nevertheless, the extension is a powerful tool for most debugging needs.

Installation

To install the CPython GDB extension, you need to do the following:

1. Compile Python from source. This will produce the file “python-gdb.py” at the root of the Python source tree.
2. Tell GDB where to find extension modules.
3. Put the generated extension from step 1 where GDB can find it.

To do step 1, simply compile Python as you normally would. This is covered extensively by Python, so I won’t go into the details here.

For step 2, you need to pick a place (or places) for GDB extension modules to live. I just created the directory “~/.gdb” for this purpose. To tell GDB to look here for extensions, what you really need to do is extend sys.path in GDB’s embedded Python interpreter. For example, I have this code in my .gdbinit:

 python
 import sys
 sys.path.insert(0, '/home/abingham/.gdb')
 end

This effectively puts GDB int python mode and then extends sys.path to look in ~/.gdb.

Finally, to activate the extension module – that is, to make it available in a particular GDB session – you need to import it. You do that like this:

 (gdb) Python
 >import python3_2_gdb
 >(gdb)

(Again, remember that you exit python mode and return to the GDB prompt with Ctrl-D.)

Once you do this, all of the extension commands (py-list, py-up, etc.) and the frame annotations are activated.

When to load the module

There is one important point to remember with regard to the timing of the extension import. If you attempt to load the extension before the corresponding Python library has been loaded into memory, you will get errors like this:

 (gdb) python
 >import python3_2_gdb
 >Traceback (most recent call last):
 File "", line 1, in
 File "/home/abingham/.gdb/python3_2_gdb.py", line 52, in
 _type_size_t = gdb.lookup_type('size_t')
 RuntimeError: No type named size_t.
 Error while executing Python code.

Because of this, you generally can not import the CPython extension module in your standard .gdbinit. Rather, you need to load it manually when you need it. In practice this is just fine since you will probably have various versions of the extension that you need at different times (see next section.)

Dealing with multiple Python versions

It’s important to remember that different variants of the Python build will generate different extensions. This is because the variants (e.g. debug and non-debug) will, in general, have different C-level structure sizes. As a result, an extension built for debug Python will not work (or may work incorrectly) for release versions of Python. One way to cope with this is simply to name the different versions of the extension differently. For example, the extension for the normal build of Python 3.2 might be “python3_2_gdb.py” while the debug version might be “python3_2_debug_gdb.py”. It doesn’t really matter what naming scheme you use, so just find one that maps to your personal taste.

After unpacking the source, first configure the build with the correct version of python:

PYTHON=python3.2 ./configure

It’s critical that you use the full python version, not just e.g. “python3″. The build determines where to install stuff based on this string, and “python3″ was putting code outside of my sys.path.

Then, I had to run ‘make’ to sort of “wake up” the autoconf system. That stuff is all a big mystery to me, for the most part, but I’ve learned to work around it.

Any, after the initial build I had to replace all uses of “site-packages” in the source with “dist-packages”:

grin -l site-packages | xargs sed -i -e 's/site-packages/dist-packages/'

This again had to do with sys.path, which doesn’t include site-packages by default. Whatever. With that done, I did a “make clean” followed by another “make”. Finally, I did normal “make install” put everything in the right place.

Here the sum of the operations:

tar zxvf dbus-python-1.0.0.tar.gz
cd dbus-python-1.0.0
PYTHON=python3.2 ./configure
make
grin -l site-packages | xargs sed -i -e 's/site-packages/dist-packages/'
make clean
make
sudo make install


1. You provide a list of keywords
2. lazy-slides finds pictures from the internet matching those keywords
3. lazy-slides creates a presentation/slideshow from those pictures
4. You present!

Right now it’s a bit rough. The only image source I’ve coded up is for flickr (if you don’t count the dummy test source.) Also, the only output backend it supports is beamer. In both cases, it should be relatively straightforward to extend lazy-slides.

This release is focused on the logging module. It addresses a number of design issue and defects, and the sphinx documentation for logging is much more complete. It even includes example code for much of the module.

As many people have pointed out before, emacs does not come with any packages for managing software projects. It comes with almost all of the parts you would need – debugger support, compilation modes, etc. – but it doesn’t come bundled with any way to tie a bunch of files together in to a project. Certainly this isn’t required to do software development, and, just as certainly, there are a myriad of ways to approach the concept of “projects”. This latter issue is likely why the basic emacs installation side-steps the issue.

In any case, I recently decided to look into the available project management options for emacs. I’ve used enough other big IDEs and tools to know that projects can be really useful, so I was curious what I could get for emacs. I wanted one that didn’t disrupt my normal emacs flow, one that had a clear and explicit model of what the project comprises, and ideally one that was highly hackable.

I looked at a number of tools, include CEDET (probably good but too bulky-feeling), proel (nice, but I didn’t fit it’s model of the world), and mk-project (not really sure…just didn’t work for me.) Honestly, I didn’t spend a whole lot of time vetting these. My sense was that if the package didn’t click with me quickly, then it wasn’t what I was looking for.

The winner: eproject

Ultimately, I settled on eproject. eproject essentially comprises three concepts:

• a top-level directory for a project
• a collection of files in the project
• a collection of commands that can be run relative to the project directory

A project is simply stored in a file in the project directory, and there is a central collection of projects stored in a file in your .emacs directory. It’s a very simple model, and everything is easily accessible.

I won’t go into any details on how to use eproject; the project page and emacswiki do a fine job of that, and I don’t really have anything to add. I do suggest, however, that you run eproject through its paces a bit on some smaller project just to get a feel for it. All in all, though, I found it intuitive, well documented, and easy to work with.

NOTE: There is another emacs project package also named eproject. The name conflict is unfortunate and confusing, so be aware. I did try this second eproject in my search, and while it’s a fine piece of software it didn’t really fit my needs.

Using eproject with anything

Many emacs users are fans of the excellent anything, and for good reason: it’s awesome! Fortunately, Daniel Hackney (good programmer name) wrote anything-eproject which marries the two projects. anything-eproject allows you to access your projects and project contents via the anything interface by providing two new “sources”.

Making it work

Putting this all together in your emacs configuration is pretty simple. Just download the eproject, anything, and anything-eproject packages to your .emacs directory and add something like this to your configuration:

; load eproject
(load-file "~/.emacs.d/eproject/eproject.el")

; Disable automatic addition/removal of files from projects. Optional.
(setq prj-autotracking nil)

; anything + eproject integration
(require 'anything-eproject)

; let project files be sources in anything
(add-to-list 'anything-sources 'anything-c-source-eproject-files t)

; let projects be sources in anything
(add-to-list 'anything-sources 'anything-c-source-eproject-projects t)

Future work

While eproject works well, there are some things I would like to add. First, it can be a bit cumbersome to add a lot of files to a project, especially if the files in your project are spread across many directories. I’ve developed a small extension to eproject that tries to address this, and I’m working with the author to get this change in.

Second, I’d like support for “cloning” projects. I often work with multiple versions of very similar (often identical) projects, and it would be nice if eproject would let me bootstrap a new project definition based on an old one. This should be relatively simple, I think.

It wasn’t.

So, after a few abortive attempts to install packages I found on the web and such, I determined that what I really needed to do was just build gcc from source. I’ve done this a number of times in the past, so I knew what I was getting into, and by-and-large the process was very straightforward. I had to patch the gcc build files in a few places, though, so I figured I should document the process to save me and others time in the future.

NOTE: This is on an amd64 machine, building a compiler for the same platform. YMMV.

Install a few packages

For the build to work, you’ll need to install a few other packages that provide headers files, etc. The packages you’ll need are:

• linux-header-<kernel-version>
• gcc-multilib

(There may be others, and I’ll try to update this list as I hear about them.)

Make MULTILIB look in the right directories

In gcc/config/i386/t-linux64 you need to correct the directories used for MULTILIB stuff. Change line 9 from:
MULTILIB_OSDIRNAMES = ../lib64 ../lib
to
MULTILIB_OSDIRNAMES = ../lib ../lib32

Change the “version” string that the build expects from ld

The gcc build parses the output of “ld –version”. Unfortunately, ubuntu decided to change that output, causing gcc to think that the ld version is really old. In libstdc++-v3/configure change line 8284 from:
sed -e 's/GNU ld version \([0-9.][0-9.]*\).*/\1/'`
to
sed -e 's/GNU ld (GNU Binutils for Ubuntu) \([0-9.][0-9.]*\).*/\1/'`

The full patch

You can download the patch from pastebin.

Be aware that this patch is only against gcc-core and gcc-g++, not against the entire gcc suite.

Step-by-step

These steps should more-or-less do everything you need to download, patch, build, and install gcc/g++-4.1. Everything will get installed to /usr/local/gcc-4.1/[bin,lib], and the programs will have the suffix “-4.1″.

wget ftp://ftp.nluug.nl/mirror/languages/gcc/releases/gcc-4.1.2/gcc-core-4.1.2.tar.bz2
wget ftp://ftp.nluug.nl/mirror/languages/gcc/releases/gcc-4.1.2/gcc-g++-4.1.2.tar.bz2
tar jxvf gcc-core-4.1.2.tar.bz2
tar jxvf gcc-g++-4.1.2.tar.bz2
<download patch from pastebin>
cd gcc-4.1.2
patch -p1 < ../FKkiciZz.txt
./configure --prefix=/usr/local/gcc-4.1 --program-suffix=-4.1
make
make install
...set up symlinks to /usr/local/gcc-4.1/bin/g++-4.1, etc...

Thialfi aims to emulate that feature. For example, I want to be able to send an email to Thialfi (using a custom address for my account, which only accepts emails from addresses I register) with a subject like “Remind me to top off the gas in the yacht next Wednesday at 5pm”. Next Wednesday, a little before 5, Thialfi will send me an email saying “Top off the gas in the yacht.” There are refinements and extensions to the idea (repeated events, SMS), but that’s the crux of it.

When Sandy went dark, some of its users talked about creating a replacement, but as far as I know none ever materialized. There are some other similar services I’ve tried, but none of them had the sheer simplicity and effectiveness (never mind the personality) of Sandy. That plus the fact that it’s an interesting project led me to start Thialfi. There’s not much there yet, and part of the curve will be learning more about Google’s AppEngine and web_app, but hopefully I’ll keep motivated long enough to create something really useful.

You can follow the action (or join in!) at the google code page.

* Thialfi is, by some estimations, the messenger of the gods in Norse mythology.

1. Immutability: users shouldn’t be able to modify the value at runtime
2. Simplicity of definition: They should be easy to define
3. Uniqueness: Two enum values aren’t equal unless they are the same, no matter what their numeric value is
4. Iteration: The user can iterate over the values in the enum
5. Pickle-able: Naturally…

Here’s what I came up with.

It uses a metaclass to add a custom __setattr__, __getattr__, and __iter__ to the user’s “enum class”, and it looks like this:

import enum

class Level(metaclass=enum.Enum):
  LOW=1
  MEDIUM=2
  HIGH=3
  
x = Level.LOW
Level.HIGH = 19 # Throws an AttributeError
etc...

I don’t know how or if linseed will continue to grow. I’m using it as a bit of a platform for learning things about distributing python, and especially using some of the new extension/plugin systems, so that may result in changes. Also, linseed is linux-only right now, so perhaps someone will decide that it needs to be extended for windows or something.