Python have a great number of cool language features and one of them is the with statement. The with statement can be used in numerous ways but I think that the most common usage is to track and release some kind of resource and at the same time encapsulate the code block with a try/finally clause.

For example:

with open('my_file.txt') as f:
    for line in f:
        print line

which could be translated to something like this:

f = open('my_file.txt')
try:
    for line in f:
        print line
finally:
    f.close()

and in a general form:

with <EXPRESSION> [as <VAR>]:
    <BLOCK>

The with statement guarantees that the file will be closed no matter what happens, for example an exception is thrown or a return is executed inside the for statement etc. The result from the expression that is evaluated after the with keyword will be bound to the variable specified after the as keyword. The as part is optional and I’ll show you a use case later.

But wait a minute, how does the with keyword know how to close the file? Well, the magic is called ‘Context Manager‘.

A context manager is a class which defines two methods, __enter__() and __exit__(). The expression in the with statement must evaluate to a context manager else you’ll get an AttributeError. The __enter__ method is called when the with statement is evaluated and should return the value which should be assigned to the variable specified after the as keyword. After the block has finished executing the __exit__ method is called. The __exit__ method will be passed any exception occurred while executing the block and can decide if the exception should be muted by returning True or be propagated to the caller by returning False.

Let’s look at the following example (not a good design, just an example):

class TransactionContextManager(object):
    def __init__(self, obj):
        self.obj = obj
 
    def __enter__(self):
        self.obj.begin()
        return self.obj
 
    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type:
            self.obj.rollback()
        else:
            self.obj.commit()
        return False
 
class Transaction(object):
    def begin(self):
        print "Start transaction"
 
    def query(self, sql):
        print "SQL:", sql
 
    def rollback(self):
        print "Rollback transaction"
 
    def commit(self):
        print "Commit transaction"
 
print "Before 'with'"
 
with TransactionContextManager(Transaction()) as tx:
    tx.query("SELECT * FROM users")
 
print "After 'with'"

The code produces the following output:

Before 'with'
Start transaction
SQL: SELECT * FROM users
Commit transaction
After 'with'

It’s easy to trace the execution but I’ll explain it anyway.

1. The code starts by printing “Before ‘with’”.
2. The with statement is executed and instances of TransactionContextManager and Transaction are created. The context manager’s __enter__ method is invoked which will invoke the Transaction instance’s begin method. The begin method prints “Start transaction” and finally the __enter__ method returns the Transaction instance.
3. The return value from the context manager’s __enter__ method is bound to the variable tx.
4. The block is executed and the query method is called.
5. The query method prints “SQL: SELECT * FROM users”.
6. The block is done executing and the context manager’s __exit__ method is invoked. Since no exception occurred the commit method is invoked on the Transaction instance. The __exit__ method returns False which means that any exceptions should be propagated to the caller, none in this case.
7. The code ends executing by printing “After ‘with’”.

Let’s change the Transaction class so that it throws an exception from the query method. This should execute the rollback method instead of the commit method. The exception should also be propagated to the caller since we return False from the __exit__ method.

def query(self, sql):
    raise Exception("SQL Error occurred")

And the output looks like following:

Before 'with'
Start transaction
Rollback transaction
Traceback (most recent call last):
  File "transaction.py", line 32, in <module>
    tx.query("SELECT * FROM users")
  File "transaction.py", line 21, in query
    raise Exception("SQL Error occurred")
Exception: SQL Error occurred

Here we see that the rollback method is called instead of commit. The rollback method is called because the code inside the with block raised an exception and a valid exc_type argument was passed to the __exit__ method. We can also see that the exception is propagated to the caller because we return False from the __exit__ method.

If we change the __exit__ method to return True instead of False we get the following output:

Before 'with'
Start transaction
Rollback transaction
After 'with'

As expected, we don’t see the exception anymore. Note that the __exit__ method is not called if an exception occurs before the with statement starts executing the block.

Alright, I hope you agree with me that the with statement is very useful. Now I’ll show you a final example where we don’t need to bind the return value from the context manager to a variable. I’ve re-factored the previous example and hopefully made a better design.

class Connection(object):
    def __init__(self):
        # Default behaviour is to do auto commits
        self._auto_commit = True
 
    def connect(self):
        pass
 
    def auto_commit(self):
        return self._auto_commit
 
    def set_auto_commit(self, mode):
        print "Auto commit:", mode
        self._auto_commit = mode
 
    def rollback(self):
        print "Rollback transaction"
 
    def commit(self):
        print "Commit transaction"
 
    def executeQuery(self, sql):
        # Handle auto commit here if it's enabled
        print "SQL:", sql
 
class Transaction(object):
    def __init__(self, conn):
        self.conn = conn
        self.auto_commit = False
 
    def __enter__(self):
        self.auto_commit = self.conn.auto_commit()
        self.conn.set_auto_commit(False)
        return self.conn
 
    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type:
            self.conn.rollback()
        else:
            self.conn.commit()
 
        self.conn.set_auto_commit(self.auto_commit)
        return False
 
conn = Connection()
conn.connect()
 
with Transaction(conn):
    conn.executeQuery("SELECT a user FROM users")
    conn.executeQuery("INSERT some data INTO users")

The output:

Auto commit: False
SQL: SELECT a user FROM users
SQL: INSERT some data INTO users
Commit transaction
Auto commit: True

As the example shows, we don’t need to use the return value from the context manager to still have use of them. We only want to be sure that the queries inside the with block is executed in the same transaction and rollbacked if an exception occurs or commited if successful. Without the with statement the code would be implemented something like the following snippet:

conn = Connection()
conn.connect()
auto_comm = conn.auto_commit()
 
try:
    conn.set_auto_commit(False)
    conn.executeQuery("SELECT a user FROM users")
    conn.executeQuery("INSERT some data INTO users")
    conn.commit()
except:
    conn.rollback()
    raise
finally:
    conn.set_auto_commit(auto_comm)

Well, for me the with statement seems more clean and convenient to use than the last example.

Writing unit tests in python is really easy. It’s the dynamic properties of the language which makes it very simple to mock class methods, instance methods and functions. This makes the task of writing isolated unit tests easy when you practice test-driven development.

Say you have the following class:

import urllib2

class UrlDownloader(object):
    def download(self, url):
        """
        Read the content from the url.
        The content is returned as a string object.
        """
        return urllib2.urlopen(url).read()

You actually don’t want to download something from the net when you do a unit test for this class. You would like to have a fixed string returned from the read() method. This can be hard to achieve in some languages (for example Java) but that’s not true for Python. Python supports redefining class methods, functions and other things in run-time. The following snippet shows how easy it’s to mock the urlopen() function in the urllib2 module to return a mocked file-like object:

import urllib2
import unittest

class MockFileObject(object):
    def read(self):
        return "DATA"

def mock_open(url):
    return MockFileObject()

class UrlDownloaderTest(unittest.TestCase):
    def testDownload(self):
        urllib2.urlopen = mock_open
        dl = UrlDownloader()
        self.assertEqual(dl.download(""), "DATA")

if __name__ == '__main__':
    unittest.main()

In this unit test we redefine the urlopen-function (remember, this is not a member of an instance, it’s a plain function) with our own which returns a mock file-like object. It’s not that hard to imagen all the possibilities this gives when writing tests.

There are of course frameworks that will help you with creating mocks etc, you could try The Python Mock Module which should cover most of the use cases.

ELCE 2009 is now at an end and we must say we are very satisfied with this year’s conference. We left with lots of inspiration and some concrete ideas for future projects. The major focus of the conference was on building systems and improving boot time  and we learned alot, especially about tools for measuring boot time and performance bottlenecks.

Have a look at session slides and videos on the ELCE 2009 homepage (as soon as they become available).

The picture above shows Adeneos demo of a Linux system booting a 3D Qt-application in less than five seconds. Both Adeneo and Free-electrons had sessions on improving the boot time of Linux, where many useful techniques and tools where presented.

In summary, we are very satisfied with this years conference and will most likely attend next year again. See you there!

On the first day of the conference we (Mario Boikov and Tomas Nilsson) attended sessions about porting Linux (keynote), how to boot a Linux system in less than 5 seconds, OpenEmbedded, OpenWRT and other interesting topics. We specially liked the OpenWRT presentation and we both felt like going back to the hotel room and start hacking after the presenter did a live demo. We also got eager to try out OpenEmbedded to create a Linux system.

Tomorrow we’ll attend sessions on boot reduction techniques, U-boot v2, state of real-time Linux and more.

Realway will send two of our employees to the Embedded Linux Conference Europe 2009 held in Grenoble, France (October 15-16). There will be a lot of talks about build systems, the kernel and other low-level stuff. We are looking forward to visit this conference and meet other Linux enthusiasts.

A Python file-like object can be seen as an abstraction to input/output streams. They are actually similar to Java’s java.io.Reader/java.io.Writer abstraction (I’m coming from Java). The typical usage of file-like objects can be found in frameworks and APIs. The file-like object provides a generic way of reading/writing data and would fit an HTML-parser for example. The parser is not interested of where the data come from (files, sockets, etc), only the content, so naturally the implementation should not depend on how to open resources etc.

Basic methods that need to be implemented by a file-like object are:

• read([size]) – read at most size bytes from the file.
• readLine([size]) – read one entire line from the file.
• write(str) – write a string to the file.
• close() – close the file.

You should read the documentation of the function/method that uses the file-like object, often they specify which methods that are expected to be implemented by the file-like object. If the file-like object is only used for reading, no write-functions need to be supported. More information about file-like objects can be found here.

Ok, so why should you use file-like objects in your code?

Let’s take a look at the ConfigParser class which can be used to read/write configuration data. You can expect to find methods for reading and writing configuration data using files but the class does also include support for file-like objects, which is great. For example it’s very easy to write unittests for this class, thanks to the support of file-like objects,  without having to create real files with test data. To create a file-like object from a string we can use the StringIO module. The module have both read and write support.

Here’s some code:

import unittest
import StringIO
import ConfigParser

class ConfigParserTest(unittest.TestCase):

    def testHasSection(self):
        """
        has_section() should return True if the section exists
        and False if is doesn't.
        """
        data = StringIO.StringIO("[My section]\nSetting=1\n")
        config = ConfigParser.ConfigParser()
        config.readfp(data)
        self.assertTrue(config.has_section("My section"))
        self.assertFalse(config.has_section("Not a section"))

    def testMissingHeader(self):
        """
        readfp() should raise MissingSectionHeaderError
        when parsing a file without section headers.
        """
        data = StringIO.StringIO("\nSetting=1\n")

        config = ConfigParser.ConfigParser()
        self.assertRaises(ConfigParser.MissingSectionHeaderError,
                          config.readfp, data)

if __name__ == "__main__":
    unittest.main()

And a simple write configuration test:

import unittest
import StringIO
import ConfigParser

class WriteConfigTest(unittest.TestCase):

    def testWriteConfig(self):
        """
        Test write a simple config
        """
        data = StringIO.StringIO()
        config = ConfigParser.ConfigParser()
        config.add_section("My Section")
        config.set("My Section", "my_option", "my_value")
        config.write(data)

        expected = "[My Section]\nmy_option = my_value\n\n"
        self.assertEquals(expected, data.getvalue())

if __name__ == "__main__":
    unittest.main()

These unittests are of course only examples and do not fully test the ConfigParser class.

It’s easy to see that file-like objects are very handy and fits well in frameworks and APIs. There are of course many advantages of using an abstraction when reading/writing data, some of them are:

• Implementations do not depend on real files, socket, etc, only on a file-like object.
• Easier to fake data in unittests.
• Caller responsible of opening/closing the resource.
• Less error-handling in framework code, delegate exceptions to the caller.

The downside with the current file-like object abstraction is that it’s tailored for text I/O. In Python 2.6 and 3.x a new module have been added named  io. The module originates from the New I/O PEP. The module enriches Python with an three-layer solution, the raw I/O layer, the buffered I/O layer and finally the text I/O layer. The raw and buffered layers deals with units of bytes and the text layer deals with units of characters. There are some things you should be aware of if you use the new I/O library in Python 2.6, read more about it here.

I’ve implemented a shoutcast playlist parser, plsparser, using the ConfigParser class. The parser is implemented as a Python generator function. Actually, I might do a post about generator functions someday.

This is the second blog post in a series of posts on embedded Linux development.

Goal

In this second post the goal is to create and boot an initial root file system, initrd, ending with the execution of the first user-space process; the /init or /linuxrc process. To keep the blog post relatively small, the init/linuxrc-process will not continue the normal system setup but instead end with printing a friendly “hello world!”. The installation of a full embedded Linux system will be covered in a future blog post.

Prerequisites

• Ubuntu Linux 9.04
• The completion of Getting started with embedded Linux development – Step 1: Compiling the Linux kernel or a similar system setup, including kernel source code and cross-compiler.

I’m going to use a computer running Ubuntu 9.04 as host during this tutorial, mainly because it is a modern and stable distrubution with a large community. It is perfectly fine to use another Linux distribution if you so desire, but be aware that some commands might not be identical on all distributions.

The Linux boot process

The Linux boot process goes through a number of steps as seen in Figure 1 below:

• At system startup, the processor starts executing at a predetermined address, usually the Bios on a PC or flash/ROM on an embedded system. The purpose of this first piece of code is to locate a suitable device to boot. After that device has been found, the stage 1 bootloader is loaded into memory and executed.

• The stage 1 bootloader is a very small program (less than 512 bytes) with the sole purpose of loading and executing the stage 2 bootloader.

• The stage 2 bootloader is responsible for loading the Linux kernel and optional initial ramdisk into memory.

  • The initial ramdisk (initrd) is a temporary root file system that holds modules and device drivers that can be loaded by the kernel on-demand instead of having to compile them statically into the kernel. This results in a much smaller kernel. There are two different initrd techniques;

    • initrd – A fixed sized gzipped file that is formatted as a block device and thus needs to be mounted to be accessed. When mounted it will be available as a special ramdisk under /dev/ram. It requires ramdisk and file system drivers to be compiled into kernel to work. It is considered an “old and ugly” technique, especially the pivot_root call which switches between the initial and the “real” root file system

    • initramfs – A compressed cpio archive that is loaded as a ramfs and only uses the size of its files in RAM. Requires neither block nor filesystem drivers.

  • The stage 2 bootloader executes the kernel by calling the start assembly routine

• The kernel is decompressed and executed. The initial ramdisk is loaded from RAM and the device is configured. Finally, the real root file system is mounted and the /sbin/init process is invoked. In this tutorial, however, the kernel will only execute the first process on the initrd (the /init process in the initramfs case and /linuxrc in the initrd case) since that process will do nothing more than print Hello world! and then sleep for a very long time. As mentioned before, the following blog post will cover the boot of the full Linux system.

Here is also a more detailed image where the different initrd techniques are compared:

For more information, see IBMs article on Linux boot sequence, from where this article is inspired.

Alternative 1 – Create an initramfs archive

Start by creating a folder to host our init file:

mkdir -pv ~/project/rootfs/mnt


Now, normally the initramfs archive would be populated with the /init file and all drivers necessary to boot the full Linux system. We will, however, only use a mockup /init file which prints Hello world! to prove that it works:

cat > ~/project/rootfs/hello.c << "EOF"
#include <stdio.h>

int main(int argc, char **argv)
{


printf("Hello world!\n");
sleep(999999999);

}
EOF

Compile the file statically using the cross-compiler:

~/project/tools/arm-2009q1/bin/arm-none-linux-gnueabi-gcc -static -o ~/project/rootfs/mnt/init ~/project/rootfs/hello.c

Now, create the cpio archive and move it to ~/project/images

cd ~/project/rootfs/mnt

find . | cpio -ov --format=newc | gzip > ../initramfs.cpio.gz

mv -v ~/project/rootfs/initramfs.cpio.gz ~/project/images

Run the image using QEMU:

qemu-system-arm -kernel ~/project/images/zImage.integrator -initrd ~/project/images/initramfs.cpio.gz

Alternative 2 – Create an initrd image

This alternative assumes you’ve done Alternative 1, especially created the hello.c file.

First we need to create a file to act as a block device. We’ll make it 8 MB large:

dd if=/dev/zero of=$HOME/project/rootfs/initrd-file bs=1M count=8

Next, format it using the ext2 file system:

mke2fs -F -m 0 -b 1024 $HOME/project/rootfs/initrd-file

In order to access the newly formatted initrd file, it must be mounted as a loopback device in the host file system:

rm -rv ~/project/rootfs/mnt/*

sudo mount -t ext2 -o loop ~/project/rootfs/initrd-file ~/project/rootfs/mnt

Now, normally the initrd file would be populated with the /linuxrc file and all drivers necessary to boot the full Linux system. We will, however, use the same hello.c file as in Alternative 1 as source for the /linuxrc file:

~/project/tools/arm-2009q1/bin/arm-none-linux-gnueabi-gcc -static -o ~/project/rootfs/mnt/linuxrc ~/project/rootfs/hello.c

Make it runnable:

chmod 755 ~/project/rootfs/mnt/linuxrc

Create two necessary device files /dev/null and /dev/console:

mkdir -v ~/project/rootfs/mnt/dev

sudo mknod ~/project/rootfs/mnt/dev/null c 1 3

sudo mknod ~/project/rootfs/mnt/dev/console c 5 1


Unmount and compress the ramdisk:

cd ~

sudo umount ~/project/rootfs/mnt

gzip ~/project/rootfs/initrd-file

Move it to ~/project/images and run QEMU:

mv -v ~/project/rootfs/initrd-file.gz ~/project/images

qemu-system-arm -kernel ~/project/images/zImage.integrator -initrd ~/project/images/initrd-file.gz

That is all for today folks! Until next time…

This is the first blog post in a series of posts on embedded Linux development.

Goals

There are two main goals of this first post. The first goal is to set up a working development environment including a cross-compiler and other build tools. The second goal is to compile and run the Linux kernel on a ARM emulator called QEMU. To keep the tutorial relatively small, we will save the boot of a fully working embedded Linux system to the next blog post. This means we will only be seeing the boot of the compiled Linux kernel, ending with the infamous kernel panic in this tutorial. Read the rest of this entry »

Just back from the KHelg (competence weekend) where we had some interesting laborations with the Cell Broadband Engine, and a great seminar with one of the few men in the world that has implemented a fully validated C99 compiler.

The Cell Broadband Engine (commonly known  as the Cell processor), is an interesting beast. It is a Power-based architecture with a twist: it has a primary PPU (Power Processor Unit) and a number of SPU (Synergistic Processing Units). Connected via a superfast interconnect bus, they can carry out enormous amounts of work. You’ll find that Cell processors are popping up everywhere, from supercomputers to gaming consoles. We’ve written some nifty code and really stressed the system – all we need now is a serious problem to solve. If you have any, do give us a call!

As for the C99 seminar – we learnt a lot about what the this new (well, 10 year old) version of the language has to offer and what support you can expect from GCC. The presenter was Mr Skeppstedt of LMPCC, one out of five companies in the world that has a validated C99 compiler. GCC’s support for the different language features is increasing real fast, and in combination with some serious optimizing, GCC is catching up performance-wise as well.

In all, this was yet another great KHelg!

On the morning January 30th, we’ll be giving a free two-part breakfast seminar on interprocess communication on Linux systems. We’ll give you an thorough introduction of and comparison between POSIX message queues and D-Bus

The seminar will be held at our offices, and we’ll even throw in a free breakfast. The only thing you need to do is check out the flyer, in which you can read more about th subjects. You’ll also find out how to register. Welcome!

(Please note that the seminar will be held in Swedish.)