Supercharging C++ Code With Embedded Python – EuroPython 2012 Talk

This is the talk that I gave at EuroPython 2012 in Florence, Italy. It was a 60-minute talk, so it’s light on technical details. I am planning to publish follow-up articles that provide step-by-step instructions along with complete code examples. The first part of the tutorial is available. If you want to know when the next part will become available, subscribe to the RSS or add me on Google+ or on Twitter.

(Watch on YouTube.)

You can download the slides in PDF format and the slide sources in SVG format. (Please note the licensing restrictions.)

BTW, our team is hiring. If you’re interested in extending/embedding Python, or just interested in Python in general, you should definitely get in touch with us. Benefits of the position include an agile development process, a variety of projects to work on…and a beach within walking distance.

About Me / About SPIELO

I work as a software architect in the mathematics department at SPIELO International in Graz, Austria.

SPIELO International designs, manufactures and distributes cabinets, games, central systems and associated software for diverse gaming segments, including distributed government-sponsored markets and commercial casino markets.

Our team is responsible for the mathematical game engine that controls all payout-relevant aspects of the game. Part of the engine is an embedded Python interpreter.

Embedded Python: What is it? When to use it?

This talk is about embedding the Python interpreter in a C/C++ program and using the Python/C API to run Python scripts inside the program.

Image may be NSFW.
Clik here to view.

Here are some examples of what you can do with this technique.

Plug-in/extension language. This is the “Microsoft Word macro” use case, in which users can extend the functionality of the program by writing their own scripts. Let’s say a users wants to apply random formatting to each word in the text. A simple macro does the trick without requiring you to add a feature that most users don’t need:

Image may be NSFW.
Clik here to view.

Test automation. Automated testing exercises a program’s functionality by following pre-defined steps. That’s not much different from the macro scenario. Throw in a few “assert” statements and you have a test case. For testing purposes, the embedded Python scripts might have access to functionality that would be unsafe or useless for macros.

Game engines. Video games have been using built-in scripting languages for a long time. While performance-critical parts like graphics, physics, and low-level AI are written in C++, the scripts control things like high-level enemy behavior, map generation, and scripted events. Civilization IV is an example of a game that uses Python for this.

class Guard(Enemy):
    def OnGettingHit(self, actor):
        self.findCover()
        if self.distanceTo(actor) < self.maxShootingDistance:
            self.shootAt(actor)
        else:
            self.team.setAlarmed(True)

All of this can be done directly in C++, of course, but there are certain advantages to using an embedded Python interpreter:

• Ease of use. You can’t expect the users of your program or the level designers on your team to write plug-ins in C++, but many of them will be willing to learn a bit of Python to get their jobs done.
• Sandboxing. Plug-ins written in C++ can do pretty much anything on the computer. This may be a security issue. A stripped-down Python interpreter, on the other hand, provides a restricted execution environment for plug-ins.
• Flexibility. Even if you do know C++, you can try new ideas faster inside a Python script. Also, Python’s reflection capabilities open up new possibilities for automated testing.

Extending Python Recap

Extending and embedding Python are closely related. Extending Python means taking existing C/C++ code and making its data types and functions available to Python programs. Whenever a C/C++ library offers “Python bindings”, it is a Python extension library that uses the Python/C API to plug into the interpreter.

Here’s the big picture.

Image may be NSFW.
Clik here to view.

When the Python code invokes the C++ function “Sum()”, three things need to happen:

• The parameters, 5 and 3.2, need to be converted from Python objects to the “int” objects that the C++ code understands.
• The parameters need to be passed to the Sum() function and the CPU needs to execute the function.
• The return value (8 if you’ve been paying attention) needs to be converted form an “int” to a Python object.

The Python program is compiled to Python opcode and executed by the Python interpreter. The C++ code is compiled to CPU instructions and executed directly by the CPU. Furthermore, what Python sees when we say “5” or “3.2” is very different from what the compiled C++ code expects to see. These are very distinct (and seemingly incompatible) worlds.

Image may be NSFW.
Clik here to view.

Image may be NSFW.
Clik here to view.

We said that the Python and C++ worlds are “seemingly incompatible”, but they are really the same world. At its lowest level, the Python interpreter is also just a C program. When you use Python’s built-in data types or the standard library modules, the interpreter calls C functions sooner or later. For example, here’s the (slightly simplified) C code of the “math.radians()” function that converts an angle from degrees to radians:

static PyObject* math_radians(PyObject* self, PyObject* arg)
{
    double x = PyFloat_AsDouble(arg);
    return PyFloat_FromDouble(x * PI / 180.0);
}

When this function is called in the Python code, for example as “math.radians(3.2)”, the argument is a Python object. The Python interpreter knows how to invoke the C function, as long as the C function accepts a Python object as the argument and returns a Python object as the result. Internally, the C code uses the Python interpreter’s “PyFloat_AsDouble()” and “PyFloat_FromDouble()” functions to convert between Python objects and the C “double” data type.

Back to our “Sum()” function. For the Python interpreter to be able to invoke it, we need to rewrite it to take “PyObject” arguments (in this case a tuple of parameters) and return a “PyObject” result. Better yet, instead of rewriting the original “Sum()”, we can introduce a wrapper around “Sum()” that conforms to the required interface:

static PyObject* WrapSum(PyObject* self, PyObject* args)
{
    PyObject* oa;
    PyObject* ob;
    PyArg_UnpackTuple(args, "pow", 2, 2, &oa, &ob);
    long a = PyInt_AsLong(oa);
    long b = PyInt_AsLong(ob);
    long result = Sum(a, b); // call the original Sum()
    return PyInt_FromLong(result);
}

To turn this into an actual extension module, a module object needs to be created that contains the function that we defined. The following code is an unabridged example of this:

static PyMethodDef MyLibMethods[] =
{
    {"Sum", WrapSum, METH_VARARGS, "Calculate the sum of two integers."},
    {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC initmy_c_lib(void)
{
    (void)Py_InitModule("my_c_lib", MyLibMethods);
}

This code is compiled and linked into a DLL/shared object named “my_c_lib.pyd” on Windows and “my_c_lib.so” on Linux. The resulting library can then be “imported” like any other Python module.

To summarize:

• An extension module is a DLL/shared object.
• The functions inside the library take PyObjects as arguments and return PyObjects as their results.
• The library uses Python’s conversion functions to convert between PyObjects and C data types.
• “import” is used to load an extension module just like any old .py module.

Making Your Life Easier

Writing extension modules in this way is repetitive, somewhat tedious, and error-prone. (And we haven’t even seen any error-checking or memory management code yet.) Here are two ways to simplify the process (I am sure there are more):

• SWIG, the Simplified Wrapper and Interface Generator.This is a tool that parses your C/C++ header files and automatically generates the code for an extension module. Using SWIG is very easy in the simple cases, but it can be a bit fiddly in the complex cases. We have been using it successfully for mission-critical projects at SPIELO and I would recommend it any time.
  • See also the tutorial Python Extensions In C++ Using SWIG on this site.
• Boost.Python is a library that basically wraps the Python/C API in C++ template classes. Compiling Boost.Python code requires a fairly modern C++ compiler and lots of memory and patience during the compilation process. For various reasons, we decided against using it for the projects at SPIELO, but that’s not to say you shouldn’t try it for yourself.

From Extending To Embedding

With an extension module, it is the Python executable that invokes functions in the C++ code when the .py code requests it. The main program is Python and the extension module merely provides services to the Python interpreter.

Image may be NSFW.
Clik here to view.

On the other hand, when you embed the Python interpreter in a C++ program, you are using the interpreter as a library. Just like you would use the Expat library to parse XML files, you can use the Python interpreter as a library to execute Python source code (inside .py files or otherwise).

Image may be NSFW.
Clik here to view.

Whether you are extending or embedding, you will be using the Python/C API in both cases. Embedding usually involves a fair amount of extending as well. After all, the Python code running inside the embedded interpreter will have to call back into the application to be useful. This means that your program will contain the same kind of wrapper functions that we saw in the earlier “Sum()” example for all the objects and functions that you want to make available to the Python plug-ins.

High-Level Embedding

As a first step, here’s the code to initialize the embedded interpreter, execute some Python source code, and shut down the interpreter.

#include <Python.h>
int main(int argc, char* argv[])
{
    Py_Initialize();
    PyRun_SimpleString("name = raw_input('Who are you? ')n"
                       "print 'Hi there, %s!' % namen");
    Py_Finalize();
    return 0;
}

This isn’t terribly exciting yet. You can’t pass any arguments to the Python code and you don’t receive any results. If you don’t go beyond this, it would be easier to just run “python.exe” as a sub-process.

Simple Plug-In

Let’s try a more involved use case. Here’s some C++ program that allows the user to write a Python plug-in to transform a string.

void program()
{
    std::string input;
    std::cout << "Enter string to transform: ";
    std::getline(std::cin, input);
    std::string transformed = CallPythonPlugIn(input);
    std::cout << "The transformed string is: " << transformed.c_str() <<
    std::endl;
}

The magic is supposed to happen inside the “CallPythonPlugIn()” function that we’ll implement in a minute. This function will invoke a function named “transform()” in a user-provided Python file:

# Example "plugin.py"
def transform(s):
    return s.replace("e", "u").upper()

With this, the “CallPythonPlugIn()” function might look something like this. (For brevity, I left out all of the error checking. In other words, don’t use the code as is! I will present a more complete implementation in a follow-up article.)

// WARNING! This code doesn't contain error checks!
std::string CallPythonPlugIn(const std::string& s)
{
    // Import the module "plugin" (from the file "plugin.py")
    PyObject* moduleName = PyString_FromString("plugin");
    PyObject* pluginModule = PyImport_Import(moduleName);
    // Retrieve the "transform()" function from the module.
    PyObject* transformFunc = PyObject_GetAttrString(pluginModule, "transform");
    // Build an argument tuple containing the string.
    PyObject* argsTuple = Py_BuildValue("(s)", s.c_str());
    // Invoke the function, passing the argument tuple.
    PyObject* result = PyObject_CallObject(transformFunc, argsTuple);
    // Convert the result to a std::string.
    std::string resultStr(PyString_AsString(result));
    // Free all temporary Python objects.
    Py_DECREF(moduleName); Py_DECREF(pluginModule); Py_DECREF(transformFunc);
    Py_DECREF(argsTuple); Py_DECREF(result);

    return resultStr;
}

The “CallPythonPlugIn()” function is roughly equivalent to this Python code:

def CallPythonPlugIn(s):
    pluginModule = __import__("plugin")
    transformFunc = getattr(pluginModule, "transform")
    argsTuple = (s,)
    result = transformFunc(*args)
    return result

And that’s the whole secret of embedding Python: Once you know what you’d like the Python interpreter to do, it’s a matter of mapping the Python code to the respective Python/C API functions using the reference docs.

Extending And Embedding Combined

At some point, you will want to access your C++ functionality from the Python plug-ins. For example, you might want to invoke the “Sum()” function that we wrapped earlier:

// Example "plugin.py"
import the_program
def transform(s):
    return the_program.TransformHelper(s).lower()

In this case, we don’t want the module “the_program” to be an extension module in a separate shared object. Instead, the module should live in the program itself so that it has access to the program’s internals.

The C++ function “TransformHelper()” needs to be wrapped using the same techniques that we applied to the “Sum()” function in an earlier example.

static PyObject* WrapTransformHelper(PyObject* self, PyObject* arg)
{
    const char* str = PyString_AsString(arg);
    std::string result = TransformHelper(str);  // invoke the C++ function
    return PyString_FromString(result.c_str());
}

// Register the wrapped functions.
static PyMethodDef TheProgramMethods[] =
{
    {"TransformHelper", WrapTransformHelper, METH_O, "Transforms a string."},
    {NULL, NULL, 0, NULL}
};

// Somewhere in your program, initialize the module. This is all
// that's required to allow the plug-in to run "import the_program".
Py_InitModule("the_program", TheProgramMethods);

With this, the plug-in can invoke the internal functionality of the program. Of course, it is also possible to wrap entire C++ classes and not just functions. This is a topic for a follow-up article.

Summary

Embedding involves these tasks:

• Using the Python/C API to do things that you would normally do in Python. The reference docs help you find the right function to do the job.
• Lots of converting from PyObjects to and from C/C++ data types, just like with extending.
• Wrapping the internal objects of your program so that the embedded Python code has access to them, just like with extending.

C++ Wrappers for the Python/C API

Using a C++ library that wraps the Python/C API offers several advantages:

• Integration with C++ data types such as std::string, iostream, etc.
• Simplified error handling using exceptions
• Avoids memory leaks by taking care of the reference count of PyObjects

We have used two libraries in the past:

• Boost.Python. Makes heavy use of C++ templates.
• PyCXX. Simple and straightforward library.

At SPIELO, we’re currently not using any wrapper library for the embedded interpreter. We do, however, use our own code generators to generate large parts of the most repetitive glue code, which reduces the need for C++ wrappers.

SPIELO Case Study

In our mathematical game engine, embedded Python allows mathematicians and game designers to define game rules that go beyond the pre-defined building blocks that the engine has to offer. The game rules are encoded in a data file that is interpreted by the engine. At certain points in the game flow, Python plug-ins may be invoked to check for additional winning conditions, award special prizes, change the game flow, etc.

The following sections briefly describe certain decisions we made when adding Python to the engine.

Why Python?

Before we added the embedded Python interpreter, our mathematical game engine already had a built-in scripting language. In fact, it was a bytecode interpreter that had to run on an ancient Z80 CPU (one of our target platforms at the time), which meant its functionality was very limited: It had only a single register for storing intermediate results, a 255-byte limit for bytecode programs, no floating-point support, etc.

We had a C-like language on top of the bytecodes. When the limitations of the bytecode interpreter became too much of a burden, we initially considered extending this C-like language with killer features like local variables and sub-routines that would support actual parameter lists and return values.

But how do you design a powerful scripting language that’s easy to learn, readable, and extensible? The answer is, you use an existing language that gets it right, and Python had a lot going for it in this respect:

• The engine team had lots of experience with Python from other projects.
• Most of the users of the engine had Python experience.
• The Python project is mature and has a great community behind it.
• The interpreter is light-weight and highly portable and its license fits our needs.

Integrating Python into the game engine took us just a few weeks.

To Fork Or Not To Fork

Usually, you can embed the Python interpreter that’s already installed on the user’s system by dynamically linking your program to the installed Python DLL/shared object. This allows embedded Python scripts to use all packages that are available in the existing Python installation.

For us, on the other hand, it was not an option to rely on the versions of Python that come pre-packaged for our target platforms. First of all, there is not even an official Python port for some of these platforms (for example, Windows CE). Second, as the Python interpreter is directly responsible for evaluating parts of the game rules, it is subject to the same strict regulations as the rest of the game engine. Therefore, we are treating the Python interpreter as an integral part of the engine: We track its source code in the same repository as the rest of the engine and include it in the testing/release process of the engine.

To make Python compile on some platforms, we had to make changes to the source code. We usually just rip the parts out that don’t work well on all platforms and that we don’t need anyway. These changes are not suitable to be patched into mainline Python, so we essentially created a fork. The fork means that it is unlikely that we’ll upgrade to a newer version of Python any time soon, but it doesn’t really matter to plug-in authors what version of Python they’re using.

Stripping Down And Sandboxing

The Python standard library gives you access to operating system services, internet protocols, graphical user interfaces, and more. Most of this isn’t needed or even desired for a plug-in language.

For the mathematical game engine, we only included built-in modules, i.e., modules that are compiled directly into the interpreter and that don’t require additional .py files to operate. Of those modules, we only included the ones that plug-in authors actually need and that are safe to use. We don’t include things like networking or operating system support, because there’s no reason why the mathematical game engine should mess with the OS, open HTTP servers, or the like. Aside from the security concerns, we don’t want to give plug-in authors (more) opportunities to shoot themselves in the foot.

In a Python interpreter that’s part of, say, a word processor, security is a major concern. You don’t want a plug-in that’s contained in an email attachment to be able to change any files, send any network requests, or run any system commands without the user’s permission. In this case, it’s a good idea to compile your own Python interpreter and leave out all the dangerous bits.

Embedded Debugging

As the saying goes, with great power comes a great danger of bugs.

With a stand-alone Python program, you can step through the code in your favorite Python IDE. With an embedded interpreter, that’s not possible. Still, we wanted to give our users a nice graphical debugger for their plug-ins, integrated into the game editor in a similar way as the VBA Editor is integrated in Microsoft Office.

Our approach uses “PyEval_SetTrace()” to register a function that’s invoked when each source line is executed. This is almost enough to build all kinds of single-stepping (“Step Into”, “Step Over”, “Step Out”) and breakpoints. In addition, you need to be able to retrieve the stack frames (to display a call stack) and to evaluate Python expressions (to display and manipulate variables in the current stack frame and for conditional breakpoints).

A follow-up article will explain this in more detail.

Closing Remarks

Even though Python is awesome, some problems are best solved in C++. But even these C++ programs can be supercharged by adding some Python back in. Hopefully this talk inspires you to build something awesome with an embedded Python interpreter. If you do, I’d love to hear about it.

This article as well as the slide sources in SVG format. are Copyright 2012 Michael Fötsch, licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. The slides in PDF format contain additional material that is Copyright 2012 SPIELO International, All Rights Reserved.