Awasu » Embedding Python: Calling Python code from your program
Sunday 30th November 2014 7:57 AM []

If you're embedding Python into your C/C++ program, it may be because you want it to do stuff that's easier to write in Python rather than C/C++. This means we need to be able to push data into the Python code, for processing, then be able to get the result back.

In this tutorial, we'll take a look at how to define a Python function, call it with some parameters, and get a result back.

Defining a Python function

We'll start off by defining a simple Python function that adds 2 numbers and returns the result.

def add( n1 , n2 ) :
    return n1+n2

The first thing we need to do is to get this definition into our embedded Python interpreter, then get Python to somehow give the function back to us, so that we can call it.

void
main( int argc , char** argv )
{
    // initialize Python
    Py_Initialize() ;
    
    // compile our function
    stringstream buf ;
    buf << "def add( n1 , n2 ) :" << endl
        << "    return n1+n2" << endl ;
    PyObject* pCompiledFn = Py_CompileString( buf.str().c_str() , "" , Py_file_input ) ;
    assert( pCompiledFn != NULL ) ;
         
    // create a module        
    PyObject* pModule = PyImport_ExecCodeModule( "test" , pCompiledFn ) ;
    assert( pModule != NULL ) ;
     
    // locate the "add" function (it's an attribute of the module)
    PyObject* pAddFn = PyObject_GetAttrString( pModule , "add" ) ;
    assert( pAddFn != NULL ) ;
     
    // clean up
    Py_DecRef( pAddFn ) ;
    Py_DecRef( pModule ) ;
    Py_DecRef( pCompiledFn ) ;
    Py_Finalize() ;
}

Everything in Python is an object, so when we define our function, Python gives us back an object that represents the compiled code. We then load that code into a module, and Python gives us another object that represents the module. Functions are attributes of their parent module, so we can locate our function by looking for an attribute called add, and since functions are also, you guessed it, objects, we get back an object that represents our function.

When Python gives us these objects, it also increments the reference count on them, since it has no way of knowing when we're going to be finished using them. So, it's important we decrement the reference count on them when we're done, otherwise they would leak.

Throughout the Python code, managing reference counts is done using the Py_INCREF and Py_DECREF macros, but using these in external code is dangerous, because their definitions depend on certain compile-time settings[1]For example, if Py_REF_DEBUG is defined, extra code is added to help debug reference counts., so if your compile-time settings are not the same, you will be using a different definition of these macros to what the Python interpreter is using, and odd things will surely happen. It's safer to use the Py_IncRef() and Py_DecRef() functions, which simply invoke the macros.

Calling the Python function

We now have an object that references our function, so we can now call it, but first, we have to figure out how to pass the 2 parameters in.

Python functions accept two types of parameter, positional and keyword, which can be accessed via the wonderfully-named splat and double-splat operators. The following code:

def foo( *args , **kwargs ) :
    print "args =" , args
    print "kwargs =" , kwargs 
    
foo( 1 , 2 , 3 , foo="bar" )

produces the following output:

    args = (1, 2, 3)
    kwargs = {'foo': 'bar'}

This is not a specially-added feature of Python, it's a side-effect of how parameters are internally passed into functions. The Python interpreter always calls functions with exactly 2 parameters: a tuple of values that represent the positional arguments, and a dictionary of values that represent the keyword arguments.

Since our function takes 2 positional arguments, we need to set up a tuple to hold them. We'll take the 2 values to add from the command-line arguments.

// create a new tuple with 2 elements
PyObject* pPosArgs = PyTuple_New( 2 ) ;

// convert the first command-line argument to an int, then put it into the tuple
PyObject* pVal1 = PyInt_FromString( argv[1] , NULL , 10 ) ;
assert( pVal1 != NULL ) ;
int rc = PyTuple_SetItem( pPosArgs , 0 , pVal1 ) ; // nb: tuple position 0
assert( rc == 0 ) ;

// convert the second command-line argument to an int, then put it into the tuple
PyObject* pVal2 = PyInt_FromString( argv[2] , NULL , 10 ) ;
assert( pVal2 != NULL ) ;
rc = PyTuple_SetItem( pPosArgs , 1 , pVal2 ) ; // nb: tuple position 1
assert( rc == 0 ) ;

While PyInt_FromString() returns a Python object to us that needs to be cleaned up, when we add it to the tuple, the tuple steals our reference to the object and will decrement the reference count when it gets destroyed. This means we don't need to do it, and if we do, Python will report an error later when it notices that an object has had its reference count decremented too many times.

We also need to set up a dictionary for the keyword arguments, even though our function doesn't accept any.

// create a new dictionary 
PyObject* pKywdArgs = PyDict_New() ;
assert( pKywdArgs != NULL ) ;

We can now call our function.

// call our function 
PyObject* pResult = PyObject_Call( pAddFn , pPosArgs , pKywdArgs ) ;
assert( pResult != NULL ) ;

If you run the program now, you won't see anything happen, but if you add a print statement to the definition of add(), you will see that the function is indeed being called, with the correct parameters.

Returning values from Python

Getting the result back from the function is much easier 🙂 We can see that PyObject_Call() returns a Python object, which is the function's return value. It's just a raw Python object and while in this case, we know it's an integer, we may not always know it's type, so the easiest way to examine its contents is to repr it.

// convert the result to a string 
PyObject* pResultRepr = PyObject_Repr( pResult ) ;
cout << "The answer: " << PyString_AsString(pResultRepr) << endl ;

Note that Python functions always return a value; even if you just run off the end of the function and don't return anything, the caller will receive None (try it).

We can now see our code in action:

Download the source code here.


   [ + ]

1. For example, if Py_REF_DEBUG is defined, extra code is added to help debug reference counts.

2 Responses to this post

THank you very much! Best tutorial on python embedding.

No worries, glad it was helpful.

Have your say