7DRL is almost upon me. I've decided to start its 168 hours as late as possible: today (Sunday), shortly before midnight. Until then, I plan to work on getting my PlomRogue engine into shape. This task got quite interesting through a recent decision I made: rewriting PlomRogue (or at least major parts of its server component) in Python.

Why?

I promised to hack new features into PlomRogue as needed during the 7DRL. But I didn't trust my code to be flexible enough for that: There'd be too much low-level C micro-management to handle – in any case, too much to add lots of new stuff on short notice. Adding features would go faster in a higher level language.

Among such languages, I chose Python (Python 3, for those who care) for two reasons mostly:

Firstly: For some time now, I wanted to refresh my memory of it. I had learned (small parts of) it over a decade ago. But since then, I had forgotten most of what I had learned. Porting PlomRogue to Python would be a great re-learning opportunity.

Secondly: As a scripting language, Python may run slower than compiled C code. But it's said to play well with the latter; that C code compiles easily to libraries Python can talk to. So I would not have to throw out all C code at once: I could keep some of it as libraries to use in my Python code – especially performance-critical inner-loop stuff.

Porting work

I've spent much of the past two weeks re-writing the PlomRogue server in Python.

At first, I carefully re-thought my engine's mechanisms – looking for ways to optimize them through Python's features. One example: One great weakness of C is text / string handling. In this area, hundreds of lines of C became very few lines in Python. It's a great joy to see my code size dwindle in that way. I'm more open now to using ready-made solutions such as Python's (which has a great many of them included in its standard library). I know now how to write their low-level basics, so they don't strike me as scarily magical as they did when I started my engine.

Later, as 7DRL came nearer, I got a bit less careful. I began to copy some algorithms almost line by line from C to Python – without trying to re-think them. "No time for research or experiments now", I told myself: "I can simplify these algorithms later, after the 7DRL." At this stage, code got less "pythonic", and uglier (not helped by my lack of Python experience). It was shorter (and mostly clearer) than the original C code. But I know that there's still lots of room for improvement …

Finally, only some hundreds of lines of C code remained unported. I threw all of them into one file, libplomrogue.c. I compiled that file into a shared library, like this (all kinds of error-testing / debugging flags removed for readibility; the full-length line may be found here):

$ gcc -shared -fPIC -o libplomrogue.so libplomrogue.c 

(The "-shared" flag obviously tells gcc to build a shared library. For explanations of the "-fPIC" flag, see David A Wheeler's "Program Library HOWTO".)

From Python, I could now (by way of the ctypes module) access all of libplomrogue.so's extern functions like this:

>>> import ctypes 
>>> libpr = ctypes.cdll.LoadLibrary("./libplomrogue.so") 
>>> libpr.rrand() # Example extern function: My pseudo-random numbers generator. 
23 

Testing, fixing, optimizing

This last step allowed me to mostly wrap up my engine re-construction – two nights ago.

Since then, I've been testing what I built, fixing bugs. I mostly compared the Python code's results to the C code's results: running the Python-scripted server next to the one compiled from C. Both basically read the same input (as text appended to the input file server/in read by them) – and should generate the same output from it (as game state save files, etc.) All differences in output should be accounted for (by, for example, slightly different save file formats). When I could not explain differences away, I knew I had a bug. Some bugs were programming errors in my Python code. In other cases, my Python code seemed correct – and I found a bug in my old C code.

After lots of testing and fixing, I'm now mostly content with my new version. It's still much slower than the compiled one. But I'm hopeful that I can optimize the critical parts. If I can't come up with better algorithms for them, I may just compile them into libplomrogue.so. Identifying such bottlenecks seems easy with Python: I just have to run my server wrapped in Python's cProfile module, like this:

$ python3 -m cProfile plomrogue-server.py 

Once the server process ends, this prints a nice statistic: How much processing time was spent in which function? Midnight is still two hours away. Maybe I can attack one or two of those worst offenders until then …