Stromberg und der „Circle of Competence“

Ich schaue mir ab und zu eine Folge Stromberg an. Früher habe ich die Serie kaum ertragen wegen andauerndem Fremdschämen. Entweder bin ich in dieser Hinsicht abgestumpft oder die Serie ist weniger peinlich geworden.

Was mir immer wieder an der Figur Stromberg auffällt ist, dass er das illustriert, was Rolf Dobelli in seinem sehr lesenswerten Buch „Die Kunst des klaren Denkens“ unter Denkfehler #15, „Das Chauffeur-Wissen“, beschreibt:

Warren Buffet […] verwendet einen wunderbaren Begriff: >>Circle of Competence<<. Zu Deutsch: Kompetenzkreis. Was innerhalb des Kreises liegt, versteht man wie ein Profi. Was außerhalb liegt,versteht man nicht oder nur zum Teil. Buffetts Lebensmotto: >>Kennen Sie Ihren Kompetenzkreis, und bleiben Sie darin. Es ist nicht so furchtbar wichtig, wie groß dieser Kreis ist. Aber es ist furchtbar wichtig zu wissen, wo genau die Kreislinie verläuft.<< Charlie Munger doppelt nach: >>Sie müssen herausfinden, wo Ihre Talente liegen. Falls Sie ihr Glück außerhalb Ihres Kompetenzkreises versuchen, werden Sie eine lausige Karriere haben. Ich kann es Ihnen fast garantieren.<<

Stromberg hätte Komiker werden sollen oder Schriftsteller oder etwas ähnliches. In einem solchen Job hätte er sein Talent als Sprücheklopfer und Wichtigtuer voll ausspielen können. In der Versicherung dagegen muss er fortwährend versuchen seine Inkompetenz durch Intrigieren, Delegieren und Arschkriechen zu kompensieren.

Ganz anders liegt die Sache übrigens bei der Hauptperson des amerikanischen Pendants zu Stromberg, „The Office“ (übrigens eine sehr sehr lustige Serie, ich empfehle sie uneingeschränkt): Michael Scott. Dieser ist eigentlich ein äußerst begabter Verkäufer in der Haut eines sozial gestörten Mitt-Vierzigers. Trotz chronischer Zeitverschwendung und sozialer Inkompetenz ist seine Zweigstelle die profitabelste des Unternehmens (wobei man sich jetzt vielleicht darüber streiten kann, ob dies ihm zu verdanken ist).

Update on Reasearch on Ecstatica

Things move slowly forward. This weekend I have continued to implement some Python routines to

  • load an Ecstatica FANT file
  • write an Ecstatica FANT file
  • write a XML representation of the FANT file
  • read a XML representation of the FANT file

Those 4 building blocks are in an early stage right now. At the moment I have verified that I can load a FANT file and write it back without introducing any relevant differences. I’m still having some precision issues with converting the fixed-point numbers to floats and back.

Using these tools I will be able to modify the game data files and see, what happens. I’m thinking about giving the main character green hair or something like that 😀

Blockheads invade the village

Ecstatica fortunately still contains a lot of debug code, which is partly not even called anymore (strangely it was not stripped from the binary).

One of those tiny little debug features is to draw a box around every ellipsoid. The result looks kinda funny ;D

Conflicting target names cause ’nmake test‘ to do nothing

Today I ran into a problem with using Microsoft’s nmake to build a project whose Makefile had been created by CMake. The build process ran fine but when I tried to run the test by calling „nmake test“, nmake just prompted

'test' is up-to-date

(in German: ‚test‘ ist aktuell). I was baffled, because under Linux it worked like a clockwork.

Thanks to Google, I found some IRC chat log where someone had had exactly the same problem and it was actually quite simple: My tests were located in a subdirectory called ‚test‘ and this name interfered with the make target ‚test‘ for running the tests. When Nmake was told to make target ‚test‘, it found the directory ‚test‘ existing already and thus prompted „‚test‘ is up-to-date“.

I changed the directory name to ‚tests‘. After that, calling ’nmake test‘ correctly executed my tests!

Ecstatica copy-protection mechanism (exit after intro)

Just for fun I googled for „Ecstatica no-cd patch“ some weeks ago and found: nothing. You could readily assume that there’s no need, that Ecstatica does not include a protection mechanism. Nothing could be farer away from the truth. You can witness the copy protection mechanism if you simply copy the ecstatica data files from your ecstatica CD onto disk and start it from there. The program will terminate suddenly after the intro. Googling for „ecstatica exit after intro“ does not bring up a solution nor even a explanation for the strange behaviour. It is obviously no crash, because it equally happend in

  • qemu
  • dosbox
  • dosemu
  • bochs

and there never is any error message visible. Well I now found out that there’s actually a protection mechanism at work here. Try this: Start Ecstatica in dosemu from (original) CD  and the program won’t terminate after the intro!

Ecstatica Scene Views

I invested quite some time into figuring out, how to decode the files containing the different camera views for all the scenes in Ecstatica. Well, I finally got somewhere, after some unsuccessfull attempts. At least those attempts made me learn a lot new things about Huffman codes, the LZW algorithm and RL enconding. The amazing game mod wiki helped much again.

The phenotypic facts

I rather quickly found out that:

  1. The scene files are much to small to contain a raw pixel representation.
  2. Changing a byte somewhere in the file sometimes affected a single pixel in the game, sometimes a sequence of pixels
  3. All changes had only local impact, just sometimes the decoding algorithm in Ecstatica seemd to crash and omitted all following pixels. Amazingly this didn’t make Ecstatica crash. Congratulations to the programmers for this stability!

Could it be LZW?

The wikipedia article about the LZW algorithm is quite insightful. After reading it and reconsidering the facts, I came to the conclusion, that it its propably not LZW. Why? As noted above, changes in the pixel values have only local impact. This is also true for the first few pixels. But in my opinion changing values in the first stage of a compressed  LZW data stream should have global impact on the decompressed stream. For example:

Input Output

Could it be RLE?

Well, could it be RLE? I learend a lot about the algorithm from the wikipedia article about the PCX format and the respective modding wiki article. What made me doubt that we have RLE here was, that in my favourite HEX editor, I could see no connection between the most-significant bits and the impact of a value change.

So what is it?

It is some kind of RLE. A special form of RLE: T’here are color bytes and there are RL bytes, like in the standard RLE. The RL bytes are structured in the following way:

  • The 6 most-significant bits are the run-length, let’s call the value L
  • The 2 least-significant bits are flags.

The flags have the following meaning

bit #0 bit #1 meaning
0 0 L pixels. Each pixel’s color is determined using the previous pixel’s color and a 4bit value giving the relative change of color
0 1 L pixels of the same color
1 0 L pixels whose colors are represented absolutely using 1 byte each. If none is set, there are following L pixels in which each pixel

The RLE data starts at position 10 (decimal) in the file.

Is that all?

No, absolutely not. What I was able to decode was merely a part of the scene files. There’s a lot more in there. Some candidates

  • z-Values. In order to correctly let walls occlude characters, there has to be a Z map or something like that.
  • Camera properties. To be able to render the characters correctly, the camera properties (focal length, etc.) have to be known.

Create a list with predefined number of entries in python

You want to create an array-like data structure in python. You know, how many entries you will need for your list. What do you do?

I always thought you have to do something like

lst = []
for i in range(N):

Today i learned here, that in python you can simply say

lst = [None]*N

and it is even much faster. I never stop learning, even after several years of using python 🙂

Mount a partition from a disk image for bochs

There are a couple of articles out there which describe, how to mount a particular partition from an image file containing not a single filesystem but a complete disk including the partition table (e.g. here). They tell you to use the offset option of the mount command and they tell you, you’ll get the partition offset from the partition table via fdisk. Well, at this point I naturally assumed, that fdisk is providing the offsets in „numbers of sectors“.

fdisk -l c.img
Platte /home/omni/.bochs/freedos/c.img: 0 MByte, 0 Byte
16 Köpfe, 63 Sektoren/Spur, 0 Zylinder
Einheiten = Zylinder von 1008 × 512 = 516096 Bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

 Gerät  boot.     Anfang        Ende     Blöcke   Id  System
 c.img1   *       1             304      153184+  6   FAT16

So I tried to run

mount -o loop,offset=512 -t vfat c.img /mnt

Unfortunately, this did not work! The official bochs documentation in fact contains the solution, you just have to know it does:

If you want access to a hard disk image, you have to calculate the size of
    the first cylinder.

What was wrong? I incorrectly assumed, that the fdisk partition table provides the start position in „numbers of sectors“. I does not. I gives them in „numbers of cylinders“! You have to explicitely run fdisk with argument -u to get the offsets in numbers of sectors!

fdisk -lu c.img
Gerät  boot.     Anfang        Ende     Blöcke   Id  System
c.img1 *         63            306431   153184+  6   FAT16

I lost some hours before realizing this. Maybe this info will help someone to not walk into the same trap!

completing file paths in python cmd module

python’s cmd module is great. You derive your class from the provided command interpreter and you can start right away implementing command. Let’s have a look at a small command interpreter for a beer-drinking bot.

import cmd

class BeerRobotCmd(cmd.Cmd):
	def __init__(self):
                #for the sake of simplicity we make
                #those states part of the cmd object. Normally,
                #of course you would use your cmd class to drive
                #some external interface.
		self.stomach_filled = 0
		self.drunkness = 0
		self.singing = None

	def do_drink(self, line):
		if self.stomach_filled >= 100:
			print "robot is full of beer"
			print "robot drinks beer"
			self.stomach_filled += 10
			self.drunkness += 1

	def do_throwup(self, line):
		print "robot throws up"
		self.stomach_filled = 0

	def do_sing(self, line):
		if len(line.strip()) > 0:
			if self.singing != None:
				print "robot is singing already"
			elif self.drunkness < 15:
				print "robot is too shy"
				print "robot starts to sing", line
				self.singing = line
			if self.singing:
				print "robot stops singing"

commandline = BeerRobotCmd()

Our robot now understands 3 important commands

  1. drink makes him drink a beer, until he’s full.
  2. throwup makes him throw up (wow, surprising 🙂 ) so his stomach is empty again
  3. sing makes him sing a custom sing. You can tell him to sing, for example, the national anthem or a lullaby. If you call „sing“ without any arguments, he will stop to sing

We have amazingly created those commands simply by implementing 3 methods

def do_commandname(self, line)

In those line contains all the characters entered by the user after the actual command string. So in order to extract arguments from there, we simply have so split or search or process the line argument in some way.

This quickly-implemented command interpreter even contains completion for commands already. Pressing TAB without entering any character will show the user all avaible commands.

argument completion

The cmd module allows you to implement a simple completion callback intended not for the command strings but for the arguments.

Here’s a completion callback for the sing command, proposing to the user a list of songs.

	def complete_sing(self, text, line, startidx, endidx):
		return [ song for song in ['yesterday', 'uptown girl', 'anything goes', 'old mcdonald has a farm'] if song.startswith(text) ]

Like in the unix shell, where hitting TAB while entering a file name will prompt you with possible completions, this callback will provide possible completions as well. It works as follows

  • the line argument contains the complete line, like in the do_foo methods
  • the text argument contains the part of the current argument, the user has entered already. This is, of course, inteded to be used as a filter for the set of possible completions. If i enter „sing yest“ and hit TAB, this callback is called with text = „yest“.
  • the startidx and endidx arguments are indices into the line string, describing which subset of line is contained in text. For the case of „sing yest“ and hitting TAB, those args would contains 1 and 4 respectively (not 0, because the space between „sing“ and „yest“ is interpreted as the first character of the line string).

argument splitting problem

Well, this is all great. Until you try to complete arguments containing characters, interpreted by the completion mechanism as separators. Like a guiltless „/“. This is bad. You won’t be able to implement the completion of file name arguments. Assume, for example, the user has entered „SOMECOMMAND /home/foo/bar/“ and wants to see now (similiar to the unix shell) what files are there in the directory /home/foo/bar/. He hit’s TAB. The completion mechanism scans the current command line and splits it into SOMECOMMAND<start of argument line> /<first argument>home</first argument>/<second argument>foo</second argument>/<third argument>bar</third argument>/<fourth argument></fourth argument>. Now your completion callback is called with text=““ (fourth argument). This does not work!

Maybe there’s a simple solution by changing the separators used? I didn’t find a way to do that. So I implemented the following workaround. First let’s introduce a function to do completion from filenames

import os

def get_possible_filename_completions(text):
	head, tail = os.path.split(text.strip())
	if head == "": #no head
		head = "."
	files = os.listdir(head)
	return [ f for f in files if f.startswith(tail) ]

def extract_full_argument(line, endidx):
	newstart = line.rfind(" ", 0, endidx)
	return line[newstart:endidx]

The first function extract possible file names from a given input path. If you have files a, b and c in a folder /foo/bar/, passing „/foo/bar/“ to this function will result in the list [„a“, „b“, „c“]. Careful: In order to work together with the cmd completion mechanism splitting arguments at „/“ characters, this function must not return absolute file paths. Assume for a moment we would have written in the above code

return [ head + "/" + f for f in files if f.startswith(tail) ]

What would happen?

We would enter „SOMECOMMAND /foo/bar/a“, then hit TAB. get_possible_filename_completions would receive text=“/foo/bar/a“, split it into head=“/foo/bar“ and tail = „a“. It would then search in „/foo/bar“ and find [„a“, „b“, „c“]. This list would then be filtered and only „a“ would remain. Now, head + „/“ + f would be the full path „/foo/bar/a“. Now comes the important point: the cmd completion mechanism would substitute what it deems to be the argument to be completed, which would be „a“ for the return value of get_possible_filename_completions. So we would end up with „SOMECOMMAND /foo/bar//foo/bar/a“. This is certainly not what we inteded!

The second function is the workaround. It takes the end index for the argument – this is where the cursor is. It then searches from this point on for a space character left of the end index. This way it does not stop at the next „/“ character. It returns the complete argument, which can then be fed to get_possible_filename_completions.