28 October 2005 3:53 PM (python)


For some reason Flumotion is running at 100% cpu when starting components. It didn't use to do this, and it doesn't really make much sense -- the rate of data input is limited by the capture devices, and a certain time's worth of data has to accumulate to fill all of the relevant buffers in the encoders, muxers, and streamers.

But I have no clue where this CPU usage is. I tried pressing control-C occasionally and getting backtraces in GDB, but the normal methods aren't working.

In the end this was all due to not having an idea of where Flumotion spends its time. I wanted to profile, but didn't want to use an instrumenting profiler, and there didn't exist a sampling profiler for Python. So I ported the one I hacked on for Guile to python; it's available here.

Usage goes like this:

>>> import statprof
>>> statprof.start()
>>> import test.pystone; test.pystone.pystones()
(1.3200000000000001, 37878.78787878788)
>>> statprof.stop()
>>> statprof.display()
  %   cumulative      self          
 time    seconds   seconds  name    
 23.01      1.36      0.31
 15.04      0.60      0.20
 11.50      0.16      0.16
 10.62      0.14      0.14
  7.96      0.16      0.11
  7.96      0.11      0.11
  6.19      0.12      0.08
  5.31      0.07      0.07
  3.54      0.20      0.05
  2.65      0.05      0.04
  2.65      0.04      0.04
  1.77      0.02      0.02
  0.88      0.01      0.01
  0.88      0.01      0.01
  0.00      1.36      0.00
  0.00      1.36      0.00  <stdin>:1:?
Sample count: 113
Total time: 1.360000 seconds

There's lots of info in help(statprof). Also it requires the itimer extension from Just unpack the tarball and run sudo python install.

It's not very well tested at this point, but it gives similar results as the stock profiler (10-20 times faster though). It's of course not the same, because an instrumenting profiler unfairly penalizes procedure calls; while they do have a cost it is much less than their cost as measured by the stock profiler or hotshot.

I'd be interested in hearing about bugs in it for the next few weeks; after that it will probably slip from my mind though. (This is of course the natural fate of profilers, to bitrot. I have a larger rant about this for later.)

Comments are closed.