Using TraceMalloc with Songbird

XULRUNNER

Start by building your own XULRunner.  See http://wiki.songbirdnest.com/Developer/Articles/Getting_Started/Core_Player_Development/Building_Songbird/Building_XULRunner_for_Songbird

We need a hybrid half-release-half-debug-plus-tracemalloc build, so modify your debug mozconfig as follows:

 -ac_add_options --disable-optimize --enable-debug
 +ac_add_options --enable-cpp-rtti
 +ac_add_options --enable-trace-malloc
 +ac_add_options --disable-debug
 +ac_add_options --enable-optimize="-O -g"

Now check out the mozilla/tools/trace-malloc directory from mozilla central and copy it into your XULRunner tree

Build as normal:
  make -f client.mk build

... and watch it fail with
  ld: library not found for -ltracemalloc
  collect2: ld returned 1 exit status

This is probably because it wasn't meant to build with XULRunner. We could patch the build system, but whatever, just copy the file to the right place:

  cd compiled/xulrunner-debug
  cp staticlib/libtracemalloc.a dist/lib/
  make

Install the new xulrunner as a release build, NOT a debug build
 /builds/songbird/trunk/tools/scripts/make-mozilla-sdk.sh /builds/xulrunner/mozilla /builds/xulrunner/mozilla/compiled/xulrunner-debug /builds/songbird/trunk/dependencies/macosx-i686/mozilla/release

 /builds/songbird/trunk/tools/scripts/make-xulrunner-tarball.sh /builds/xulrunner/mozilla/compiled/xulrunner-debug/dist/bin /builds/songbird/trunk/dependencies/macosx-i686/xulrunner/release xulrunner.tar.gz

SONGBIRD

Before building Songbird, hack your configure.ac so that you get a release build with symbols, RTTI, and tracemalloc.

 Index: configure.ac
 ===================================================================
 --- configure.ac    (revision 12576)
 +++ configure.ac    (working copy)
 @@ -685,18 +685,18 @@
      MACOSX_FRAMEWORKS="$MACOSX_CONTENTS/Frameworks"
      XULRUNNERDIR="/XUL.framework"
  
 -    GCC_COMPILER_FLAGS="-c -fexceptions -fnon-call-exceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -fpascal-strings -fno-rtti -no-cpp-precomp -Wall -Wconversion -Wpointer-arith -Wcast-align -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wno-long-long -fshort-wchar -pipe"
 +    GCC_COMPILER_FLAGS="-c -fexceptions -fnon-call-exceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -fpascal-strings -frtti -no-cpp-precomp -Wall -Wconversion -Wpointer-arith -Wcast-align -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wno-long-long -fshort-wchar -pipe"
      if test "$enable_debug" = "yes"; then
        GCC_COMPILER_FLAGS="-ggdb3 $GCC_COMPILER_FLAGS -fno-inline -O1"
      else
        # Use -O2 on OS X, better than -Os according to moz perf tests
 -      GCC_COMPILER_FLAGS="-gstabs+ $GCC_COMPILER_FLAGS -O2"
 +      GCC_COMPILER_FLAGS="$GCC_COMPILER_FLAGS -O -g"
      fi
  
      # We always want to include "mozilla-config.h"
      GCC_COMPILER_FLAGS="$GCC_COMPILER_FLAGS -include \"mozilla-config.h\""
  
 -    GCC_LINKER_FLAGS="-lpthread -lobjc -shared-libgcc -lstdc++ -Wl,-executable_path,\$(topsrcdir)/$OBJDIRNAME/$DISTDIRNAME$MACOSX_FRAMEWORKS$XULRUNNERDIR"
 +    GCC_LINKER_FLAGS="-lpthread -ltracemalloc -lobjc -shared-libgcc -lstdc++ -Wl,-executable_path,\$(topsrcdir)/$OBJDIRNAME/$DISTDIRNAME$MACOSX_FRAMEWORKS$XULRUNNERDIR"
  
      CFLAGS="$CFLAGS $GCC_COMPILER_FLAGS"
      CXXFLAGS="$CXXFLAGS $GCC_COMPILER_FLAGS" 

Then copy over the tracemalloc library, since the make-sdk script ignores it
cp /builds/xulrunner/mozilla/compiled/xulrunner-debug/staticlib/libtracemalloc.a /builds/songbird/trunk/dependencies/macosx-i686/mozilla/release/lib/


Now build Songbird in release mode.
make -f songbird.mk clobber ; make -f songbird.mk


If you have MacPorts installed, there is a chance Songbird will fail to build with your XULRunner due to library complaints in xpidl.  
If this happens, try the following hack:

 Index: build/rules.mk
 ===================================================================
 --- build/rules.mk    (revision 12576)
 +++ build/rules.mk    (working copy)
 @@ -50,10 +50,6 @@
    endif
  endif
  
 -ifeq (macosx,$(SB_PLATFORM))
 -  SB_DYLD_LIBRARY_PATH = $(DEPS_DIR)/libIDL/$(SB_CONFIGURATION)/lib:$(DEPS_DIR)/glib/$(SB_CONFIGURATION)/lib:$(DEPS_DIR)/gettext/$(SB_CONFIGURATION)/lib
 -  export DYLD_LIBRARY_PATH = $(SB_DYLD_LIBRARY_PATH)
 -endif 

TRACEMALLOC SNAPSHOTS

When it finishes building, run Songbird with tracemalloc enabled, but the log output ignored.  
Either the log output is bad, or the tools (like leaksoup) are obsolete. Either way, the log is useless, and we only really care about the runtime snapshots.

cd compiled/dist/Songbird.app/Contents/MacOS/
./songbird --trace-malloc /dev/null

Now take a snapshot before performing the task you want to analyze. Open the error console and run
  TraceMallocDumpAllocations("before.snapshot");
 
It should hang for a moment while dumping the snapshot.

Perform the task you want to analyze (in my case, letting it play for several hours).

Once complete, take another snapshot
  TraceMallocDumpAllocations("after.snapshot");

The snapshots in their current form are pretty useless, since the stacks are incorrect. To fix this, run
 /builds/songbird/trunk/tools/tracemalloc/fix-macosx-stack.pl before.snapshot > before.fixedsnapshot
 /builds/songbird/trunk/tools/tracemalloc/fix-macosx-stack.pl after.snapshot > after.fixedsnapshot

This is a forked version of http://mxr.mozilla.org/mozilla-centr...acosx-stack.pl
It has been slightly modified so that the output matches the format of the input, allowing it to be used with existing tools like diffbloatview.pl and histogram.pl.  

ANALYSIS

To yield a new snapshot with only the allocations that appeared between the two captured snapshots, try the whatsnew.rb script:
  ruby /builds/songbird/trunk/tools/tracemalloc/whatsnew.rb before.fixedsnapshot after.fixedsnapshot > whatsnew.fixedsnapshot

You can look through this snapshot manually, or use it as input into the histogram and visualization scripts.


For a collapsed tree view of the differences between two snapshots, try the diffbloatdump.pl
 /builds/xulrunner/mozilla/tools/trace-malloc/diffbloatdump.pl --depth=50 before.fixedsnapshot after.fixedsnapshot > snapshot.diff

This groups allocations by stack frame, and lists them in reverse order. Large values for Songbird related stacks deserve further investigation.  Make sure you try this script. It is probably the most useful.

Example:

 3951360 AllocateAudioBufferList
   3951360 open_decoder
     3951360 qtwrapper_audio_decoder_sink_setcaps
       3951360 gst_pad_set_caps
         3951360 gst_pad_configure_sink
           3951360 gst_pad_chain_unchecked
             3951360 gst_pad_push
               3951360 gst_base_src_loop
                 3951360 gst_task_func
                   3951360 g_thread_pool_thread_proxy
                     3951360 g_thread_create_proxy
                       3951360 _pthread_start

This indicates that our quicktime wrapper allocated 4MB that was never freed.


To generate a basic type histogram from a snapshot, use histogram.pl.  
  /builds/xulrunner/mozilla/tools/trace-malloc/histogram.pl whatsnew.fixedsnapshot

The output from this tool is similar to about:bloat and the OSX 'heap' command:

  void* 7103 2551872
  gfxTextRun 19 6341
  sbPropertyArray 91 3276
  sbPropertyData 66 2640
  sbSimpleProperty 13 416
  gfxImageSurface 9 360
  sbLocalDatabaseResourcePropertyBag 3 312
  gfxQuartzImageSurface 9 144
  gfxQuartzSurface 2 80
  gfxAtsuiFontGroup 1 80
  sbLocalDatabaseMediaItem 1 64 

There is a type database that can be used to infer void* allocations, but it is outdated and Windows specific.  We could update it, but the effort required doesn't seem worthwhile.


Instead of a thorough type database, I prefer a loose, easy to configure system for mapping/excluding/blaming allocations on components and classes.  For example, I don't really care which specific classes and functions in the Javascript engine made allocations, I just want to see which Songbird components are ultimately responsible.  To answer questions like this, use the blame-map.rb script, and the treemap.html viewer.

For a high level view of what is in a snapshot, run:
 ruby /builds/songbird/trunk/tools/tracemalloc/blame-map.rb < whatsnew.fixedsnapshot > json-treemap-data.js
 open treemap.html

Which produces a map like:


When viewing the treemap, left-click to zoom in on a box, and right-click to zoom out.

To change the granularity of the data displayed, modify $blame_list in blame-map.rb.  For example, if you wanted to see all classes in the database engine, you would map "sbDBEngine.dylib" to :breakapart.  Rather than treating sbDBEngine as a single entity, each class within it would appear in the tree.  You can also extract individual classes from a library, leaving the rest as a single entity, by mapping a class name like "nsStringBuffer" to a display name.

When investigating memory usage, you might start out showing only libraries. After browsing around this view, you might notice an unusual amount of memory attributed to the metadata component, so you would re-run the script requesting more detail for the metadata component only.  From there you might expand all nsString related classes and better understand what kinds of objects were being allocated.

Experiment a bit.  This approach is extremely powerful once you get the hang of it, as it lets you understand the entire snapshot at once, selectively raising and lowering the level of detail as you go.