Mercurial Source Server Integration 0.1 Released!

You (the reader of course) will have to forgive my lack of updates on the 0.1 release, it's been a busy week with studying for midterms and whatnot. However, the good news is my 0.1 release has been completed. For instructions about how to set up the release, see this page

I guess since I never officially listed my objectives for this release I'll do it now.

• Set up existing source server to see how it works(will download pdb's off mozilla server)


• Set up localhost server


• Build firefox and create pdb files (symbols), host them on local server


• Insert my own data block into one of the mozilla pdb files (will use xul.pdb)


• Get xul.pdb to grab source files off of hgweb instead of cvs repo

It was a long process trying to accomplish these tasks. The most daunting was trying to properly insert my data block into xul.pdb and have it work.

First I should point out that I had 2 locations for symbols set up. The first was a folder that stored the symbols i downloaded from http://symbols.mozilla.org/firefox (ill refer to this as source 1), the second was a folder that contained symbols i generated myself using make buildsymbols on my local version of minefield that I grabbed from hg's mozilla-central tree. I then copied these symbols to my local IIS server (ill refer to this as source 2).

My initial goal was to modify the xul.pdb in source 2 to grab the source files off of hgweb. To do this, I had to insert my own data block into xul.pdb. The data block is used to tell the symbol where to retrieve it's source files. Here is the data block I created

SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
SRCSRV: variables ------------------------------------------
HGSERVER=http://hg.mozilla.org/mozilla-central
SRCSRVVERCTRL=http
HTTP_EXTRACT_TARGET=%hgserver%/index.cgi/raw-file/7356c512e9e1/%var3%/%fnfile%(%var1%)
SRCSRVTRG=%http_extract_target%
SRCSRVCMD=
SRCSRV: source files ---------------------------------------
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.cpp*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.h*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheMetaData.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheService.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.h*MYSERVER*widget/src/xpwidgets
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.cpp*MYSERVER*widget/src/xpwidgets
SRCSRV: end ------------------------------------------------

This data block was hard to nail down, because there was almost NO information on the web in regards to how to properly use http to extract source files. I hope that my blog can become a valuable resource in that regard.

The key is the variables section, the HTTP_EXTRACT_TARGET var tells the pdb where to grab the source file from. You'll notice keywords such as %var3%, this is referring to the lines in the source files section, where each var is seperated by an *. So for the first source file line, var3 is xpcom/io. You've probably noticed that I don't really need var1 and var2 in this instance, you're right. This data block was a modification of the existing data block in source 1's xul.pdb, i just took it and modified it.

Using the pdbstr.exe found in the Debugging Tools for Windows, I wrote the data block to source 2's xul.pdb. I ran srctool -f on the pdb to see if it was indexed, the response was that no files were indexed. I was suprised at this. When I tried to analyze the code in symbolstore.py (which I will have to modify in later releases), I came to the conclusion that the script simply gathered all the source file paths, wrote them to an auto-generated data block, and then wrote that data block to pdbstr. So when I tried to use this method manually, I was suprised to find out that I was unsuccessful. Perhaps my data block wasn't written properly.

I tried to test the data block instead on source 1. When I ran srctool on source 1's xul.pdb after inserting my own data block, it showed me that 6 files were indexed, and you'll noticed that there are 6 source files in the data block. To further check, I re-ran Visual Studio, and re-attached to firefox with source 1 as my source server (see the instructions above for further information on the process). Success! Clicking on the call to nsBaseAppShell.cpp in the Call Stack downloaded the source file from hgweb.

I still have a lingering question. Why was the source 1's xul.pdb able to index/work with my data block, while source 2's xul.pdb wasn't? I'm thinking that perhaps something is done to mozilla's symbols that I'm not aware of. Either I missed something in symbolstore.py or something happens outside of symbolstore.py that indexes the files.

Another observation I had was in regards to using Minefield. When debugging Minefield using source 2, all the source files were downloaded from my own machine. I guess the way pdb's work is that they first check for a local copy of the source file, and if it's not there they make the call to whatever remote site is specified in the data block.

My next objective for 0.2 is still to be determined. I'd also like to thank ted and lblakk for all the help they've given me with various aspects of this release.

Just one final note: My style has traditionally been to leave things alone and finishing them later (or to put it bluntly, procrastinating). I've learned through doing this release that for a project like this, that won't fly. Between having to study for midterms this week and doing this release, I feel lucky to have gotten everything done. For 0.2 and beyond, I plan on getting an early jump on things. That means smaller, more frequent blog posts instead of the essay I'm typing right now.

My biggest obstacle with this project so far was just getting started, because I absolutely knew nothing about it. Now that I've come to understand certain aspects of it, I feel a lot more enthusiastic about it, and I hope my future work reflects that enthusiasm.