0.3 Release Submitted

Well, I've submitted my third patch, which I will be considering as my 0.3 release. Whether or not it gets accepted remains to be seen but I feel like I've done enough to submit a good 0.3.

Most of the issues I needed to correct were minor things such as naming conventions, however there were also some other issues that ted pointed out that I needed to fix.

The first was a change to the data block. Ted didn't like the fact that the SRCSRVTRG variable was set to HTTP_EXTRACT_TARGET. Instead he wanted this variable to point to a local target because SRCSRVTRG was supposed to resolve to the path that the file is extracted to. What I found is that while this is true for a CVS extraction, I believe it serves a different purpose when used for HTTP extraction.

When I run srctool on a pdb using the old CVS implementation, I see entries like this:

[e:\fx19rel\winnt_5.2_depend\mozilla\layout\style\nsrulenode.h] cmd: cvs.exe -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot checkout -r 1.64 -d 1.64 -N mozilla/layout/style/nsRuleNode.h

As you can see, the header file has a "cmd" which is run to retrieve the source file. The SRCSRVTRG is then set as a local path to tell srcsrv where to dump the file. However, in my HTTP implementation srctool showed this:

[c:\mozilla\src\toolkit\xre\nsupdatedriver.h] trg: http://hg.mozilla.org/mozilla-central/raw-file/3611062ed098/toolkit/xre/nsUpateDriver.h 

As you can see, the files no longer use a "cmd" but a "trg". I wasn't sure if this was related to HTTP_EXTRACT_TARGET or SRCSRVTRG so I experimented by changing SRCSRVTRG to ted's recommendation, and srctool showed this:

[c:\mozilla\src\toolkit\xre\nsupdatedriver.h] trg: C:\mozilla\symbols\xul.pdb\.... 

The trg was now the location of the xul.pdb file. This showed me that SRCSRVTRG was being used to determine where the file was located, and thus was being used differently than if the source server was using CVS. Testing in Visual Studio proved this, as the source server didn't know where to find the file. So, to sum things up, SRCSRVTRG had to remain the way it was.

I was also asked to remove the SRCSRVCMD command from the data block, as it wasn't being set to anything. I assumed I needed it as the srcsrv documentation said it was a required variable, but it turns out that I didn't need it so I was happy to remove it.

So, I was ready to submit a new patch when I found out that my entire solution didn't work in Visual Studio 2005! I had been testing on 2008 the entire time and it worked like a dream, however 2005 didn't seem to know how to do HTTP retrieval. After discussing this with ted and sending him my stuff, he got it working fine in WinDBG (I had tried WinDBG myself without any success but I must have done something wrong in the setup) so in the end not working in 2005 was ok with ted, he speculated that perhaps srcsrv wasn't properly implemented in 2005 and I'm inclined to agree based on what I've seen in my testing.

So, now that I've resolved these issues I feel confident about getting my project implemented in the tree. Maybe not with this patch, but eventually.

Mercurial Source Server Integration 0.1 Released!

You (the reader of course) will have to forgive my lack of updates on the 0.1 release, it's been a busy week with studying for midterms and whatnot. However, the good news is my 0.1 release has been completed. For instructions about how to set up the release, see this page

I guess since I never officially listed my objectives for this release I'll do it now.

• Set up existing source server to see how it works(will download pdb's off mozilla server)


• Set up localhost server


• Build firefox and create pdb files (symbols), host them on local server


• Insert my own data block into one of the mozilla pdb files (will use xul.pdb)


• Get xul.pdb to grab source files off of hgweb instead of cvs repo

It was a long process trying to accomplish these tasks. The most daunting was trying to properly insert my data block into xul.pdb and have it work.

First I should point out that I had 2 locations for symbols set up. The first was a folder that stored the symbols i downloaded from http://symbols.mozilla.org/firefox (ill refer to this as source 1), the second was a folder that contained symbols i generated myself using make buildsymbols on my local version of minefield that I grabbed from hg's mozilla-central tree. I then copied these symbols to my local IIS server (ill refer to this as source 2).

My initial goal was to modify the xul.pdb in source 2 to grab the source files off of hgweb. To do this, I had to insert my own data block into xul.pdb. The data block is used to tell the symbol where to retrieve it's source files. Here is the data block I created

SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
SRCSRV: variables ------------------------------------------
HGSERVER=http://hg.mozilla.org/mozilla-central
SRCSRVVERCTRL=http
HTTP_EXTRACT_TARGET=%hgserver%/index.cgi/raw-file/7356c512e9e1/%var3%/%fnfile%(%var1%)
SRCSRVTRG=%http_extract_target%
SRCSRVCMD=
SRCSRV: source files ---------------------------------------
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.cpp*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.h*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheMetaData.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheService.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.h*MYSERVER*widget/src/xpwidgets
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.cpp*MYSERVER*widget/src/xpwidgets
SRCSRV: end ------------------------------------------------

This data block was hard to nail down, because there was almost NO information on the web in regards to how to properly use http to extract source files. I hope that my blog can become a valuable resource in that regard.

The key is the variables section, the HTTP_EXTRACT_TARGET var tells the pdb where to grab the source file from. You'll notice keywords such as %var3%, this is referring to the lines in the source files section, where each var is seperated by an *. So for the first source file line, var3 is xpcom/io. You've probably noticed that I don't really need var1 and var2 in this instance, you're right. This data block was a modification of the existing data block in source 1's xul.pdb, i just took it and modified it.

Using the pdbstr.exe found in the Debugging Tools for Windows, I wrote the data block to source 2's xul.pdb. I ran srctool -f on the pdb to see if it was indexed, the response was that no files were indexed. I was suprised at this. When I tried to analyze the code in symbolstore.py (which I will have to modify in later releases), I came to the conclusion that the script simply gathered all the source file paths, wrote them to an auto-generated data block, and then wrote that data block to pdbstr. So when I tried to use this method manually, I was suprised to find out that I was unsuccessful. Perhaps my data block wasn't written properly.

I tried to test the data block instead on source 1. When I ran srctool on source 1's xul.pdb after inserting my own data block, it showed me that 6 files were indexed, and you'll noticed that there are 6 source files in the data block. To further check, I re-ran Visual Studio, and re-attached to firefox with source 1 as my source server (see the instructions above for further information on the process). Success! Clicking on the call to nsBaseAppShell.cpp in the Call Stack downloaded the source file from hgweb.

I still have a lingering question. Why was the source 1's xul.pdb able to index/work with my data block, while source 2's xul.pdb wasn't? I'm thinking that perhaps something is done to mozilla's symbols that I'm not aware of. Either I missed something in symbolstore.py or something happens outside of symbolstore.py that indexes the files.

Another observation I had was in regards to using Minefield. When debugging Minefield using source 2, all the source files were downloaded from my own machine. I guess the way pdb's work is that they first check for a local copy of the source file, and if it's not there they make the call to whatever remote site is specified in the data block.

My next objective for 0.2 is still to be determined. I'd also like to thank ted and lblakk for all the help they've given me with various aspects of this release.

Just one final note: My style has traditionally been to leave things alone and finishing them later (or to put it bluntly, procrastinating). I've learned through doing this release that for a project like this, that won't fly. Between having to study for midterms this week and doing this release, I feel lucky to have gotten everything done. For 0.2 and beyond, I plan on getting an early jump on things. That means smaller, more frequent blog posts instead of the essay I'm typing right now.

My biggest obstacle with this project so far was just getting started, because I absolutely knew nothing about it. Now that I've come to understand certain aspects of it, I feel a lot more enthusiastic about it, and I hope my future work reflects that enthusiasm.