0.3 Release Submitted

Well, I've submitted my third patch, which I will be considering as my 0.3 release. Whether or not it gets accepted remains to be seen but I feel like I've done enough to submit a good 0.3.

Most of the issues I needed to correct were minor things such as naming conventions, however there were also some other issues that ted pointed out that I needed to fix.

The first was a change to the data block. Ted didn't like the fact that the SRCSRVTRG variable was set to HTTP_EXTRACT_TARGET. Instead he wanted this variable to point to a local target because SRCSRVTRG was supposed to resolve to the path that the file is extracted to. What I found is that while this is true for a CVS extraction, I believe it serves a different purpose when used for HTTP extraction.

When I run srctool on a pdb using the old CVS implementation, I see entries like this:

[e:\fx19rel\winnt_5.2_depend\mozilla\layout\style\nsrulenode.h] cmd: cvs.exe -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot checkout -r 1.64 -d 1.64 -N mozilla/layout/style/nsRuleNode.h

As you can see, the header file has a "cmd" which is run to retrieve the source file. The SRCSRVTRG is then set as a local path to tell srcsrv where to dump the file. However, in my HTTP implementation srctool showed this:

[c:\mozilla\src\toolkit\xre\nsupdatedriver.h] trg: http://hg.mozilla.org/mozilla-central/raw-file/3611062ed098/toolkit/xre/nsUpateDriver.h 

As you can see, the files no longer use a "cmd" but a "trg". I wasn't sure if this was related to HTTP_EXTRACT_TARGET or SRCSRVTRG so I experimented by changing SRCSRVTRG to ted's recommendation, and srctool showed this:

[c:\mozilla\src\toolkit\xre\nsupdatedriver.h] trg: C:\mozilla\symbols\xul.pdb\.... 

The trg was now the location of the xul.pdb file. This showed me that SRCSRVTRG was being used to determine where the file was located, and thus was being used differently than if the source server was using CVS. Testing in Visual Studio proved this, as the source server didn't know where to find the file. So, to sum things up, SRCSRVTRG had to remain the way it was.

I was also asked to remove the SRCSRVCMD command from the data block, as it wasn't being set to anything. I assumed I needed it as the srcsrv documentation said it was a required variable, but it turns out that I didn't need it so I was happy to remove it.

So, I was ready to submit a new patch when I found out that my entire solution didn't work in Visual Studio 2005! I had been testing on 2008 the entire time and it worked like a dream, however 2005 didn't seem to know how to do HTTP retrieval. After discussing this with ted and sending him my stuff, he got it working fine in WinDBG (I had tried WinDBG myself without any success but I must have done something wrong in the setup) so in the end not working in 2005 was ok with ted, he speculated that perhaps srcsrv wasn't properly implemented in 2005 and I'm inclined to agree based on what I've seen in my testing.

So, now that I've resolved these issues I feel confident about getting my project implemented in the tree. Maybe not with this patch, but eventually.

0.2 - Finished

Well, after a lot of trial and error I got my local server to serve the pdb files properly.

The big problem I was having was that although the symbols were being loaded, when I tried to retrieve source code it would only ask me to locate my local source files and not try to retrieve the source files from hgweb. I did a lot of searching, checking, fixing etc.

It just so happens that I lucked out bigtime, as I finally noticed that the symbols being loaded when I started up Minefield weren't coming from my symbol server. Instead, the symbols being loaded were symbols inside my object directory. I'm not sure when these symbols got created but they were scattered throughout the object directory, and unlike the symbols I was using, they weren't indexed. So, to test this out I deleted the xul.pdb in the object directory and re-attached to Minefield. Success! Now xul.dll was loading the pdb from my server instead of the objdir one, the source code was being pulled from hgweb and all was well.

Now, I'm not sure what my deliverable should be. Perhaps it should just be the new pdb's I've written, or the new symbolstore.py even. Or, maybe I've done enough to submit a patch.

EDIT: I have filed a patch under Bug 440001

0.2 Release - Lots of Progress

Firstly, to add to my last post I decided to set my deliverable for 0.2

• Hack symbolstore.py to index the pdb files


• Serve the pdb files over my server and make sure they successfully retrieve source code from hgweb

Well, I'm happy to report that I've managed to accomplish the first of these tasks. A big thank you to lsblakk and ted for their help.

After reading up on Python, I came to understand a lot more about how symbolstore.py works.

Basically, it parses out a list of source files from each .sym file, writes a .stream file using the list of source files and calls pdbstr to write the data block to the pdb files.

Fortunately for me, lsblakk's previous modifications to symbolstore.py to make it work with CVS gave me all of the code I needed. All that was required from me was to make some slight changes.

Firstly, I needed to change the SourceIndex() method to write my data block instead of the one used previously for CVS. My new method is listed below:

def SourceIndex(fileStream, outputPath, hg_root):
    """Takes a list of files, writes info to a data block in a .stream file"""
    # Creates a .pdb.stream file in the mozilla\objdir to be used for source indexing
    # Create the srcsrv data block that indexes the pdb file
    result = True
    pdbStreamFile = open(outputPath, "w")
    pdbStreamFile.write('''SRCSRV: ini ------------------------------------------------\r\nVERSION=2\r\nINDEXVERSION=2\r\nVERCTRL=http\r\nSRCSRV: variables ------------------------------------------\r\nHGSERVER=http://hg.mozilla.org/mozilla-central''')
    # pdbStreamFile.write(hg_root)
    pdbStreamFile.write('''\r\nSRCSRVVERCTRL=http\r\nHTTP_EXTRACT_TARGET=%hgserver%/index.cgi/raw-file/%var3%/%var2%\r\nSRCSRVTRG=%http_extract_target%\r\nSRCSRVCMD=\r\nSRCSRV: source files ---------------------------------------\r\n''')
    pdbStreamFile.write(fileStream) # can't do string interpolation because the source server also uses this and so there are % in the above
    pdbStreamFile.write("SRCSRV: end ------------------------------------------------\r\n\n")
    pdbStreamFile.close()
    return result

Secondly, I simply needed to change the ProcessFile method inside of the Dumper class to look for hg files instead of cvs files. So from:

if filename.startswith("cvs"):

To:

if filename.startswith("hg"):

Like I said, no crazy coding required.

I ran into problems when trying to run 'make buildsymbols'. For some reason empty pdb files were being created, so I concluded that symbolstore.py wasn't running properly. I eventually discovered that the makefile wasn't running symbolstore.py using the -i flag. This flag is what tells symbolstore.py to add source file information to the pdb files.

I discovered that there was actually code in the makefile to add this flag:

ifdef PDBSTR_PATH
MAKE_SYM_STORE_ARGS += -i
endif

So, the reason that the pdb files were empty was because PDBSTR_PATH wasn't being set, so I simply added it to my environment variables. After that, voila! The pdb files were successfully indexed upon creation.

My next step is to serve these pdb files from a server and make sure that the source files are being retrieved from hgweb. I don't think I can do this with a localhost server because the debugger will find my local source files before it tries to look on hgweb. So, I need to find a place to serve my pdb files (I would like to find a remote server to do it so that I can maybe offer a way for people to test it).

Mercurial Source Server Integration 0.1 Released!

You (the reader of course) will have to forgive my lack of updates on the 0.1 release, it's been a busy week with studying for midterms and whatnot. However, the good news is my 0.1 release has been completed. For instructions about how to set up the release, see this page

I guess since I never officially listed my objectives for this release I'll do it now.

• Set up existing source server to see how it works(will download pdb's off mozilla server)


• Set up localhost server


• Build firefox and create pdb files (symbols), host them on local server


• Insert my own data block into one of the mozilla pdb files (will use xul.pdb)


• Get xul.pdb to grab source files off of hgweb instead of cvs repo

It was a long process trying to accomplish these tasks. The most daunting was trying to properly insert my data block into xul.pdb and have it work.

First I should point out that I had 2 locations for symbols set up. The first was a folder that stored the symbols i downloaded from http://symbols.mozilla.org/firefox (ill refer to this as source 1), the second was a folder that contained symbols i generated myself using make buildsymbols on my local version of minefield that I grabbed from hg's mozilla-central tree. I then copied these symbols to my local IIS server (ill refer to this as source 2).

My initial goal was to modify the xul.pdb in source 2 to grab the source files off of hgweb. To do this, I had to insert my own data block into xul.pdb. The data block is used to tell the symbol where to retrieve it's source files. Here is the data block I created

SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
SRCSRV: variables ------------------------------------------
HGSERVER=http://hg.mozilla.org/mozilla-central
SRCSRVVERCTRL=http
HTTP_EXTRACT_TARGET=%hgserver%/index.cgi/raw-file/7356c512e9e1/%var3%/%fnfile%(%var1%)
SRCSRVTRG=%http_extract_target%
SRCSRVCMD=
SRCSRV: source files ---------------------------------------
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.cpp*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.h*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheMetaData.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheService.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.h*MYSERVER*widget/src/xpwidgets
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.cpp*MYSERVER*widget/src/xpwidgets
SRCSRV: end ------------------------------------------------

This data block was hard to nail down, because there was almost NO information on the web in regards to how to properly use http to extract source files. I hope that my blog can become a valuable resource in that regard.

The key is the variables section, the HTTP_EXTRACT_TARGET var tells the pdb where to grab the source file from. You'll notice keywords such as %var3%, this is referring to the lines in the source files section, where each var is seperated by an *. So for the first source file line, var3 is xpcom/io. You've probably noticed that I don't really need var1 and var2 in this instance, you're right. This data block was a modification of the existing data block in source 1's xul.pdb, i just took it and modified it.

Using the pdbstr.exe found in the Debugging Tools for Windows, I wrote the data block to source 2's xul.pdb. I ran srctool -f on the pdb to see if it was indexed, the response was that no files were indexed. I was suprised at this. When I tried to analyze the code in symbolstore.py (which I will have to modify in later releases), I came to the conclusion that the script simply gathered all the source file paths, wrote them to an auto-generated data block, and then wrote that data block to pdbstr. So when I tried to use this method manually, I was suprised to find out that I was unsuccessful. Perhaps my data block wasn't written properly.

I tried to test the data block instead on source 1. When I ran srctool on source 1's xul.pdb after inserting my own data block, it showed me that 6 files were indexed, and you'll noticed that there are 6 source files in the data block. To further check, I re-ran Visual Studio, and re-attached to firefox with source 1 as my source server (see the instructions above for further information on the process). Success! Clicking on the call to nsBaseAppShell.cpp in the Call Stack downloaded the source file from hgweb.

I still have a lingering question. Why was the source 1's xul.pdb able to index/work with my data block, while source 2's xul.pdb wasn't? I'm thinking that perhaps something is done to mozilla's symbols that I'm not aware of. Either I missed something in symbolstore.py or something happens outside of symbolstore.py that indexes the files.

Another observation I had was in regards to using Minefield. When debugging Minefield using source 2, all the source files were downloaded from my own machine. I guess the way pdb's work is that they first check for a local copy of the source file, and if it's not there they make the call to whatever remote site is specified in the data block.

My next objective for 0.2 is still to be determined. I'd also like to thank ted and lblakk for all the help they've given me with various aspects of this release.

Just one final note: My style has traditionally been to leave things alone and finishing them later (or to put it bluntly, procrastinating). I've learned through doing this release that for a project like this, that won't fly. Between having to study for midterms this week and doing this release, I feel lucky to have gotten everything done. For 0.2 and beyond, I plan on getting an early jump on things. That means smaller, more frequent blog posts instead of the essay I'm typing right now.

My biggest obstacle with this project so far was just getting started, because I absolutely knew nothing about it. Now that I've come to understand certain aspects of it, I feel a lot more enthusiastic about it, and I hope my future work reflects that enthusiasm.