Saturday, October 18, 2008

Mercurial Source Server Integration 0.1 Released!

You (the reader of course) will have to forgive my lack of updates on the 0.1 release, it's been a busy week with studying for midterms and whatnot. However, the good news is my 0.1 release has been completed. For instructions about how to set up the release, see this page

I guess since I never officially listed my objectives for this release I'll do it now.


  • Set up existing source server to see how it works(will download pdb's off mozilla server)

  • Set up localhost server

  • Build firefox and create pdb files (symbols), host them on local server

  • Insert my own data block into one of the mozilla pdb files (will use xul.pdb)

  • Get xul.pdb to grab source files off of hgweb instead of cvs repo



It was a long process trying to accomplish these tasks. The most daunting was trying to properly insert my data block into xul.pdb and have it work.

First I should point out that I had 2 locations for symbols set up. The first was a folder that stored the symbols i downloaded from http://symbols.mozilla.org/firefox (ill refer to this as source 1), the second was a folder that contained symbols i generated myself using make buildsymbols on my local version of minefield that I grabbed from hg's mozilla-central tree. I then copied these symbols to my local IIS server (ill refer to this as source 2).

My initial goal was to modify the xul.pdb in source 2 to grab the source files off of hgweb. To do this, I had to insert my own data block into xul.pdb. The data block is used to tell the symbol where to retrieve it's source files. Here is the data block I created


SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
SRCSRV: variables ------------------------------------------
HGSERVER=http://hg.mozilla.org/mozilla-central
SRCSRVVERCTRL=http
HTTP_EXTRACT_TARGET=%hgserver%/index.cgi/raw-file/7356c512e9e1/%var3%/%fnfile%(%var1%)
SRCSRVTRG=%http_extract_target%
SRCSRVCMD=
SRCSRV: source files ---------------------------------------
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.cpp*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.h*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheMetaData.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheService.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.h*MYSERVER*widget/src/xpwidgets
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.cpp*MYSERVER*widget/src/xpwidgets
SRCSRV: end ------------------------------------------------


This data block was hard to nail down, because there was almost NO information on the web in regards to how to properly use http to extract source files. I hope that my blog can become a valuable resource in that regard.

The key is the variables section, the HTTP_EXTRACT_TARGET var tells the pdb where to grab the source file from. You'll notice keywords such as %var3%, this is referring to the lines in the source files section, where each var is seperated by an *. So for the first source file line, var3 is xpcom/io. You've probably noticed that I don't really need var1 and var2 in this instance, you're right. This data block was a modification of the existing data block in source 1's xul.pdb, i just took it and modified it.

Using the pdbstr.exe found in the Debugging Tools for Windows, I wrote the data block to source 2's xul.pdb. I ran srctool -f on the pdb to see if it was indexed, the response was that no files were indexed. I was suprised at this. When I tried to analyze the code in symbolstore.py (which I will have to modify in later releases), I came to the conclusion that the script simply gathered all the source file paths, wrote them to an auto-generated data block, and then wrote that data block to pdbstr. So when I tried to use this method manually, I was suprised to find out that I was unsuccessful. Perhaps my data block wasn't written properly.

I tried to test the data block instead on source 1. When I ran srctool on source 1's xul.pdb after inserting my own data block, it showed me that 6 files were indexed, and you'll noticed that there are 6 source files in the data block. To further check, I re-ran Visual Studio, and re-attached to firefox with source 1 as my source server (see the instructions above for further information on the process). Success! Clicking on the call to nsBaseAppShell.cpp in the Call Stack downloaded the source file from hgweb.

I still have a lingering question. Why was the source 1's xul.pdb able to index/work with my data block, while source 2's xul.pdb wasn't? I'm thinking that perhaps something is done to mozilla's symbols that I'm not aware of. Either I missed something in symbolstore.py or something happens outside of symbolstore.py that indexes the files.

Another observation I had was in regards to using Minefield. When debugging Minefield using source 2, all the source files were downloaded from my own machine. I guess the way pdb's work is that they first check for a local copy of the source file, and if it's not there they make the call to whatever remote site is specified in the data block.

My next objective for 0.2 is still to be determined. I'd also like to thank ted and lblakk for all the help they've given me with various aspects of this release.

Just one final note: My style has traditionally been to leave things alone and finishing them later (or to put it bluntly, procrastinating). I've learned through doing this release that for a project like this, that won't fly. Between having to study for midterms this week and doing this release, I feel lucky to have gotten everything done. For 0.2 and beyond, I plan on getting an early jump on things. That means smaller, more frequent blog posts instead of the essay I'm typing right now.

My biggest obstacle with this project so far was just getting started, because I absolutely knew nothing about it. Now that I've come to understand certain aspects of it, I feel a lot more enthusiastic about it, and I hope my future work reflects that enthusiasm.

Monday, October 13, 2008

Working Towards my 0.1

I'm not exactly sure what I'll need to do to complete my 0.1, however one thing I know I need to do is get this source server working in CVS so that I can get a better idea of how it needs to work with Mercurial.

Getting the symbols to load was easy enough. All I had to do was add the location of Mozilla's symbols to Visual Studio.

Getting the source server to work properly has been slightly more tricky. Although it seemed easy enough in the Instructions I've been unable to retrieve the source files. My errors for the source files look like this:

SRCSRV: cvs.exe -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot checkout -r 3.68 -d 3.68 -N mozilla/widget/src/windows/nsAppShell.cpp
SRCSRV: Source server cannot retrieve the source code for file 'e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\windows\nsappshell.cpp' in module 'C:\Program Files\Mozilla Firefox\xul.dll'. The system cannot find the file specified.

It seems like I'm close enough to getting it to work, so I'm not worried. My biggest worry at the moment is not knowing where to start to get this thing working in Mercurial.

One clue I have is from downloading the Debugging Tools for Windows. Inside the srcsrv folder is a bunch of .PM files (perl modules), including ones for cvs and svn. Using a Document I found on Lukas's blog, I found this piece of info:

The .PM files are the Perl modules that can talk to the version control systems; the SSINDEX.CMD loads these modules and calls through a standard interface to isolate the system differences. The other CMD files, such as VSSINDEX.CMD, are wrappers for working with specific version control systems, as SSINDEX.CMD needs the version control system on the command line.


Does this mean I need to create a .PM file for Mercurial? I'm hoping more talks with Ted and Lukas will bring that to light.

Saturday, October 11, 2008

Lab 6: Thunderbird Bux Fix Lab

I found that building Thunderbird was easy enough. The challenge with this assignment was finding a regular expression that could properly match all of the following email address formats:

Eventually I realized that finding a regular expression was pointless since I wouldn't be able to actually use it, so I tried thinking of general rules that would make an email address invalid:

  1. 1. Periods can't be right next to eachother
  2. 2. An address can't end in a period
  3. 3. An address can't begin with an @
  4. 4. An address can't have a period right after the @


The next step was to find the relevant code. Using the hint given to us by Dave, I simply did a find on the word mailto and came across line 198 of the mozTXTToHTMLConv.cpp

if (inString.FindChar('.', pos) != kNotFound)
{
aOutString.AssignLiteral("mailto:");
aOutString += aInString;
}

This seemed to be the block of code that evaluated the e-mail address. I noticed that it only checked for a . right after the @ sign, so using the rules I established for a valid e-mail, I changed the if statement to this:

if (inString.FindChar('.', pos) != kNotFound && inString.Find("..", 0) == kNotFound && inString.CharAt(inString.Length() - 1) != '.')
{
aOutString.AssignLiteral("mailto:");
aOutString += aInString;
}

Now the if checks for a '.' right after the @, 2 '.' next to eachother, and a '.' in the last character of the string.

Still In Progress....

Tuesday, October 7, 2008

Lab 2: Firefox Build Lab

I know this is quite late, however I felt it would be better to at least do this lab and blog about it later rather than never.

What was hard?

I didn't find anything about building Firefox too hard. I had everything I needed to know in the documentation. I guess I could say that the trickiest part was figuring out what I needed to include in my .mozconfig file, but in the end this didn't cause me too much trouble.

What did you learn that you didn't know before?
Where do i start? I had the chance to use software I hadn't touched before in my life such as Mercurial and Minefield. I discovered how mozilla's build system worked and what was needed to grab, setup and run that build. Pretty much every aspect of this lab was something I had never done before.

How would you do it differently next time?

I don't think I could've approached this lab any better than how I did. I didn't really have any problems, I read the documentation thoroughly, so I don't think I really needed to do anything differently to be successful. I think if I ever built Firefox again it would be within Linux, as I've heard that it builds faster there than it does in Windows.

What advice would you give others trying this?

Read the instructions carefully. I had heard of others having problems performing this lab so I made sure that when I did it (finally) that I understood what was in the documentation. Because of that, I was able to complete the lab without any problems

Here is a screenshot of my build:




I know I haven't been a model student with these labs, however I fully intend to go back and complete the labs I've missed so I can be more prepared when doing my project (which I will also have updates on soon).
Note: I also made sure to submit my build times to Patrick Lam's wiki page.