0.3 Release Submitted

Well, I've submitted my third patch, which I will be considering as my 0.3 release. Whether or not it gets accepted remains to be seen but I feel like I've done enough to submit a good 0.3.

Most of the issues I needed to correct were minor things such as naming conventions, however there were also some other issues that ted pointed out that I needed to fix.

The first was a change to the data block. Ted didn't like the fact that the SRCSRVTRG variable was set to HTTP_EXTRACT_TARGET. Instead he wanted this variable to point to a local target because SRCSRVTRG was supposed to resolve to the path that the file is extracted to. What I found is that while this is true for a CVS extraction, I believe it serves a different purpose when used for HTTP extraction.

When I run srctool on a pdb using the old CVS implementation, I see entries like this:

[e:\fx19rel\winnt_5.2_depend\mozilla\layout\style\nsrulenode.h] cmd: cvs.exe -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot checkout -r 1.64 -d 1.64 -N mozilla/layout/style/nsRuleNode.h

As you can see, the header file has a "cmd" which is run to retrieve the source file. The SRCSRVTRG is then set as a local path to tell srcsrv where to dump the file. However, in my HTTP implementation srctool showed this:

[c:\mozilla\src\toolkit\xre\nsupdatedriver.h] trg: http://hg.mozilla.org/mozilla-central/raw-file/3611062ed098/toolkit/xre/nsUpateDriver.h 

As you can see, the files no longer use a "cmd" but a "trg". I wasn't sure if this was related to HTTP_EXTRACT_TARGET or SRCSRVTRG so I experimented by changing SRCSRVTRG to ted's recommendation, and srctool showed this:

[c:\mozilla\src\toolkit\xre\nsupdatedriver.h] trg: C:\mozilla\symbols\xul.pdb\.... 

The trg was now the location of the xul.pdb file. This showed me that SRCSRVTRG was being used to determine where the file was located, and thus was being used differently than if the source server was using CVS. Testing in Visual Studio proved this, as the source server didn't know where to find the file. So, to sum things up, SRCSRVTRG had to remain the way it was.

I was also asked to remove the SRCSRVCMD command from the data block, as it wasn't being set to anything. I assumed I needed it as the srcsrv documentation said it was a required variable, but it turns out that I didn't need it so I was happy to remove it.

So, I was ready to submit a new patch when I found out that my entire solution didn't work in Visual Studio 2005! I had been testing on 2008 the entire time and it worked like a dream, however 2005 didn't seem to know how to do HTTP retrieval. After discussing this with ted and sending him my stuff, he got it working fine in WinDBG (I had tried WinDBG myself without any success but I must have done something wrong in the setup) so in the end not working in 2005 was ok with ted, he speculated that perhaps srcsrv wasn't properly implemented in 2005 and I'm inclined to agree based on what I've seen in my testing.

So, now that I've resolved these issues I feel confident about getting my project implemented in the tree. Maybe not with this patch, but eventually.

0.3 Update #2, failed review again :(

Well, it seems ted has found some more problems with my patch so I'll have to try again. I'm still very optimistic that I can get this done on time however I'm in the middle of exam week so it will take some time before my next patch is created. Stay tuned.

0.3 Update

Well, it's been awhile since my last post but I'm in overdrive mode for some of my other classes so unfortunately I've gotten a little behind on my 0.3. Luckily, I believe I'm close to having my patch accepted.

I guess I should explain that my first review didn't go so well. Ted pointed out a few things that he didn't like about my patch (one of which was a dumb mistake on my part) but fortunately none of the required changes was difficult (mostly naming conventions and a syntax error inside of the data block) and Ted even mentioned that my patch was close. So, I've uploaded my next patch and I'm hoping for the best.

0.2 - Finished

Well, after a lot of trial and error I got my local server to serve the pdb files properly.

The big problem I was having was that although the symbols were being loaded, when I tried to retrieve source code it would only ask me to locate my local source files and not try to retrieve the source files from hgweb. I did a lot of searching, checking, fixing etc.

It just so happens that I lucked out bigtime, as I finally noticed that the symbols being loaded when I started up Minefield weren't coming from my symbol server. Instead, the symbols being loaded were symbols inside my object directory. I'm not sure when these symbols got created but they were scattered throughout the object directory, and unlike the symbols I was using, they weren't indexed. So, to test this out I deleted the xul.pdb in the object directory and re-attached to Minefield. Success! Now xul.dll was loading the pdb from my server instead of the objdir one, the source code was being pulled from hgweb and all was well.

Now, I'm not sure what my deliverable should be. Perhaps it should just be the new pdb's I've written, or the new symbolstore.py even. Or, maybe I've done enough to submit a patch.

EDIT: I have filed a patch under Bug 440001

0.2 Release - Lots of Progress

Firstly, to add to my last post I decided to set my deliverable for 0.2

• Hack symbolstore.py to index the pdb files


• Serve the pdb files over my server and make sure they successfully retrieve source code from hgweb

Well, I'm happy to report that I've managed to accomplish the first of these tasks. A big thank you to lsblakk and ted for their help.

After reading up on Python, I came to understand a lot more about how symbolstore.py works.

Basically, it parses out a list of source files from each .sym file, writes a .stream file using the list of source files and calls pdbstr to write the data block to the pdb files.

Fortunately for me, lsblakk's previous modifications to symbolstore.py to make it work with CVS gave me all of the code I needed. All that was required from me was to make some slight changes.

Firstly, I needed to change the SourceIndex() method to write my data block instead of the one used previously for CVS. My new method is listed below:

def SourceIndex(fileStream, outputPath, hg_root):
    """Takes a list of files, writes info to a data block in a .stream file"""
    # Creates a .pdb.stream file in the mozilla\objdir to be used for source indexing
    # Create the srcsrv data block that indexes the pdb file
    result = True
    pdbStreamFile = open(outputPath, "w")
    pdbStreamFile.write('''SRCSRV: ini ------------------------------------------------\r\nVERSION=2\r\nINDEXVERSION=2\r\nVERCTRL=http\r\nSRCSRV: variables ------------------------------------------\r\nHGSERVER=http://hg.mozilla.org/mozilla-central''')
    # pdbStreamFile.write(hg_root)
    pdbStreamFile.write('''\r\nSRCSRVVERCTRL=http\r\nHTTP_EXTRACT_TARGET=%hgserver%/index.cgi/raw-file/%var3%/%var2%\r\nSRCSRVTRG=%http_extract_target%\r\nSRCSRVCMD=\r\nSRCSRV: source files ---------------------------------------\r\n''')
    pdbStreamFile.write(fileStream) # can't do string interpolation because the source server also uses this and so there are % in the above
    pdbStreamFile.write("SRCSRV: end ------------------------------------------------\r\n\n")
    pdbStreamFile.close()
    return result

Secondly, I simply needed to change the ProcessFile method inside of the Dumper class to look for hg files instead of cvs files. So from:

if filename.startswith("cvs"):

To:

if filename.startswith("hg"):

Like I said, no crazy coding required.

I ran into problems when trying to run 'make buildsymbols'. For some reason empty pdb files were being created, so I concluded that symbolstore.py wasn't running properly. I eventually discovered that the makefile wasn't running symbolstore.py using the -i flag. This flag is what tells symbolstore.py to add source file information to the pdb files.

I discovered that there was actually code in the makefile to add this flag:

ifdef PDBSTR_PATH
MAKE_SYM_STORE_ARGS += -i
endif

So, the reason that the pdb files were empty was because PDBSTR_PATH wasn't being set, so I simply added it to my environment variables. After that, voila! The pdb files were successfully indexed upon creation.

My next step is to serve these pdb files from a server and make sure that the source files are being retrieved from hgweb. I don't think I can do this with a localhost server because the debugger will find my local source files before it tries to look on hgweb. So, I need to find a place to serve my pdb files (I would like to find a remote server to do it so that I can maybe offer a way for people to test it).

0.2 Release Update

I've been working on some preliminary tasks in preparation for my 0.2 release. These tasks include learning more about Python and learning more about how symbolstore.py works. So far I think I have a decent grasp of both.

I'm thinking my deliverable for this release will be an updated version of symbolstore.py. I'm not sure if 0.2 is going to be able to solve the full problem or if it will simply solve a portion of it, but I will try to shoot for something that indexes all of the source files.

Stay tuned.

FSOSS Report

My paper has been finished, you can find it here. Overall FSOSS was a very interesting experience, and I definitely enjoyed it. I especially enjoyed the Mozilla and Mobile, Komodo and Open Source Design presentations.

Lab 7: Modifying Firefox

This was a pretty interesting lab. I really enjoyed being able to see my own changes to the Firefox code affecting the actual browser.

It required me to dig a little deeper into the code than I am used to, fortunately MXR is always there to help me out with that.

Using the recommended changes from the wiki, I found that there were still some bugs. There was a bug when trying to delete a tab when there was less than 3, and also when I tried to create a new tab when there was only one open it would create 2 instead of 1.

Mercurial Source Server Integration 0.1 Released!

You (the reader of course) will have to forgive my lack of updates on the 0.1 release, it's been a busy week with studying for midterms and whatnot. However, the good news is my 0.1 release has been completed. For instructions about how to set up the release, see this page

I guess since I never officially listed my objectives for this release I'll do it now.

• Set up existing source server to see how it works(will download pdb's off mozilla server)


• Set up localhost server


• Build firefox and create pdb files (symbols), host them on local server


• Insert my own data block into one of the mozilla pdb files (will use xul.pdb)


• Get xul.pdb to grab source files off of hgweb instead of cvs repo

It was a long process trying to accomplish these tasks. The most daunting was trying to properly insert my data block into xul.pdb and have it work.

First I should point out that I had 2 locations for symbols set up. The first was a folder that stored the symbols i downloaded from http://symbols.mozilla.org/firefox (ill refer to this as source 1), the second was a folder that contained symbols i generated myself using make buildsymbols on my local version of minefield that I grabbed from hg's mozilla-central tree. I then copied these symbols to my local IIS server (ill refer to this as source 2).

My initial goal was to modify the xul.pdb in source 2 to grab the source files off of hgweb. To do this, I had to insert my own data block into xul.pdb. The data block is used to tell the symbol where to retrieve it's source files. Here is the data block I created

SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
SRCSRV: variables ------------------------------------------
HGSERVER=http://hg.mozilla.org/mozilla-central
SRCSRVVERCTRL=http
HTTP_EXTRACT_TARGET=%hgserver%/index.cgi/raw-file/7356c512e9e1/%var3%/%fnfile%(%var1%)
SRCSRVTRG=%http_extract_target%
SRCSRVCMD=
SRCSRV: source files ---------------------------------------
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.cpp*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\io\SpecialSystemDirectory.h*MYSERVER*xpcom/io
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheMetaData.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\netwerk\cache\src\nsCacheService.h*MYSERVER*netwerk/cache/src
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.h*MYSERVER*widget/src/xpwidgets
e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsBaseAppShell.cpp*MYSERVER*widget/src/xpwidgets
SRCSRV: end ------------------------------------------------

This data block was hard to nail down, because there was almost NO information on the web in regards to how to properly use http to extract source files. I hope that my blog can become a valuable resource in that regard.

The key is the variables section, the HTTP_EXTRACT_TARGET var tells the pdb where to grab the source file from. You'll notice keywords such as %var3%, this is referring to the lines in the source files section, where each var is seperated by an *. So for the first source file line, var3 is xpcom/io. You've probably noticed that I don't really need var1 and var2 in this instance, you're right. This data block was a modification of the existing data block in source 1's xul.pdb, i just took it and modified it.

Using the pdbstr.exe found in the Debugging Tools for Windows, I wrote the data block to source 2's xul.pdb. I ran srctool -f on the pdb to see if it was indexed, the response was that no files were indexed. I was suprised at this. When I tried to analyze the code in symbolstore.py (which I will have to modify in later releases), I came to the conclusion that the script simply gathered all the source file paths, wrote them to an auto-generated data block, and then wrote that data block to pdbstr. So when I tried to use this method manually, I was suprised to find out that I was unsuccessful. Perhaps my data block wasn't written properly.

I tried to test the data block instead on source 1. When I ran srctool on source 1's xul.pdb after inserting my own data block, it showed me that 6 files were indexed, and you'll noticed that there are 6 source files in the data block. To further check, I re-ran Visual Studio, and re-attached to firefox with source 1 as my source server (see the instructions above for further information on the process). Success! Clicking on the call to nsBaseAppShell.cpp in the Call Stack downloaded the source file from hgweb.

I still have a lingering question. Why was the source 1's xul.pdb able to index/work with my data block, while source 2's xul.pdb wasn't? I'm thinking that perhaps something is done to mozilla's symbols that I'm not aware of. Either I missed something in symbolstore.py or something happens outside of symbolstore.py that indexes the files.

Another observation I had was in regards to using Minefield. When debugging Minefield using source 2, all the source files were downloaded from my own machine. I guess the way pdb's work is that they first check for a local copy of the source file, and if it's not there they make the call to whatever remote site is specified in the data block.

My next objective for 0.2 is still to be determined. I'd also like to thank ted and lblakk for all the help they've given me with various aspects of this release.

Just one final note: My style has traditionally been to leave things alone and finishing them later (or to put it bluntly, procrastinating). I've learned through doing this release that for a project like this, that won't fly. Between having to study for midterms this week and doing this release, I feel lucky to have gotten everything done. For 0.2 and beyond, I plan on getting an early jump on things. That means smaller, more frequent blog posts instead of the essay I'm typing right now.

My biggest obstacle with this project so far was just getting started, because I absolutely knew nothing about it. Now that I've come to understand certain aspects of it, I feel a lot more enthusiastic about it, and I hope my future work reflects that enthusiasm.

Working Towards my 0.1

I'm not exactly sure what I'll need to do to complete my 0.1, however one thing I know I need to do is get this source server working in CVS so that I can get a better idea of how it needs to work with Mercurial.

Getting the symbols to load was easy enough. All I had to do was add the location of Mozilla's symbols to Visual Studio.

Getting the source server to work properly has been slightly more tricky. Although it seemed easy enough in the Instructions I've been unable to retrieve the source files. My errors for the source files look like this:

SRCSRV: cvs.exe -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot checkout -r 3.68 -d 3.68 -N mozilla/widget/src/windows/nsAppShell.cpp
SRCSRV: Source server cannot retrieve the source code for file 'e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\windows\nsappshell.cpp' in module 'C:\Program Files\Mozilla Firefox\xul.dll'. The system cannot find the file specified.

It seems like I'm close enough to getting it to work, so I'm not worried. My biggest worry at the moment is not knowing where to start to get this thing working in Mercurial.

One clue I have is from downloading the Debugging Tools for Windows. Inside the srcsrv folder is a bunch of .PM files (perl modules), including ones for cvs and svn. Using a Document I found on Lukas's blog, I found this piece of info:

The .PM files are the Perl modules that can talk to the version control systems; the SSINDEX.CMD loads these modules and calls through a standard interface to isolate the system differences. The other CMD files, such as VSSINDEX.CMD, are wrappers for working with specific version control systems, as SSINDEX.CMD needs the version control system on the command line.

Does this mean I need to create a .PM file for Mercurial? I'm hoping more talks with Ted and Lukas will bring that to light.

Lab 6: Thunderbird Bux Fix Lab

I found that building Thunderbird was easy enough. The challenge with this assignment was finding a regular expression that could properly match all of the following email address formats:

Eventually I realized that finding a regular expression was pointless since I wouldn't be able to actually use it, so I tried thinking of general rules that would make an email address invalid:

1. 1. Periods can't be right next to eachother
2. 2. An address can't end in a period
3. 3. An address can't begin with an @
4. 4. An address can't have a period right after the @

The next step was to find the relevant code. Using the hint given to us by Dave, I simply did a find on the word mailto and came across line 198 of the mozTXTToHTMLConv.cpp

if (inString.FindChar('.', pos) != kNotFound)
{
aOutString.AssignLiteral("mailto:");
aOutString += aInString;
}

This seemed to be the block of code that evaluated the e-mail address. I noticed that it only checked for a . right after the @ sign, so using the rules I established for a valid e-mail, I changed the if statement to this:

if (inString.FindChar('.', pos) != kNotFound && inString.Find("..", 0) == kNotFound && inString.CharAt(inString.Length() - 1) != '.')
{
aOutString.AssignLiteral("mailto:");
aOutString += aInString;
}

Now the if checks for a '.' right after the @, 2 '.' next to eachother, and a '.' in the last character of the string.

Still In Progress....

Lab 2: Firefox Build Lab

I know this is quite late, however I felt it would be better to at least do this lab and blog about it later rather than never.

What was hard?

I didn't find anything about building Firefox too hard. I had everything I needed to know in the documentation. I guess I could say that the trickiest part was figuring out what I needed to include in my .mozconfig file, but in the end this didn't cause me too much trouble.

What did you learn that you didn't know before?

Where do i start? I had the chance to use software I hadn't touched before in my life such as Mercurial and Minefield. I discovered how mozilla's build system worked and what was needed to grab, setup and run that build. Pretty much every aspect of this lab was something I had never done before.

How would you do it differently next time?

I don't think I could've approached this lab any better than how I did. I didn't really have any problems, I read the documentation thoroughly, so I don't think I really needed to do anything differently to be successful. I think if I ever built Firefox again it would be within Linux, as I've heard that it builds faster there than it does in Windows.

What advice would you give others trying this?

Read the instructions carefully. I had heard of others having problems performing this lab so I made sure that when I did it (finally) that I understood what was in the documentation. Because of that, I was able to complete the lab without any problems

Here is a screenshot of my build:

I know I haven't been a model student with these labs, however I fully intend to go back and complete the labs I've missed so I can be more prepared when doing my project (which I will also have updates on soon).

Note: I also made sure to submit my build times to Patrick Lam's wiki page.

A Little Late But...My First Post

So, I've finally gotten around to making a blog post, better late then never I suppose.

I've been thinking about my potential project for the last couple of days now, I've narrowed my choices down to two: adding source server support to Mercurial and creating a self-serve symbol upload system.

On the one hand, I've been leaning towards web development for the last year or so, so creating a web-based system definitely appeals to me and would allow me to perhaps brush up on my PHP skills(although I'm not entirely sure how I would approach the project yet). On the other hand, taking the Mercurial project would give me a great resource in Lukas(who worked on the original project) as well as ted, who I have yet to talk to. Also, the Mercurial project sounds like a more important project at first glance, although I can't be too sure about that.

Hopefully I'll be able to make a decision within the next couple of days, in the mean time I'll be updating this blog soon with my lab results.

Test

this is a test post