How to migrate a CVS module to Git and GitHub

Since it took me a while to figure out, I figured this might be useful to document: migrating code from CVS to Git. Specifically, I was moving modules in a CVS repository on SourceForge over to GitHub.

Here are the versions of the tools that I used:

$ rsync --version | head -1
rsync version 3.1.0 protocol version 31
$ cvs --version | head -2 | tail -1
Concurrent Versions System (CVS) 1.12.13 (client/server)
$ git --version
git version 2.6.3
$ cvsps -V
cvsps: version 3.13

First, I grabbed a copy of the CVSROOT, and checked out the module so I had a reference copy to compare to when I’m done.

$ rsync -av aolserver-cvs
$ cvs -d $(pwd)/aolserver-cvs co -d nsmysql-cvs nsmysql

Then, I create the working directory for the local git repo.

$ mkdir nsmysql
$ git init nsmysql

Next, do the actual CVS-to-Git conversion.

$ cvsps --root $(pwd)/aolserver-cvs nsmysql --fast-export | git --git-dir=nsmysql/.git fast-import

Finally, do a diff to compare the two working directories to make sure the import worked correctly.

$ cd nsmysql
$ git checkout master
$ diff -x .git -x CVS -urN . ../nsmysql-cvs

If everything looks good, go ahead and push it up to GitHub.

$ git remote add origin
$ git config remote.origin.push HEAD
$ git push -u origin master

I don’t do this often, but when I do, I always have to figure it out each time, so hopefully next time I’ll find this blog post at the top of my search results and save myself some time.

AOLserver in Google’s Summer of Code 2009

Google Summer of Code 2009

Are you a student that meets Google’s eligibility criteria, who is interested in contributing to an Open Source project this summer? Google’s Summer of Code 2009 is on, and once again, the Tcl community has been accepted as a mentoring organization, which includes AOLserver.

Students can apply until the deadline of 19:00 UTC on April 3rd. If you have any questions about GSoC or AOLserver’s involvement in it, please ask me! I’ll try to help answer whatever questions you might have.

Good luck to all the students and mentoring organizations this summer!

Tags: , , ,

Netcraft 2008 survey shows AOLserver is far from dead

The Netcraft February 2008 Web Server Survey says:

Unusually, America Online’s open source AOLserver sees tremendous growth, jumping from 35 thousand to 105 thousand sites in just one month. AOLserver is a multithreaded, Tcl-enabled web server which can be used for large scale, dynamic web sites, but has not seen the release of a new version since 2006. The majority of the new sites served by AOLserver are hosted in Poland.

This isn’t going to make any headlines, but for all those doubters out there who keep wondering who actually uses AOLserver, check out that growth.

The sad thing is, we could easily game the Netcraft survey by doing a few simple things:

  1. Register to become an ICANN accredited domain registrar.
  2. Offer domain registration at-cost, to offer the lowest possible prices to get customers.
  3. Offer free web domain parking or static file only hosting, all on AOLserver.

The costs involved would be around $10K for the first year, plus the cost of the actual AOLserver hosting, plus the $70K working capital in reserve to meet ICANN’s requirements. This could all be set up on Amazon EC2/S3 to avoid having to provision real hardware as the customer demand grows.

Of course, what would be the point? Would having more significant numbers in the Netcraft surveys give AOLserver more credibility? I sure hope not–that would be foolish.

(via Mark Mcgaha)

nscgi: ns_request_cgi and REQUEST_URI change

In the “scratch your own itch” department, I decided to get WordPress working under AOLserver. WordPress is written in PHP, and while AOLserver can load PHP as a module, it doesn’t play nice when you configure AOLserver with multiple virtual servers. The other option, running PHP as CGI using nscgi and php-cgi, does work for the most part. However, there were a few issues I ran into that required changes to nscgi, which I’ll outline below.

Adding REQUEST_URI to CGI env. variables

PHP expects an additional environment variable, REQUEST_URI, to be included on CGI requests. Of course, this isn’t part of the CGI/1.1 specification. Basically, PHP expects REQUEST_URI to contain the Request-URI (from the HTTP/1.1 specification, see section 5.1.2). So, I added this in.

Mapping URLs to CGI with ns_register_cgi

In order to map an URL to be handled by nscgi, you previously had to make the change in your config .tcl and restart AOLserver. One of the great things about AOLserver is the fact that you can modify some settings at runtime, without a restart. So, I looked at whether it would be possible to add additional mappings at runtime, and it appears that under the hood, nscgi also uses the Ns_RegisterRequest C API which is supposed to be safe to execute at runtime. So, I added a new Tcl command, ns_register_cgi, which is now part of nscgi. It is just a wrapper around CgiRegister, and thus takes the same argument.

For example, if you wanted to map all requests to “/foo” (including “/foo/bar“) to a PHP script “/var/www/myfoo.php“, you would use something like this:

foreach method {GET POST HEAD} {
    ns_register_cgi "$method /foo /var/www/myfoo.php"

This maps GET, POST and HEAD requests to URLs starting with “/foo” to the script specified.

Caveat: It is currently possible to load the nscgi module multiple times in the same AOLserver process, for each virtual server. It is also possible to load it multiple times for the same virtual server. I’m not sure why anyone would do that, but it’s possible. In that circumstance, the behavior of ns_register_cgi is undefined–as in, I haven’t tested it. It may or may not work correctly. If you run such a configuration, where nscgi is loaded multiple times in the same virtual server, please test this carefully and share your findings. Thanks.

Where’s the code?

I’ve commited the changes to SourceForge CVS, both in HEAD (rev 1.33, diff) and the aolserver_v40_bp branch (rev, diff).

Tags: , ,

Not quite an AOLserver roadmap

I have way too many saved emails in my inbox, over 5,500 when I started writing this. Of those, over 2,200 are AOLserver-related. I haven’t them filed away yet because they represent a loosely organized “to do” list for me. But, with so many open items, it’s a daunting task to go pluck out interesting things to work on, so I’ve mostly avoided doing it and haven’t gotten anything done.

I’ve wanted to get my inbox back down to a manageable size for a while, so I’m going to go through it and create a high-level summary of the open issues so I can finally file all those old emails (dating back to 2004, some of them) away.

On with the list …

  • Making everything binary safe and aware (i.e., nsdb, ns_set, etc.)
  • Review nsodbc, the ODBC nsdb driver, make sure it works on Win32 as well as with unixODBC
  • Review nsperm, the stock HTTP authentication implementation that comes with AOLserver out-of-the-box
  • Clean up the nsmysql driver, properly license it with the AOLserver Public License (or perhaps some other license) so it can be included in Debian, etc.
  • Fix the nslog hack to handle X-Forwarded-For headers, make it do the right things
  • Commit the changes I’ve been sitting on for nsopenssl and tag a new release of it
  • Review Eric Lorenzo’s tclprof (OpenACS thread here), get it checked into CVS, consider Jamie Rasmussen’s suggestions for changes, and rationalize it with the existing nsprofile module
  • Start on a manuscript for an AOLserver book

… and this is only a partial list. I still have plenty of emails to go through in my inbox–I’m only down to 5,100 messages, now.

Of course, if you’d like to help out by tackling any of the items on the above list, don’t let me stop you–go right ahead! If you want to know more about anything in particular, just ask.


Server-Side JavaScript with jQuery and AOLserver at jQueryCamp07

Dossy presenting at jQueryCamp07

Here are the slides for my presentation, Server-Side JavaScript with jQuery and AOLserver, that I gave at jQueryCamp07 today.

Title slide screenshot

Tags: , , , ,

Progress on nsjsapi running jQuery under AOLserver

It’s not fully baked yet, but let me take a break from coding to … squee! You have no idea how excited I am right now.

Back in February, I blogged about nsjsapi, an AOLserver module that makes the SpiderMonkey JavaScript interpreter (written in C) available to Tcl. Then, in July, John Resig blogged about env.js, which he used to load jQuery under the Rhino JavaScript interpreter (written in Java). env.js is neat because it’s a JavaScript implementation of a basic browser and DOM environment which isn’t provided by either SpiderMonkey or Rhino.

However, env.js assumes its running under Rhino, or at least has all the java.* and javax.* and org.w3c.dom.* APIs available, so it doesn’t completely work under SpiderMonkey. So, I’ve set out to implement all those missing bits in JavaScript, in nsjsapi.js. (Yes, I know I’m crazy …)

After a few days of furious coding and much head-desking, I’ve gotten far enough that I can do this from an AOLserver control port:

nscp 1> set js [js create]
nscp 2> js load $js /home/dossy/htdocs/js/nsjsapi.js
nscp 3> js load $js /home/dossy/htdocs/js/env.js
nscp 4> js load $js /home/dossy/htdocs/js/jquery-1.2.1.js
nscp 5> js eval $js {jQuery.fn.toString = DOMNodeList.prototype.toString; true;}
nscp 6> js eval $js {window.location = 'test/index.html';}
nscp 7> js eval $js {$('body').html()}
<div><span id='foo'>This is a test.</span></div>
nscp 8> js eval $js {$('div').append('<span>Hello world!</span>')}
[ <div> ]
nscp 9> js eval $js {$('body').html()}
<div><span id='foo'>This is a test.</span><span>Hello world!</span></div>

I guess this doesn’t seem like much of an accomplishment, but it’s a huge first step towards being able to do actual JS-based server-side web development under AOLserver.

I want to iron out a strange crash bug when the JS interp. is destroyed via [js destroy], and complete the file I/O stub that isn’t finished yet, then check in nsjsapi.js. I plan to get this done in time for my talk at jQueryCamp07 this weekend.

Tags: , , ,

Running AOLserver … on the iPhone?

Apparently Eric O’Laughlen has managed to compile AOLserver for the iPhone. Then, he went and forked the code into what he’s calling “Joggame Server“. (If you thought “AOLserver” was a bad name, it’s nothing compared to that!)

I haven’t jumped on the iPhone bandwagon, sticking with my old Treo 650 for now, but this makes me wish I had access to one. Running web applications locally on the iPhone opens the door to applications that are usable online as well as offline. Of course, someone has to write such applications, first, which remains to be seen.

Still, maybe this will help get some new folks interested in AOLserver and that can only be a good thing!

Tags: ,

Enough of the politics, on with the geeking!

Okay, I’m sorry I indulged myself the last few days with the spurt of political blogging. I’ve gotten it out of my system now. Time to get back to the hardcore geeking!

Here’s a short list of things that are either on my mind or somewhere in my to-do list:

  • Commit my changes to AOLserver to build easily with MinGW/MSYS on Win32. I even put together a quick NSIS script, so there’s a nice one-click installer for AOLserver 4.0.10 on Win32, now.
  • Commit my changes to Gnash to build and link correctly using MinGW/MSYS on Win32. No fancy installer, but here’s my previous blog post about it.
  • Do some more work on nsjsapi, the AOLserver module to integrate Mozilla SpiderMonkey into AOLserver for server-side JavaScript. I’d like to get it working well enough to load and use jQuery so I can show it off at jQueryCamp07 this October.
  • Work with the community to put together a list of “Top 5 Goals” for AOLserver, then assemble a real plan to get those things done. A big bonus would be if I didn’t have to do everything myself.
  • Find someone who wants to check out Gobby with me. It’s a free, open source, multi-platform, collaborative real-time file editing application with chat and syntax highlighting. Think: World of Notepad. Could be a nice way to do remote paired programming with two or more people. Or, a good way to do code reviews with a geographically distributed team.

If you’d like to hear more about any of these things–or better yet, do them together with me, let me know!

64-bit AOLserver on Debian 4.0 (etch) Linux works

John Buckman pointed out in this thread that AOLserver wasn’t working right on his 64-bit Linux box that he wants to upgrade BookMooch to. I offered to help, but pointed out that I don’t have a machine to test on–so, he graciously provided me temporary access to one of his machines to diagnose the problem.

After some poking and prodding, I believe the correct change to get AOLserver to build is to edit include/ns.mak on line 79, after running configure which generates it from Change the line from:

LDLIB           = ${CC} -shared ${CFLAGS} ${LDFLAGS}


LDLIB           = ${CC} -nostartfiles -shared ${CFLAGS} ${LDFLAGS}

Simply add the “-nostartfiles” after “${CC}” in the LDLIB variable. I reported this to the AOLserver mailing list as well.

John confirms that he’s now able to run AOLserver on his luscious new 64-bit 2-CPU Xeon E5335 Clovertown quad-core box.

In return, I asked John to do me a favor and perform a simple benchmark, fetching a 4K static file from AOLserver using ApacheBench, and here’s what you might be able to expect:

# ab -c 50 -n 50000 http://.../x.txt
Server Software:        AOLserver/4.5.0
Server Hostname:        ...
Server Port:            80

Document Path:          /x.txt
Document Length:        4070 bytes

Concurrency Level:      50
Time taken for tests:   3.281485 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      217654353 bytes
HTML transferred:       203504070 bytes
Requests per second:    15237.00 [#/sec] (mean)
Time per request:       3.281 [ms] (mean)
Time per request:       0.066 [ms] (mean, across all concurrent requests)
Transfer rate:          64773.42 [Kbytes/sec] received

No, that’s not a typo: 15,237 requests per second. This is without any tuning or tweaking of the IP stack or anything. Of course, 15K req/sec at 4K per request is ~218MB–over a span of 3.2 seconds is effectively 544 Mbps, so you’re more likely to run out of upstream network bandwidth than run out of horsepower serving static assets out of AOLserver on this kind of hardware.

Naturally, you shouldn’t epxect this kind of throughput on dynamic requests and benchmarking those isn’t really meaningful since the nature of your application code will greatly affect the request execution time. However, it’d be fun to do a “web server shootout” where each implements a trivial piece of dynamic functionality in their best environment–Apache with mod_perl, mod_python, FastCGI/PHP, etc. vs. AOLserver with Tcl–and see which one has the least amount of overhead.

Still, it’s great news that John’s BookMooch service is going to be able to move to a beefy 64-bit box, and I’m very glad to know that 64-bit AOLserver does run just fine for at least one application on Debian Linux.

Tags: , , , ,