Janine Sisk at Furfly has been working hard to migrate her customers to the latest AOLserver 4.0.x release, which has been helping the project find bugs and answer migration questions. This has been incredibly helpful and the most recent issue involved encoding and character set settings.

Her problem, in a nutshell, is that users are entering data into the system originally authored in Microsoft Word which uses the Microsoft Windows Codepage 1252 character set which contains codepoints for things like en-dashes (0x96 = U+2013), ellipses (0x85 = U+2026), and some other elements. However, if you treat CP1252-encoded data as UTF-8 (RFC 2279), it won’t render correctly: all bytes in the range of 0x80-0xFF, which contains en-dashes and ellipses, are encoded using two bytes instead of just one. In this particular case, the en-dashes become transcoded from 0x96 to the sequence 0xC2 0x96. The 0xC2 looks like this — Â or 쎂 — depending on which one your browser rendered correctly. You may recognize it and wonder how it got in your own pages and how to remove it, too.

For Janine, her solution was to instruct AOLserver to use the ISO-8859-1 character set as the output character set, which would mean the inputs by the users originally in CP1252 would get output from the server as ISO-8859-1 which is a close enough match, I guess. The configuration might look like this:

ns_section "ns/parameters"
ns_param OutputCharset iso-8859-1

I’m not sure what she has the other related encoding and character set settings set to, but I’ve asked and hopefully will find out and update this entry with the details.

I’ve been doing a lot of core dump analysis at work on Solaris, digging deep into the bag of coder kung-fu using adb, which is a debugger that is not for the faint of heart.

So, trying to help someone troubleshoot a problem with AOLserver on Linux, I asked the simple question of “did it drop a core file”? The answer was, “No, it didn’t.” That prompts the next question, “Did you set ulimit -c unlimited?” The affirmative answer surprised me: how is it not dropping a core file? Ahh, that’s when it occurred to me: Linux has had issues with threads (same info, more polished) for a long time now. So, I decided to do some hacking on my own Debian Linux box running my own hand-compiled 2.6.7 kernel.

Apparently, there are some kernel patches to implement multithreaded core dumps which suggests that the patches were merged into the 2.5.47 kernel. So, why isn’t AOLserver writing a core file when it segfaults?

After making dinner and putting the kids to bed, I thought about it and wrote a little test program that starts a thread then aborts in the thread. It wrote out a core file just fine! Then it dawned on me: AOLserver does a setuid to drop root privileges — Linux, by default, won’t write a core file out if the process has changed uid/euid. Figuring I wasn’t the first person to want Linux to behave like every other Unix out there, I found my answer in the unofficial comp.os.linux.development.* FAQ: How can I make a suid executable dump core? Making the change in nsd/nsmain.c, adding the prctl(PR_SET_DUMPABLE) call, I can now get AOLserver to dump a core file on Linux. Yay.

I’ve just filed SF Bug #1031599, attached the diff’s and committed the change.

While this doesn’t solve the original problem of why this person’s AOLserver is crashing, at least now I might be able to get a core file to look at …

OK, I don’t know how I possibly missed this, but last month Computer Associates has released its Ingres r3 database product as open source software!

Now, I’ve been using MySQL for many years, and have been recently considering using PostgreSQL in order to get features that MySQL is still lacking (views, stored procedures, etc.) — but now, I can use Ingres instead!

You can download Ingres r3 today! (Requires free site registration.) The current source tarball is ~76MB. I’ll be downloading it tonight … expect to see an AOLserver nsingres module soon.

I don’t know if you can tell … but damn, I’m excited …

David S. Miller, a world-reknowned Linux kernel hacker, key contributor to the sparclinux/sparc64 port, and all-around nice guy, wrote to debian-sparc a great message that starts:

You can laugh and call it Wintendo all you want, but even the I/O performance of modern x86 (and Opteron) systems blows Sparc away.

Later on, he writes:

[…] If I’m setting up a high end web server,
I don’t think I’d be using Sparc thanks :)

If I wanted a fast machine with good cpu performance and a fast disk
and networking, I’d get a 3+ Ghz Xeon or an Opteron with PCI-X or
perferably PCI Express. […]

Read the whole message to get a good explanation of why he feels this way and the numbers behind his assertions.

Okay, I’ve gone and implemented some ghetto user registration functionality and hooked it up to the blog’s comment submission code, so you can leave comments on the blog again!

If you run into any kinks or errors, well, please let me know by sending me email. Thanks!

Back on August 7th, Nathaniel Haggard reports a problem with nsopenssl where it repeatedly sends a bunch of stuff into the server log. Janine Sisk confirms that she is also seeing the same thing. However, neither were really able to put a finger on why it was happening or how to reproduce it, so I couldn’t really do much about it at the time. On August 12th, I identified one issue with the sample config. that ships with nsopenssl having to do with “SSLv2” being omitted from the “protocols” list but being (incorrectly) included in the “ciphersuite” list which would result in the server crashing when SSL clients attempt a SSLv2 connection. But, this wasn’t the root cause of the problem.

Then, almost a week later on August 18th, Bruno Mattarollo brought up the issue again, but this time was different. Bruno indicated that he was able to reproduce the problem fairly reliably! He said,

What I did, that
triggered the error was click on a link and immediately click on
another link without giving the server time to actually return the
page, so I guess what’s happening is that there is no socket for
nsopenssl to send the results to … right?

Bruno and I spent the next few days trying to diagnose the problem — he even blogged about it. Along the way, I found some other unrelated issues which I logged at SourceForge in Bug #1012892 along with patches against AOLserver 4.0.8a and 4.1.0a that address them. However, continuing to try and get at the root cause of our nsopenssl issue, I realized that fixing the problem would not be a trivial change. The nsopenssl code needed some serious clean-up — I was having a hard time getting a grasp of what it was doing (or, more importantly, what it wasn’t doing).

So, today, I sat down and began to clean up the nsopenssl code. After several cigarettes and some head-scratching, I got the code to a state where I could really start tracing it in the debugger and see what was happening. And, what I found was that when the remote client abruptly terminates the SSL connection, the server notices (because SSL_write() fails) but because the browser requested an HTTP Keep-Alive connection, the server returns the connection to the pool to read the next HTTP request. When it goes to read, it fails on SSL_read() (because there’s no peer connected) and so begins the error loop. I managed to clean up the code and ensure that when an error occurs, we mark the SSL connection as “shut down” so the driver knows not to use it for Keep-Alive and thus will properly close the connection. I announced the fix at 4:13 PM today, and around 9:01 PM, Bruno logged in, applied the patch to nsopenssl, tested and verified that he can no longer reproduce the problem!

I’m going to wait a few days for others to apply and test the patch, then I’ll commit the patches in Bug #1012892 to CVS. Noah Robin asked if a similar fix could be backported to nsopenssl 2.x which I said could be possible if folks verify the fix to nsopenssl 3.0 to be complete, that I would look into backporting the fix.

Overall, I’m hoping this makes nsopenssl 3.0 beta stable enough for us to consider the upcoming nsopenssl 3.0 beta 22 a release candidate. We’ll see …

As a long-time fan and practitioner of Extreme Programming (a lightweight or “Agile” software development methodology), I ran across an excellent article in Software Development magazine entitled You’re Still Needed (free registration req’d) by Esther Derby and Diana Larsen.

Esther and Diane do a great job of describing specific ways a traditional “manager” in a software development team can (must?) change their role and activities to best mesh with an Agile team. For anyone who’s a manager in the field now who’s feeling threatened by the whole Agile movement, I strongly suggest you read this article.