I just got back home from the Linux Symposium. Happy Canada Day! I wish I could have stayed longer to join in the festivities. Alas, I've been taking an awful lot of vacation days from work lately.

Let's see if I can review the whole conference in one post. Then I can go back to my Kyoto travelogue posts.

Wednesday started out with the Kernel Report. I hadn't really been keeping up with the latest kernel news and changes, so this talk was of interest to me, but one can only really scratch the surface of this topic in an hour-long talk. I have sfllaw to thank for setting up WPA-PSK on my laptop. I thought the conference organizers did a good job with the wireless access this year. So I could've been more diligent about posting and responding while I was in Ottawa, but I tried not to be distracted during talks, and I tried to be social the rest of the time, so I didn't fully take advantage of the 'net access.

I spent the rest of the day in tutorials. First was How to Write a Linux File System in 21 Days. For some reason I came to that one late, so I probably missed some information that would've helped me understand the rest better. Following that was the longer More Linux For Less tutorial, wherein they handed out Blackfin 537 development boards and walked us through cross-compiling a kernel for it. I would've been lost if it hadn't been for ostraya's friend Dana, who shared a board with me. As it is, we both succeeded in cross-compiling and loading the kernel on the board. I got to bring the board home with me; I'm welcoming suggestions as to what interesting things to do with it.

On Thursday, I got up in time to miss only the first few minutes of clahey's talk on developing Democracy (soon to be renamed Miro). Afterwards I joined a standing-room-only crowd to hear a talk on the status of ext4. I haven't heard from anyone who's particularly impressed by ext4, though I haven't yet tried any of the filesystems I have heard recommended, like xfs or zfs. Next I heard The Hiker Project: An Application Framework for Mobile Linux Devices. After lunch, I caught some of Internals of the RT Patch. This one had some relevance to work I had been doing for my employer, though I wasn't able to follow many of the details, so I'm making a note to myself to read the paper.

And then, one of the highlights of my trip: tea at the Château Laurier with sfllaw, ostraya, scjody, and fabien. It was excellent—I'd recommend it to anyone visiting Ottawa, and I do hope it becomes an annual tradition as long as the Linux Symposium is there.

We returned to the conference for IBM's Linux on Power/Cell BoFS, which was simultaneously half-reception and half-presentation. We schmoozed a little before stopping by LCBO to pick up a few bottles of good wine to bring to a party. clahey also brought a couple of the many sorbets he made throughout the week, which included pineapple, strawberry, grape, apple, and carrot.

On Friday I attended talks on Thermal Extensions for Linux Handhelds; Comparison of SMB2, CIFS and NFS; cpuidle; and ext4 Online Defragmentation. None of the talks was really outstanding. I got really tired of bullet points and the whole lifeless way of presenting material that's ubiquitous at conferences and in offices these days. maddog's Thin Clients/PHAT results - Are we there yet? rant had bullet points, but it was more inspired.

After a dinner where we stuffed ourselves at an Indian restaurant, the plan was to make it back for Andrew's One Laptop Per Child session, but alas, our service was very slow, and we didn't make it back in time. Instead we went right to the LinuxChix gathering at Oh So Good. It was awfully loud in there, but there were delicious desserts and good company. If only the majority of attendees could have been women!

On Saturday I attended With No Tears: Building A Mobile Linux Device, the kernel.org BoFS, and Evolution and Diversity: The Meaning of Freedom and Openness in Linux (James Bottomley's keynote address). I made my traditional stop at Paper Papier in the afternoon as well, buying—among other lovely and not-strictly-necessary things—a fountain pen. After dinner I attended the traditional party at the Black Thorn Cafe, where I had a surprisingly good time. I say 'surprisingly' because in the past I've experienced it as a very loud, crowded event where I knew almost no one and had a difficult time elbowing my way to the bar to order a drink. This year was better though, as it was less crowded, I knew more people, and the waitstaff circulated quite readily with food and drinks.

You know, it seems a large part of the reason I return to the Linux Symposium year after year is to see far-flung friends and engage in annual traditions. From where I'm sitting now, that's just fine.

Hello from Ottawa! I'm here for the Linux Symposium, of course. [Ottawa is also one of the host cities for the FIFA Under-20 World Cup 2007, so if I'm feeling ambitious I may see about getting a ticket for one of the first few matches.]

As is the tradition, one of my flights here experienced weather-related delays, so we sat on the tarmac at O'Hare for what would add up to hours. At one point our plane had to give up its spot in line to take off (fifth out of about seventy-five waiting planes, we were told) to get more fuel. So I didn't get here in time to register this evening as I'd planned (and set up WPA on my laptop, which I've never done before, to access the conference wireless network), but I've experienced worse.

The sleet we got overnight has made today a snow day from work, although I'd already planned to spend up to the whole day at home waiting for the DSL tech, since today's my scheduled "Telco Date" with Covad (through Speakeasy). So far I'm still in my flannel pajamas, catching up on e-mail and friends' weblogs and disinterestedly reloading my county's real-time snow removal status web site. I'd been planning to go to Speakeasy DC last night, but the twenty-mile drive to the nearest metro station seemed like a bad idea, so I skipped it.

I've been watching a bunch of John Cusack movies lately—I've seen High Fidelity, Better Off Dead, One Crazy Summer, Grosse Pointe Blank, Being John Malkovich (years ago), and Say Anything.

My parents visited last weekend, my dad fixing my house's heat pump and my mom cooking up lasagna and homemade chicken noodle soup, the leftovers of which I'm really enjoying. Without any specific shopping list, we went to IKEA for an hours-long round of "This is so cool!" "What is it?!" Our local IKEA is huge. And I can't resist their lighting department.

I expect I'll be able to attend the Ottawa Linux Symposium this year, though I wouldn't be presenting, so there's a good chance it'd be out of my own pocket (as in previous years) rather than paid for by my employer.

Saturday was the last day of the Ottawa Linux Symposium.

The first talk of the day was on the NPTL Stabilization Project. That project's goal is to reduce the number of bugs in the Native POSIX Threading Library. They collected test cases largely from the Open POSIX Test Suite. To deal with the avalanche of test case results they got, they wrote TSLogParser. More interestingly, they started the NPTL Trace Tool project to allow multi-threaded application developers and NPTL hackers to analyze library calls. Their rationale was that with more and more multi-processor systems being manufactured, more multi-threaded applications are being written, and their writers need to be able to record multi-threaded operations at the application level, record internal NPTL operations, and capture first-failure data. Pre-existing tools just didn't cut it.

Implementation-wise, the project members strived to effect no change in application behavior, minimize the tool's performance impact, support long runs and large volumes of traces, and not lose any traces. They chose to use one buffer for each traced application, a decision that provied simple and reliable. The speaker showed us visualizations of some PTT traces of basic multi-threaded applications and explained that they're only part-way into their testing effort. They're soliciting beta testers and hope to get their tool into major Linux distributions.

Next was We are not getting any younger: A new approach to timekeeping and timers, which covered John Stutz's new Time of day proposal. It sounds like he'll be making the timekeeping code a lot cleaner: he's splitting the time of day management from timer interrupts, consolidating a lot of code, using nanoseconds as the kernel's base time unit, separating the NTP code from the time code, and using generic algorithms that choose time-source drivers at runtime. No one's going to disagree with that.

Third was the last and most interesting presentation of the day, on the sysfs filesystem. I'd gotten a taste of sysfs at the driver-writing tutorial two days earlier, but this talk provided a broader overview. Briefly, sysfs is an in-memory filesystem that provides an easily accessible kernel-user interface by exporting kernel objects, their attributes, and their relationships. It's a strictly hierarchical structure of dynamically-created files and directories. The files are mostly ASCII and usually contain exactly one value each. Each sysfs directory is equivalent to something called a kobject. The directory is created when the driver core registers the kobject, behind the scenes. It's like this:

sysfs property		filesystem representation
kobject	...	directory
kobject attribute	...	file
relationship between kobjects	...	symbolic link

A lot of utilities use sysfs, including lspci, cpufreq, and block device tuning. And more—like module utilities and system management tools—will use it.

But sysfs has a number of shortcomings. If you're trying to do something simple, the whole kobject infrastructure can be a headache with all the overhead. It's also lacking good external access: there are no tools for manipulating or visualizing it, no decent library for access, no "transaction"-type locking mechanism. And there's no sysfs support for embedded systems, clusters, or virtualized—hey, we can't let half an hour go by without mentioning virtualization!—systems. Possibly except for that first flaw, though, I expect we'll see these shortcomings addressed in time.

A keynote address by kernelslacker concluded the symposium. OLS's keynotes have always been very good in my experience, and this one was no different. If you're going to sit several hundred Linux geeks down in a room together and get them to shut up, you need to have someone they respect up at the podium, and for that speaker, it's a chance to talk about big ideas and get his message out. Dave's theme was "Tools! Tools! Tools! Tools! Tools! Tools! Tools! Tools! Tools!" He shared some thoughts on bug-reporter psychology and some quotes by Linus to the effect that userland, not the kernel, is where all the excitement's at. I don't know how the quotes were related to the rest of the speech; maybe it's simply de rigeur to quote The One in the keynote address. kernelslacker asserted that the bugs that cause the most trouble are in device drivers, an area where automated testing isn't really practical. He made what might be a controversial statement discounting the effectiveness of the Linux Test Project at finding any bugs he cared to fix. The solution, instead, is to make heavy use of static analysis tools. Also, he likes monkeys.

clahey, a friend of his, hpa, DrSuzi, and I had a very nice dinner at Cafe Crepe de France, repeating the tradition from last year. We had a fascinating conversation about languages—I might note here that I know absolutely no French—and travel. The steamed mussels I had were great, all plump and meaty. We made our way to the annual end-of-conference party at the Black Thorn Cafe, which was no less crowded for being closed to the general public. Maybe next year we should take over some nearby bars too. :) I didn't know many people there, especially with the low LinuxChix turnout this year, and I'm not one for mingling in noisy, crowded places, so I took off after around forty minutes.

I had some time Sunday to walk around Ottawa before my flights home. I didn't think I had time to make it across the river to Gatineau, so I mostly stuck to Byward Market, making the usual rounds: The Tea Store for a sack of Lapsang Souchong Butterfly #1, Wang's Noodle House for a bubble tea and lunch, and Paper Papier for greeting cards, some of which I may never send because they're so beautiful or just because they're so true.

I paused for a moment to look at the American embassy as I passed in front of it. It's still got three layers of fortification: jersey barriers outside of a line of bollards outside of a high cast iron fence. ::sigh:: My country, 'tis of thee.

Virtualization is a big topic in Linux now. Friday's schedule offered no less than seven events on that topic. I'm passingly familiar with User Mode Linux (my web server's on a virtual Debian machine under UML), but this year's star is a virtual machine monitor I'd never heard of called Xen.

I started out the day with a usbmon talk. The idea's simple enough: a human-readable display of USB traffic. The goal is to have software that provides ethereal-like parsing and a tcpdump-like command-line utility, but for USB, plus a binary, versioned API for user mode. zaitcev talked about how he used it to save time debugging and showed us a demo. Since I've used a bus analyzer (for a very different bus type) at work, I feel somewhat comfortable with them; there's something satisfying about seeing every bit going back and forth and understanding just what's going on. Nobody needs to convince me of their value. zaitcev envisions analogous projects for other protocols: scsimon, infinimon, bluemon, firewiremon, wifimon, fibrechannelmon, and so forth. Right on.

Aside from the device driver-writing tutorial, the event at OLS that I was most looking forward to was going to be Jon Trowbridge's Beagle/inotify talk. So I was disappointed to see that the speaker couldn't make it, so the talk was canceled. Dejected, I returned to my hotel room and took a nap.

After I awoke for the second time yesterday, I went to a "non-technical" talk on SeqHoundRWeb.py, a Python-based interface to online bioinformatics data. It started with the shortest-ever lesson on molecular biology so we could get an idea of the kind of data our presenter works with. We learned that the world of genomics includes many large sources of data collected by different methodologies and characterized by different vocabularies. The challenge is integrating them and making them easily available and searchable for researchers. The speaker chose Python largely for the usual reasons: it's a clean object-oriented language, there's good learning documentation, it has broad cross-platform support. He mentioned that it also has good support for other resources like BioPython—bioperl and BioRuby notwithstanding, I suppose.

Next on my agenda was Case Study: Usage of Virtualized GNU/Linux to Support Binary Testing Across Multiple Distributions. The motivation was a need to test system initialization scripts on multiple distributions independently and concurrently, when each test takes a long time to run. Multibooting with grub or lilo wouldn't allow concurrent testing; vmware uses too much memory. The solution in this case was to run User Mode Linux, configured with a Gentoo host operating system with a slightly modified disk I/O "elevator sorting" algorithm. The guest operating systems were Novell Linux Desktop 9, Red Hat Enterprise Linux 3 and 4, and one of the Red Flag products. The result: they saved money and lab/desk space, and they ended up with a quickly re-usable hardware platform. They noted that a virtual Linux setup wouldn't be so useful for hardware abstraction tests, resource sharing, or inter-client communications. Future plans include replacing the host operating system with a hypervisor and addressing resource protection.

Delving deeper into the world of virtualization, I next attended Testing the Xen Hypervisor and Linux Virtual Machine. (In retrospect, without having attended any of the earlier talks on Xen, I was unprepared for one that assumed a degree of familiarity with it.) As part of the Linux Test Project, the researchers used a tool called Xenfc that makes random hypervisor calls. Most of the calls are expected to fail, and the tool checks to see that the correct error codes are returned. They tested Xen's balloon driver, a feature that lets you change the amount of memory that's available to each domain. They tested SMP support and found that SMP in Xen isn't yet stable. And they developed Xentest, an automated testing framework centered around Xen.

For dinner I headed to Elephant & Castle, a chain-restaurant pub, where I had some good Irish stew. I thought it would fun to try a Tom Collins cocktail, but I found I didn't like it much. I tend to assume that classic cocktails are subtle drinks, and this one is vaguely like a Gin and Tonic. What I failed to think seriously about is that fact that it's like a Gin and Tonic except you replace the tonic water with a combination of lemon and lime juices and sugar. Just too sour for my taste.

After dinner, I made it a point to attend the Debian Women: Encouraging Women without Segregation Birds-Of-a-Feather Session. The first thing anyone said about this session beforehand was, "A guy's leading it?" Yes, a guy led it, an enthusiastic Brazilian computer science student who reported that the girls were busy since Debconf 5 had just concluded. He presented the history of the Debian Women project and explained its mentoring program. My impression is that he's someone who generally "gets it," in spite of his habit of referring to women as "girls" while referring to men as men. I'm going to give him the benefit of the doubt and presume that he isn't familiar enough with English to understand the implications of his word choice there. Dana and Erinn were among the 'chix mentioned as being active in the Debian community. Also mentioned were linuxchix, KDE Women, GNOME Women, and—speculatively—Ubuntu Women. Possibly too much time was devoted to laughing at the latest misogynist troll, the response to whom is discussed by Mary. Overall, though, I think it was a worthwhile session.

The final session of the day that I attended was H'Uru - Coding Beyond MYST. The idea is that after a MMOG named Uru was released in 2003 and then canceled, our speaker set out to develop a Linux port of the game, with nothing to work with but a bunch of data files and some hints about the file format by some folks at the Clockwork Orange BBS. Much of the structural data was encoded in a PRP file format, organized like a relational database to store data like game-world geometry and transformations. Other file types contained Python bytecode for more complex interactions, saved game states, and which PRP files make up an age, among other things. Our speaker started by looking at hex dumps of these data files, or what he called "reverse engineering by meditation." He then wrote a utility called prpdump to interpret the data in a slightly more readable format. The Uru gameplay server is apparently a Python interpreter with lots of extension classes, so our speaker used decompyle to understand what the code does.

Uru's graphics were pretty sophisticated; the game used some advanced features of graphics cards, only some of which were supported in OpenGL, if I understand correctly. Its multilayer texturing combined up to four textures in various modes, and each texture possibly was animated, used a different material for each layer, used color and alpha at the vertex, used reflection coordinates, and/or used vertex blending. Our speaker found that rather than using one generic shader for this texturing, it's faster to generate specialized shaders and switch between them. Other performance hacks he discovered were doing static lighting as much as possible and using bounding boxes for occlusion culling.

Unfortunately, he couldn't get a demo running adequately on the laptop he'd brought, so I didn't get to actually see H'Uru. He reported that there's still lots of room for improvement: the shaders are still awfully slow, home-brew collision detection kinda sucks, octrees are "simply bad," sound doesn't work, materials and shading aren't always right… it's a work in progress.

I seem to be fairly consistently a day behind in my write-ups; here's Day Two, yesterday, of the Ottawa Linux Symposium:

First up is a tutorial I was really looking forward to: Write a Real, Working Linux Driver. One of the prerequisites for this tutorial was a laptop running the latest version of the 2.6 kernel from kernel.org. Being a total slacker, I didn't compile that kernel until yesterday morning, about 100 minutes before the tutorial began. On the good side, my laptop was able to compile it in time. On the bad side, I didn't entirely know which files to copy over to /boot and how to specify where root is in the new grub.conf entry (copying the existing Fedora examples didn't work). So I ran the 2.6.11-1.1369_FC4 kernel instead.

The device for which we were writing—or, perhaps, simulating writing—the driver is the Go!Temp thermometer. Each of us got one, along with various iterations of the code we were supposedly writing. Before looked at the code, we got a lesson in how USB works, which is more complicated than I'd imagined. We explored the kernel's device model by poking around in /sys. Things got over my head pretty quickly. But I had no problem compiling our developmental driver as it evolved toward something functional. Where I couldn't keep up was with the module load process; sometimes our kernel driver module loaded for me, and sometimes it didn't. After two-and-a-half hours, I didn't manage to reach the goal of having a little temperature output. It is unclear to me whether that's related to my running a (Fedora Core 4) kernel that was neither the latest nor one that I had the source code for.

I eventually did get the 2.6.12.2 kernel that I'd compiled installed and running, and I plan to use that to try to get that driver working. I'll keep you updated.

Kernel driver writing is hard work, so I was hungry. In the adjacent mall's food court, a Québecois navy officer struck up a conversation with me as we ate. We talked about different places we'd each visited; he recommended the historic neighborhoods of Gatineau. I'll have to see if I have time to get over there on Sunday.

After lunch, I attended Building Murphy-Compatible Embedded Linux Systems. The speaker's contribution to this goal is "a framework for safe remote configuration and software upgrade of a Linux system that supports atomic transactions, parallel, interactive and programmed updates and multiple software versions with roll back." I could understand this framework—which he'd implemented too—only shallowly, but it sounds really clever. One part of the framework involves mounting a storage device "into a mount point in [a] temporary mount point of the version image." I can't help but think of Aaron in Primer taking a folded-up box with him into a box.

Next I went to Networking Driver Performance and Measurement - e1000 A Case Study [sic]. The speaker went over the history of e1000 driver development, and there was some discussion of, I think, a rarely-manifested flaw in TCP Segmention Offload that is popping up now that e1000 devices are handling massive amounts of traffic. Linux support for I/O Acceleration Technology reportedly "is just around the corner."

Following that was a talk on Enhancements to Linux I/O Scheduling. The speaker went over the four canonical types of I/O scheduling in Linux: anticipatory (AS), deadline, completely fair queueing (CFQ), and no-op. He had experienced problems with AS, the default, exhibiting a behavior he called deceptive idleness. So he came up with a cooperative anticipatory scheduler (CAS) that detects cooperative processes and anticipates accordingly. My understanding is that this scheduler thus includes both per-process and process group anticipation. He tested CAS using the Flexible Filesystem Benchmark, and CAS looked superior to AS. But the larger conclusion was that different situations call for different scheduling algorithms. His current work is toward selecting the best scheduler dynamically and automatically. It doesn't sound promising, though; last year I attended an I/O scheduler performance talk in which the researcher concluded that scheduler selection is a complicated issue at best, and not worth performing in many cases.

My day of talks over, I enjoyed a very nice dinner party hosted by sfllaw, who brought together some of the finest kernel hackers and their significant others, and me, for a delicious meal and conversation. This event was smaller than last year's party, but I ended up with the same dilemma of feeling sleepy at what this crowd considers an early hour at the same time that I can't tear myself away from the fascinating stories. I guess I can't complain, can I?

Day One, yesterday, started with a heavily-attended talk on the 2.6 kernel development process, as well as the previous process and plans for the future. In the past, there were even-numbered releases that were stable, and there were odd-numbered releases that were developmental. The 2.6 kernel started this way. But there were problems: the development cycle was so long that new features would sit for years. The feature freezes never held. "Stable" kernels stabilized only slowly, and development would stop for long periods. To get new features in, distributors would backport them, adding to the overall complexity. Enter Andrew Morton, Linus' new sidekick. Enter BitKeeper: no more dropped patches. In fact, patches get into the mainline more quickly than ever before.

Now, Linus' 2.6.x releases are the stable series, and Andrew Morton's -mm is the stable tree. Andrew has introduced more professionalism to the kernel development process.

But there are still problems:

• not enough testing of -rc kernels
• too many late additions
• little formal bug tracking
• the mainline kernels don't always stabilize well

And one more problem: BitKeeper decided to turn commercial-only. Linus responded by rolling his own content tracker, git.

"I'm an egotistical bastard, so I name all my projects after myself. First Linux, now git." - Linus Torvalds

The second talk I attended was on the Novell Linux Kernel Debugger. Maybe the presenter had a head cold; anyway, he sounded bleary, and the slides were hard to read. I left partway through the presentation. I'm wondering how a kernel debugger differs from an ordinary debugger.

Next up: Hot Keys, Video Control, Suspend/Resume, Oh My! -- Recent Advances and Current Challenges in Linux/ACPI. The ACPI 3.0 Spec was released last year, and I suppose progress is being made, but the usual raise-your-hand-if-you've-got-suspend-working-on-your-laptop poll results weren't dramatically different from last year. One interesting bit of information we learned is that a blackened (LCD, presumably) screen consumes more power than an all-white screen. I had no idea.

After that, I attended a talk on TWIN: An Even Smaller Window System for Even Smaller Devices. Keith Packard is a great speaker, in a low-key kind of way. The motivation behind TWIN is that X takes a lot of CPU and memory, which takes a lot of power, which doesn't work on small devices. So he wrote a tiny new window system that spends more CPU time in order to save memory ("different compromises for different environments"). He found all kinds of shortcuts. Really, to me the idea of writing a window system is mind-boggling enough, but to write one that sports overlapping translucent windows, anti-aliased graphics and scalable fonts in a total memory budget of 100 kilobytes? Remarkable.

Next was Automated BoardFarm: Only Better with Bacon. The BoardFarm is a system that lets developers get remote access to embedded systems boards and to run their tests automatically. It would be nice to have something like that at work, where our testbed environments allow some remote access but where manual intervention is often needed. I'm mainly doing testing of flight software at work now, and I've spent what seems like a lot of time lately going up to our lab and suiting up in anti-electrostatic-discharge gear just to flip a switch. For the most part, I'm so new to it all that I assume there are good reasons for things to be set up the way they are.

clahey showed me a presentation he made just a few days ago at the Desktop Developers' Conference, on GOffice, and more specifically on present, the slim-and-fast Powerpoint viewer he's adapting from slow-and-bloated OpenOffice. I'm betting I can get him a Powerpoint file that it can't handle. :) It's all in the interests of improving the software, of course.

Dinner was delicious Indian food at Café Shafali with ostraya, clahey, DrSuzi, and hpa. Afterward, free alcohol and snacks at the welcome reception. I've learned that unless the reception speaker is really interesting, he is largely ignored. This reception started with an Intel marketroid (they sponsored the event), and it was hard enough to hear the proceeding speaker above the cocktail-hour din or see his packed-with-text slides that we gave up early on. We passed around clahey's newly-acquired tavern puzzle, watched DrSuzi spin yarn out of some amazingly soft linen hemp, and chatted about weird forms of neurotrauma (DrSuzi's a clinical psychologist, which seems weird because she's, like, our age).

Thus ended Day One of the Ottawa Linux Symposium.

I've arrived for my third year at the Ottawa Linux Symposium, and by now this city is starting to feel familiar. The flights went smoothly, more or less; I can't complain about sitting on the tarmac for an hour after the debacle of two years ago (or, the Nightmare in Newark).

My suite is sweet. (The hotel randomly upgraded me.) It has a kitchen, living area with sofa, washer/dryer, balcony, office area with a multifunction fax machine, ethernet and wireless internet access, a coat closet, plushy robes… not that I have any use for most of that stuff. Well, I could make myself a cup of tea.