HTML

I’ve been doing a lot with HTML lately. HTML is great, because it’s
important.

• I’ve started working with microformats some. This initially started
  with an OLPC project, where I wrote an
  hReview parser
  (hReview is a
  microformat).
• Then related to that, I wanted to download a bunch of pages in an
  isolated form, so all the internal links are fixed up. This became
  PageCollector. Since
  then I’ve added a script frontend. I think this could be a
  generally handy export system. It doesn’t just make links relative,
  it also lets you completely remap filenames and is fairly careful
  about links (including links in CSS).
• Doing all this HTML-related stuff, I’ve been using lxml a lot (and it should be inferred I
  therefor also like libxml2 a lot). I like
  lxml a lot; probably the biggest reason is that it has a fast
  permissive HTML parser. But the ElementTree API that lxml uses can
  be a bit awkward for HTML — it works well for XML, but in HTML
  there’s lots of mixing of text and tags, and that feels really weird
  in the ElementTree API. So I’ve started lxml.html in a branch with lots
  of feedback from Stefan Behnel. This adds methods that are useful
  in HTML. With just a few smallish methods, I think it makes HTML
  much easier to work with in lxml.
• Though BeautifulSoup is nice, it’s not
  a C-based HTML parser. With Deliverance we are literally
  parsing and rewriting every page as it comes through our site. I
  couldn’t do this with BeautifulSoup. And we’re not doing any crappy
  regex stuff; lxml lets us actually understand the HTML the way the
  browser understands the HTML. That’s important. This is kind of
  the promise of XHTML — the ability to create smart and useful
  intermediaries — that XHTML never really fulfilled. The idea of
  smart intermediaries is a powerful one, XHTML just wasn’t the right
  basis; intermediaries are supposed to work with real endpoints,
  and those endpoints were and continue to be HTML, not XHTML.
  html5lib is cool too as an
  HTML parser, and important as a reference implementation, but I’ll
  mostly be waiting for that work to filter back into something like
  libxml2.
• I should note that I think the stream model of handling HTML (as
  HTMLParser uses)
  is lame. Typical HTML is not very big, and a document model (where
  the whole document is loaded into memory) is way easier to handle.
  HTML is best understood as a document. (Genshi would probably be rockin’ fast if it
  used lxml)
• As part of lxml.html I’ve been writing an HTML cleaning module.
  It’s a bit messy, and incomplete, but I think it will be really
  useful. Because we fully parse and then serialize the HTML, a lot
  of XSS attacks are avoided really early. At work David Turner has
  been working on a Transclusion intermediary called
  Transcluder.
  It’s a little frightening to fold together disparate pieces of HTML;
  this might make it less frightening.
• I took the pattern from formencode.doctest_xml_compare and also
  ported it to lxml. Now you can do semantic HTML and XML diffs in
  your doctests, instead of literal string comparison. This means
  that things like the order of your attributes are ignored, as is
  whitespace. And the diffs help highlight problems in a more
  fine-grained way. Using the html branch you can do from lxml
  import usedoctest or from lxml.html import usedoctest to
  enable the comparison inside a doctest. And you don’t need a custom
  doctest runner. I managed to do this using horrible horrible
  monkeypatching. Also from FormEncode I’m hoping to port htmlfill over to lxml.html sometime
  too.
• With Luke Tucker we wrote an html diff module,
  which I’ve also put into lxml.html. This shows textual differences
  while trying to avoid markup differences (since it’s really hard to
  visually represent changes in markup, and also usually not what
  people care about). This is surprisingly difficult because you also
  need to preserve the markup — you can’t just smash all the words
  together. I think it’s a pretty good implementation, and there
  aren’t that many implementations of this out there. You can also do
  blame style annotations, showing who added what over the history of
  the page.

Open Plans

Stuff from work (which is a little
vague, because work, private projects, HTML, OLPC, all overlap a lot).

• Deliverance is
  deployed on the live site. We haven’t really used its
  functionality much yet. But it’s awesome functionality once we get
  around to using it.
• Related to that deployment, I spent some time with
  paste.httpserver (trunk only) on the thread pool. It now adapts
  the thread pool to add threads when necessary, kill threads that
  seem to be wedged, and various tools around that. This recipe was handy, as now I
  can kill threads. It’s surprisingly slow, though — it frequently
  takes several minutes for the thread to be killed. But it shows up
  as a regular exception, which is great. You can read about the
  details in this document.
• I’ve been extracting some stuff into WSGIProxy for your non-browser-based
  proxying. This is mostly for handling HTTP-based dispatch between
  server applications.
• Not really for work, but related to proxying I whipped together a
  little HTTP proxy using Paste
  and WSGI middleware. It kind of works. It was just a prototype, so
  I haven’t worked on it more, but with just a little more work I
  think it should be easy to turn any WSGI middleware into an HTTP
  proxy. This is the kind of proxy you configure in your browser.
• A followup to the HTML WYSIWYG post, we
  decided to use Xinha. I feel
  pretty good about the decision, which is primarily motivated by
  Xinha’s UI — in small but important ways it feels like the best UI
  among the bunch. It’s just more polished and higher quality than
  the other options. It’s also a Real Open Source Project, which
  matters to us. But we haven’t actually implemented this yet.
• I wrote a total hack of doctest that lets you do
  from dtopt import ELLIPSIS, and then the ELLIPSIS option (that
  lets you use … as a wildcard in output) will be enabled for
  the rest of your doctest. It’s ugly, but I might add a few more
  similar hacks to that package and make it into something more real.
  But still ugly.
• With Whit Morris I’ve been working on a Tagging application. It’s just
  getting started, so nothing working yet. I like the atom module,
  which represents Atom feeds and entries as plain XML. This didn’t
  occur to me at first — I was going to create a Python
  representation of the Atom data model and then add parse and
  serialize methods to it. This is wrong, because the Atom data model
  is XML. If you add a funky custom element to your entry and I
  normalize it into oblivion, then I’ve made a crappy library; by
  making the XML the one and only source of information I avoid such
  things.
• I wrote a little proof of concept for web application help called
  WebClippy
  (svn here).
  It’s inspired by Hackety Hack (a neat
  Ruby educational environment) and how tutorials are written there –
  just a very simple bit of persistent help. Writing it made me
  realize that S5 should
  totally be rewritten, because Javascript-based slideshow scripts are
  easy to write and S5 is annoying. Anyway, I need to actually try to
  document something with WebClippy, because my test slides are lame,
  and I’ve decided realistic examples are essential to good
  development.

OLPC

Stuff from One Laptop Per Child (besides the
HTML experiments)…

• I went to Boston for OLPC, and we talked a lot about Web-Based
  Annotation. We want there
  to be a way for children to add notes (shared or private) to any web
  page. There seems to be people from several directions working
  towards this, outside of OLPC. Annotation is something that seems
  to pop up every few years, then die down; maybe this time we can
  actually make it work. Maybe Web 2.0 should just be “good ideas
  that aren’t new, but this time they might actually work”.
• I didn’t do any of it, but PyXPCOM is in the Sugar
  browser now (Sugar is the OLPC UI). This excites me, except then I
  looked at the code and I couldn’t figure out how to get access to
  any interesting XPCOM objects (the DOM in particular) — XPCOM seems
  very indirect. Oh well, at least now it’s possible and I just
  have to figure out the details. I hope/want/am-optimistic that OLPC
  will have a great browser experience. It doesn’t yet.
• We had a little OLPC sprint for Chicago people interested in the
  project. Again
  getting a working Sugar environment was a bit of a problem.
  Afterward I played around with some VMWare techniques.
  These need some polish, but I think it’s the right direction and
  it’s not that hard. If someone could get this working in a scripted
  way, I’m sure it could be added to the standard build process at
  OLPC.

Miscellaneous

• I wrote WaitForIt, a little
  WSGI middleware that you can put in front of slow WSGI applications.
  If the application is too slow, it gives the user a page asking them
  to wait. This is particularly handy for things like administrative
  areas where you realize you are doing some really slow blocking
  operation and you don’t want it to time out but you really don’t
  feel like setting up a fancy queue for it. Just pop this in front
  and your problem is pretty much solved. This is part of a secret
  framework I’m developing.
• I started revisiting buildutils and cleaning some
  stuff up. Mostly I’ve added some commands to make it easier to add
  and make new commands for setup.py. This relates to this post. I added a
  command to WebHelpers
  for Javascript compression.
• It’s a little old now, but I added a templating language to
  Paste. Using string.Template for the little internal templating
  things just wasn’t good. Cheetah has been too hard to install. So
  this is a simple, single-file templating language based on string
  substitution. I should probably extract it and do a few more things
  with it. I feel a little dumb about writing Yet Another Templating
  Language, but I couldn’t find a templating language that I liked for
  small text-related tasks. For actual web programming a more
  complete language is good.
• I made a little package called CmdUtils to make it a bit
  easier to build command-line scripts, with the patterns that I
  personally like.
• I kind of got a new encoding working. It encodes
  text to spaces, newlines, and tabs. Now whitespace can be even more
  significant! You can also put non-whitespace in the encoded text as
  a form of steganography.
• Among my little recipes I added
  propertyclass for
  making properties. I also added a little script for pasting
  text to a pastebin (only for X users). And also one for creating
  easy_install links from a subversion repository.
• I wrote about my desktop for a
  new blog that Steve Holden created.

Hmm… more stuff than I thought. Working at The Open Planning
Project is great, because I spend all my
time working on open source projects in public repositories. And also
working on Stuff That Interests Me. If you’ve gotten an email from me
recently, you may notice I link to our careers page in my sig, simply because I
think everyone cool should work at TOPP. We’re doing other
interesting things too that I’m not directly involved with; I’ll try
to write about other people’s projects before too long.

After vacation I should probably start looking more into getting
WordPress on our site. TOPP is not technically tribalist! We do
accept PHP, even if we don’t really trust PHP. Once that’s up, I’ll
move my own blog from this crappy piece of spam ridden crap (that I
wrote, so no one but myself to blame), and then maybe I’ll post more
because I’ll be less embarrassed about the software.