Serpent Addiction

python builtin inflation

2006-02-18T14:05:00.000-08:00

There's some talk on python-dev about adding defaultdict. One consequence of tinkering with Python is the number of builtins increase over time. Here's my only-slightly-tongue-in-cheek proposal. The total amount of builtins should be limited to a standard sized 80x24 xterm screen:

Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
>>> dir(__builtins__)
['ArithmeticError', 'AssertionError', 'AttributeError', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FloatingPointError', 'FutureWarning', 'IOError', 'ImportError', 'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'OverflowWarning', 'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError', 'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '__debug__', '__doc__', '__import__', '__name__', 'abs', 'apply', 'basestring', 'bool', 'buffer', 'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'min', 'object', 'oct', 'open', 'ord', 'pow', 'property', 'quit', 'range', 'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']
>>>

We've got a little room for futher expansion, but many more will have to come at the cost of removing others. How did 'zip' ever get in there? ;-)

Interesting to compare with "classic" Python 1.52 ...

>>> dir(__builtins__)
['ArithmeticError', 'AssertionError', 'AttributeError', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'FloatingPointError', 'IOError', 'ImportError', 'IndexError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplementedError', 'OSError', 'OverflowError', 'RuntimeError', 'StandardError', 'SyntaxError', 'SystemError', 'SystemExit', 'TypeError', 'ValueError', 'ZeroDivisionError', '__debug__', '__doc__', '__import__', '__name__', 'abs', 'apply', 'buffer', 'callable', 'chr', 'cmp', 'coerce', 'compile', 'complex', 'delattr', 'dir', 'divmod', 'eval', 'execfile', 'exit', 'filter', 'float', 'getattr', 'globals', 'hasattr', 'hash', 'hex', 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'len', 'list', 'locals', 'long', 'map', 'max', 'min', 'oct', 'open', 'ord', 'pow', 'quit', 'range', 'raw_input', 'reduce', 'reload', 'repr', 'round', 'setattr', 'slice', 'str', 'tuple', 'type', 'vars', 'xrange']

It's not all fun and games

2006-01-20T11:50:00.000-08:00

I'm wearing my sysadmin hat today. Last night
I started an upgrade for trajan (Debian Sarge).
When I got in early this morning it wouldn't
boot, just hung at "LI". The usual panicky
minutes before verifying that the nightly
backups completed.

Next, attempt to boot up on Knoppix
and verify the filesystems are okay.

I attempted to rebuild LILO (yes, I use GRUB
for all the new installs):


# mount /dev/hda3 /mnt/hda3
# chroot /mnt/hda3
# /sbin/lilo

But Knoppix won't write lilo to /dev/hda,
even after I've given it write access. This
is one of the few times Knoppix has let me
down. Fortunately, I've got another Linux
Rescue CD with the 2.4 kernel. Booting up
on it finally allows me to rebuild LILO. My
decision process was much more time consuming
than appears above, but that was the general
flow. Stupid Jeff: I didn't have a rescue
diskette built, at least not one that was
readily available. A boot diskette would have
saved valuable time.

Unfortunately there are now some residual
problems from the upgrade. Python spews some
wierd linkage warnings (fixed by recompiling
the source), but Apache won't allow any users
to log on.

An awful lot of cursing/head-scratching later
reveals a couple of very obscure Apache bugs:

The gist of the matter is that I must swap the
order in which the following modules appear in
the configuration.

LoadModule auth_module /usr/lib/apache/1.3/mod_auth_ssl.so
LoadModule apache_ssl_module /usr/lib/apache/1.3/libssl.so

Also, I had to turn off SSLFakeBasicAuth, which
was on by default.

Quite a nerve-wracking morning.

Twisted online API documentation is hopeless

2005-12-10T10:39:00.000-08:00

There's an outstanding book on Twisted by Abe Fettig. I've been meaning to get down into the details of Twisted rather than devote all my time with Nevow. The examples are clear and well-written, e.g.:


from twisted.internet import reactor
import time

def printTime():
    print "current time is", time.strftime("%H:%M:%S")

def stopReactor():
    print "stopping reactor"
    reactor.stop()

if __name__ == '__main__':
    reactor.callLater(1, printTime)
    reactor.callLater(2, printTime)
    reactor.callLater(3, printTime)
    reactor.callLater(4, printTime)
    reactor.callLater(5, stopReactor)
    print "running the reactor ..."
    reactor.run()
    print "reactor stopped"

However, I next go to the twisted website to review the reference API. Guess what? There's no entry tor twisted.internet.reactor. Reading the source reveals that selectreactor is the default implementation of reactor. It's pretty obvious that the parameter to the callLater method are time (in seconds) and and argument-less function, but I'd like to review that in the API documentation.

The twisted.internet.selectreactor section lists no callLater method. Grepping through the source reveals that it's implemented in four places, but the one I'm probably using is twisted.internet.base. There's a reference in the docstring: "See twisted.internet.interfaces.IReactorTime.callLater."

Okay, I can live with this, but it means the online API is a waste of time. I'll simply have to read through the source code to learn anything useful about twisted. The online API actually slows down the discovery process.

Update: Here's a line of Nevow code:

request = inevow.IRequest(ctx)

Want know to what IRequest returns? Check out the API docs.

Acroread on Unix

2005-12-07T14:37:00.000-08:00

Adobe's acroread is another non-GPL piece of software that I find useful for software development, since many specifications are now distributed in pdf format. Unfortunately, too many spec sheets are still tied to the proprietary Microsoft Word format, for which the only remedy is OpenOffice.org or some of the various lightweight MS-Word readers.

Acroread is a bit heavyweight for most viewing tasks, which is why xpdf is still my default pdf viewer. But for certain kinds of documentation, such as 700+ page HIPAA implementation guides, having acroread is indispensible with it's window tiles to aid navigation.

Having acroread installed is useful for software development with *
ReportLab. It's handy to view the output of generated pdf files with two different renderers, though I also check the results on Windows as well.

Acroread on Unix is much improved. For a while it appeared that the Linux/Solaris versions were languishing behind the Windows versions. Unix acroread has one feature not available on the Windows version. It can emit the Postscript to a file or stdout - useful for batch print jobs.

Something new I just learned: Compressed pdf files can be viewed with the zxpdf command.

remote backups with Strongspace.com

2005-12-03T10:03:00.000-08:00

I've been wanting to use an off-site backup process that was simple to administer, yet easy for non-techies to use. Strongspace.com appears to fit the bill. You basically sign up for a particular plan that gives you a specific amount of disk space and a number of users. There's no signup fee and you can discontinue the service at any time.

Strongspace works with sftp/scp and rsync+ssh. For windows users, it can be accessed via WinScp or Filezilla. But for making convenient backups from Windows machines nothing beats an rsync batch file linked to a desktop icon. Even better, would be to schedule automatic backups. The advantage with rsync is that after the initial backup, updates are generally very quick.

The disadvantage with rsync is that it's not an easy program for Windows users to install. Fortunately, David Barrett has written an excellent How-To.

A typical rsync batch file entry will look like this:

rsync -avz "/Documents and Settings/user/My Documents" myaccount@mydomain.strongspace.com:Documents

Backups are only half the story, however. Eventually, there will come a day when the user needs to retrieve lost data. This is where having a browseable backup site is handy. For the user, retrieving a lost document is as simple as accessing her account on Strongspace.com. No intervention from me is necessary. Furthermore, the data can be retrieved from any other location, e.g. modifying a contract or spreadsheet while off-site. It would certainly be possible to configure something like this internally, but it would be a hassle to maintain with limited advantages.

Thanksgiving

2005-11-25T05:10:00.000-08:00

"The miraculous thing about this country is that almost everybody has food, clothing, shelter, and extraordinary devices undreamed of until a moment ago in human history: radios, telephones, color televisions, cars, radios in their cars even enough dough to fly across the country once a year, if they plan ahead and stay over a Saturday. I speak here not just of the great middle class." Andrew Tobias, 1996

Andrew talks about the material possessions, of which food, clothing, and shelter can only be taken for granted by most people in this country, but I'm certainly grateful for them. Working in the healthcare field, I'm thankful for my health. Also, my family and friends.

But what I'm mostly thankful for today is free software. Not just because it costs less, but the breadth and quality of open source software makes it a joy to work with computers. So thanks to Guido van Rossum, Linus Torvalds, and all the others who have dedicated contributed to free, high-quality software.

PostgreSQL is pretty cool. I'm now starting to use it in favor of MySQL. But I spent almost an hour today figuring out how to retrieve column names from a table. Resulting SQL:


  SELECT attname FROM pg_class, pg_attribute WHERE relname='billcode' and 
    pg_class.oid=attrelid and attnum > 0 ORDER BY attnum;

Anyone know of a better way?

jEdit

2005-11-23T06:12:00.000-08:00

A couple people whose opinions I respect have expressed admiration for jEdit, a programmer's editor written in Java.

Since Java is non-free software, it requires more than a simple apt-get command to install jEdit on my machine. Fortunately, the Ubuntu How-To lists a fairly straightforward procedure to install the Java runtime. I used to write quite a bit of code in Java. It's interesting that I haven't had Java installed on my development machine in some time.

A few minor tweaks later and jEdit is running. It supports an large number of features, plug-ins, language support, etc. You move the cursor around with the standard arrows, Home/End, PgUp/PgDn keys, which will make it easy for most Windows users. As a touch typist, however, I prefer to keep my fingers on the keyboard - even moving my right hand to the arrow keys disrupts my speed and concentration. It's certainly possible to remap keys to control-key combinations, but I dislike building special-purpose key configurations. It's always a problem, when you move to another machine, even temporarily. The final negative is the memory required for jEdit. For comparison:

vi, Vim: 4.7MB

xemacs: 19.4MB

jEdit: 258MB

Memory is getting cheaper, but I can run a dozen emacs sessions (or 50 vi sessions) for the cost of one running jEdit process. I think this is the showstopper for me, though I'll continue to play with jEdit.

The high price of low-bandwidth data

2005-11-15T17:32:00.000-08:00

The current situation with mobile phone communications is crazy. In the U.S., it's relatively easy to send a short text message (SMS) to someone's cell phone via email, e.g.:

T-Mobile: phonenumber@tmomail.net
Cingular: phonenumber@cingularme.com
Sprint: phonenumber@messaging.sprintpcs.com
Verizon: phonenumber@vtext.com
Nextel: phonenumber@messaging.nextel.com

Apparently in Europe the carriers make sms-to-email more difficult.

However in Euroland, they make it relatively easy to send outbound SMS messages from your cell phone to internet email accounts. In the U.S., I couldn't find a single provider for sms-to-email delivery. T-Mobile will allow me to send email (not SMS) to the internet, but I must first setup a POP server. Since all I want is to send simple outbound messages via Python code, this seems like overkill. I don't want or need inbound email.

So my current round-trip scenario:

server-to-phone: send automated email to phonenumber@tmomail.net
phone-to-server: send sms via connectotel

Connectotel is an free offshore sms-to-email gateway service. Free is a bit of a misnomer, however, as each overseas message costs me $0.15. Even domestic messages cost $0.05 each.

There's something quite screwy in that my voicemail, which consumes orders of magnitude more transmission and storage, is free. I could easily spend over $50/month in SMS fees for less than 12K of data.

Most of the time this is a non-issue since I'll be using my hacked bluetooth server for local data communications.

svk vs. svn

2005-11-13T11:58:00.000-08:00

Fredik Lundh says "If you're using subversion for anything, you really want to use svk as a front-end, instead of svn." I installed version 1.00 of svk via apt-get.

svk is written in Perl and I notice the command line response seems very sluggish.

After creating a respository, I attempted to import a project. No response. A query to #svk chat confirms that it's failing silently. The recommendation is that I upgrade to version 1.05, but I think I'll just pass for now. Or maybe just just svk as a client for Subversion.

Upgrading to Ubuntu Breezy / XEmacs

2005-11-13T07:44:00.000-08:00

It's a rainy Sunday, a good time to upgrade my Ubuntu desktop from Hoary 5.04 to Breezy 5.10. A few hours later I'm frustrated. The upgrade process gives me errors which I spend too much time resolving. I have backups of /etc and /home/jbauer, but a system restore would not have been pleasant. I won't go into the details, but enough time spent on Ubuntu forums found the answers to most of my questions. Upgrading is a task for the steel-willed, not nervous nellies such as myself.

I prefer XEmacs to GNU/Emacs, though I could live with either. One problem with running XEmacs from the command line is the annoying messages:


Warning: Missing charsets in String to FontSet conversion
Warning: Missing charsets in String to FontSet conversion
Warning: Unable to load any usable fontset

My LANG environment is en_US.UTF-8, so my short-term fix is to create a command line script, ~/bin/xe:


LANG=C xemacs $* &

Success with SWIG

2005-11-11T10:56:00.000-08:00

It bothered me yesterday that I didn't get a Bluetooth server running with Python, so I had to go back to SWIG. The Pyrex framework is probably better for major projects, but SWIG wins out for write-once applications that won't require a lot of maintenance.

My swig file, pyobexserver.i:


    %module pyobexserver
    %{
    #include "openobex/obex.h"
    %}
    int get_my_file(char *outfile);

From the Python interpreter:


    >>> from pyobexserver import get_my_file
    >>> get_my_file('/home/jbauer/nokia/files')
    Waiting for connection...

What could be easier? From downloading SWIG to running a functional program: 30 minutes. I love Python.

Subversion: status vs. update

2005-11-10T16:43:00.000-08:00

Today I learned something important about svn status the hard way. Prior to this project, I've only used the rudiments of Subversion. For most SourceForge projects, I would run svn update. The svn status command displays all files that have been modified locally. I assumed it compared changes against the repository. So yesterday when I tried to commit some changes, I got an "Out of date" error message. I had to copy my mods elsewhere, update, manually diff the files, then commit. No big deal. I just didn't read the documentation closely enough.

Gnome: How do I get the properties of a program listed in my applications menu? Most windowing systems do this with a right mouse click. It's not obvious with gnome, so my workaround is to install the application on my panel, right-click, then remove it from the panel. This is how I discovered the "Bluetooth File Sharing" application is actually gnome-obex-server.

Speaking of gnome-obex-server (written in Python) and obexserver (written in C) -- they are incredibly limited pieces of software. They both accept incoming files from bluetooth devices -- in my case pictures from my Nokia 6600 -- but give no option to choose where you want them saved. The gnome-obex-server drops the files in your home directory, and obexserver puts everything in /tmp. Another issue with obexserver -- it shuts down after it has received the first incoming file. It's a simple matter to modify the code in obexserver, but both of these are Debian distribution apps -- obexserver even has a man page.

I looked into using Pyrex to build a Python interface to the OBEX stuff. Pyrex is an awesome piece of work, but I don't have time to wrap up all the OBEX C structures in a .pyx file. There's a bit of a learning curve to get it right. Since the interface is so simple, I might SWIG it instead. I shouldn't be wasting time on this anyway.

YAML vs. JSON

2005-11-09T14:38:00.000-08:00

I have some information that doesn't quite fit into a relational database model, though it would be convenient to store it as such. Here's an example:

>>> data = {'therapies': ['4'], 'inpatient_diagnoses': [('751.1', 'ATRESIA SMALL INTESTINE')], 'prior_conditions': ['3'], 'diagnoses_and_severity': [('879.2', 'OPN WND ANTERIOR ABDOMEN', '2'), ('751.1', 'ATRESIA SMALL INTESTINE', '4'), ('751.2', 'ATRESIA LARGE INTESTINE', '2')], 'race': ['6'], 'regimen_changed': '1', 'current_payment_sources': ['1'], 'overall_prognosis': '1', 'high_risk_factors': ['2','5'], 'inpatient_discharge_date': 2005-02-19, 'financial_factors': ['0'], 'rehab_prognosis': '1', 'inpatient_facilities': ['1'], 'changed_diagnoses': [('751.1', 'ATRESIA SMALL INTESTINE')], 'life_expectancy': '0'}

It's a combination of lists, strings, and associative arrays, supported by most modern languages. One goal is to keep the data human-readable, which precludes some kinds of binary serialization. The data could be represented as XML, but XML's just barely readable.

Matt Goodall offered the following suggestions:

I recall playing with the Python version of YAML over a year ago. It was pretty cool, though slow, and not a complete implementation. There's a certain appeal to a Python programmer in representing data with whitespace. For example, an invoice:


invoice: 34843
date   : 2001-01-23
bill-to: &id001
    given  : Chris
    family : Dumars
    address:
        lines: |
            458 Walkman Dr.
            Suite #292
        city    : Royal Oak
        state   : MI
        postal  : 48046
ship-to: *id001
product:
    - sku         : BL394D
      quantity    : 4
      description : Basketball
      price       : 450.00
    - sku         : BL4438H
      quantity    : 1
      description : Super Hoop
      price       : 2392.00
tax  : 251.42
total: 4443.52
comments:
    Late afternoon is best.
    Backup contact is Nancy
    Billsmer @ 338-4338.

I couldn't google up an emacs mode for .yml files, but I'm sure someone must have written one by now. Ultimately though, I'd prefer to keep embedded newlines out of the field data, so I next looked at JSON.


>>> import json
>>> s = json.write(data)  # data is returned as a string.
>>> d = json.read(s)      # converted back into a dictionary.

Pretty simple.

JSON wins out over YAML for this project. It was dirt simple to round-trip the example data. Speed isn't much of an issue for this particular use of JSON and I like storing the data as TEXT rather than BLOB.

There is also JSON-RPC, a remote procedure call protocol similar to XML-RPC. Interesting.

Trac: Integrated SCM & Project Management

2005-11-08T15:15:00.000-08:00

After working on my current project I have a better
appreciation for having issue tracking, a wiki, and the
Subversion revision control system integrated into
a complete package.

If you're already familiar with cvs or Subversion, Trac
is useful without having to learn a bunch of arcane
commands. It would be fairly easy for even a non-programmer
to participating in a project managed with Trac.

The Trac wiki isn't as robust as MoinMoin, but the
effortless integration with the ticket tracker
is a pleasure to use. Project members can
be issued notifications via email or RSS for tickets
that have issued, modified, or resolved.

The source browser is also well integerated. The
current version of each code module has a link to
the revision, which in turn displays all the other
files that were committed as part of the revision.

Python for Series 60

2005-11-07T00:40:00.000-08:00

Yesterday my Dad loaned me his USB bluetooth dongle. I put it on sulla, a linux box running Ubuntu 5.04. After the usual frustrations, I got my Nokia 6600 to communicate, sending files in each direction.

At last year's PyCon I met Jukka Laurila and Erik Smartt from Nokia who were demonstrating Python for the series 60 phones.Ever since then, I've been wanting to get Python running on my nokia 6600. With some help from the #pys60 channel, I was able to run the interpreter on the phone via bluetooth. Pretty cool.

I spent too much time with bluetooth and not enough on learning Nevow. Today was unseasonably warm, so I got out on the bike for a couple hours despite wanting to spend more time on development projects.