Encoding ampersands with Python

I need to replace ampersands in a text file with the HTML entity '&'. I could simply use Python's string replace method, however, this will mess up my text if some of the ampersands have already been turned into HTML entities. The same is true if I use regular expressions to match a single '&'. What I really need to do is replace an ampersand providing it is not followed by 'amp;'.

Using negative lookahead assertion with our regular expression is the answer. Negative lookahead is used when you want to match something not followed by something else. It starts with (?! and finishes at the ).

Our expression now becomes: &(?!amp;) and means the text it contains, amp;, must not follow the expression that preceeds it.

In this example I also added an expression to not match any HTML entity numbers as well.

>>> import re
>>> s = "<Title>Eugene&#039;s Software Emporium & Arcade</Title>"
>>> pattern = re.compile('&(?!#)(?!amp;)')
>>> if pattern.search(s):
...   iterator = pattern.finditer(s)
...   for match in iterator:
...     print match.span()
... 
(38, 39)
>>> s[match.start():match.end()]
'&'
>>> 

Python - Web application frameworks

The PythonInfo Wiki defines a a web framework as,

a collection of packages or modules which allow developers to write Web applications or services without having to handle such low-level details as protocols, sockets or process/thread management.

As a testiment to Python's power and simplicity it would seem that many developers have created their own frameworks rather than use a solution already in existence. As a result one will find solutions in various stages of development and feature implementation.

I have always tried to subscribe to the basic principle of using the right tool for the job. With that in mind I have embarked on an exploratory journey to investigate some of Python's existing Web frameworks with hopes of finding one that will work for a couple of big projects I have in the works. My requirements are fairly simple; I do not want to learn a behemoth of an API that will take months to figure out, yet I do not want something so simplistic that it will expect me to handle to many low-level details. Finally, until I can bring my own server back online, the chosen framework needs to work with my current hosting provider, DreamHost.

The five high-level frameworks I am looking at include:

• Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Because Django was developed in a fast-paced newsroom environment, it was designed to make common Web-development tasks fast and easy.
• TurboGears builds on other open source projects. In TurboGears, CherryPy controllers sit at the hub of your project. This is the biggest area for integration. Providing tools that allow the controllers to more easily work with SQLObject databases, answer asynchronous calls from MochiKit and render out completed Kid templates is where the big win will come.
• Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework. It's also one of the first projects to leverage the emerging WSGI standard, which allows extensive re-use and flexibility - but only if you need it. Out of the box, Pylons aims to make web development fast, flexible and easy.
• Webware is a suite of Python packages and tools for developing object-oriented, web-based applications. The suite uses well known design patterns and includes a fast Application Server, Servlets, Python Server Pages (PSP), Object-Relational Mapping, Task Scheduling, Session Management, and many other features. Webware is very modular and easily extended.
• Zope is an open source application server for building content management systems, intranets, portals, and custom applications. The Zope community consists of hundreds of companies and thousands of developers all over the world, working on building the platform and Zope applications. Zope is written in Python.

HTML Slidy - A browser based XHTML presentation framework

While at LinuxFest Northwest 2008 in Bellingham, WA this past weekend I attended a session on natural language processing in Python presented by Sean Boisen. His slide presentation was done, not with PowerPoint, but with HTML Slidy, a browser based XHTML presentation framework. The best thing about HTML Slidy is that it is cross-browser compatible using simple XHTML, JavaScript and CSS, operates like PointPoint and best of all, is accessible.

The next time I have a presentation to make, HTML Slidy is definitely something I am going to try.

Online dining guide and restaurant reviews

The other day I managed to secure the dining-guide.org domain name.

The Dining Guide will offer restaurant and cafe patrons a place to share their experiences with others. How many times have you had a bad experience in a restaurant and later talked with a friend to learn they had a simliar one? Would you rather know where the exceptional food and service is? Would you rather know a head of time if the restaurant is family friendly? I certainly would and that is one of the reasons why I am building the site.

I have seen many sites offering variations on this same theme, however, none of them have the feature sets and usability that I will be implementing on the Dining Guide site.

Stay tuned and be sure to watch the site over the next few weeks.

Effigy Interactive Mindscape - Flash

This is one of the sites I did for my company, Effigy Interactive, back in 2002. I broke every usability and accessibility rule creating it, however, most self-promotional sites usually do.

The Flash portion of the site is meant to be atmospheric and encourage the viewer to explore and discover. I never got around to finishing the html portion as I was to busy with customer sites and other paying projects.

Linux and the Insignia Pilot Video MP3 Player

Last month I picked myself up an 8GB Insignia Pilot video MP3 player from Best Buy for Christmas.

I looked at a number of MP3 players before deciding on the Insignia Pilot. I had two main requirements: it had to work with Linux and it had to support Ogg Vorbis audio. The Insignia Pilot does both of these and more. It supports an impressive list of formats: MP3, WMA, WMA Lossless, WMA DRM, WMA Pro, OGG, WAV, Audible, MPEG4 (30 fps), WMV (30 fps) and JPEG.

The Insignia Pilot supports 320x240 MPEG4 video at 30 fps. Very nice. The installation CD includes a Windows based application that will convert video clips and images into a format compatible with the Pilot. This is great if you are running Windows, however, not so helpful if you are running Linux.

Over the next few days I learned more about video codecs and encoding than I had planned to. The Anything But iPod site was a great source of information and thanks in part to this initial thread on supported video formats, MPlayer and hours of reading the documentation for MEncoder and Xvid followed by trial and error I have a workable solution.

First copy the DVD (which you own) to your hard drive using the following,

mplayer dvd://1 -dumpstream -dumpfile dump.vob

Next is the encoding process. The Pilot's screen is 320x240, however, if the DVD is in widescreen format and you want to preserve the ratio you need to scale the video to 320x176 instead.

mencoder dump.vob -oac mp3lame -lameopts cbr:br=96 -srate 44100 -af resample=44100:0:0 -af volume=20 \ 
-ovc lavc -lavcopts vcodec=mpeg4:mbd=1:vbitrate=384 -sws 2 -vf scale=320:176,harddup \
-noskip -skiplimit 1 -ffourcc XVID -ofps 29.97 -o output.avi

With the above settings I can encode the Matrix to 481.1 MB. For me, these settings provide a reasonable trade off of size over quality. If one wants a slightly higher quality you can change the audio and video bitrates to cbr:br=128 and vbitrate=512 respectively.

Python - Recursive Directory Crawl Using Generators

I was looking for an os.walk example to crawl through a file system and found the locate function below on ActiveState's Python Cookbook site. I incorporated it into a simple routine that dumps the output to an XML file that can then be transformed using XSLT to sort and tally the results.

#!/usr/bin/env python

import os
import fnmatch
import time
from xml.dom import minidom

def locate(pattern, root=os.curdir):
    for path, dirs, files in os.walk(os.path.abspath(root)):
        for filename in fnmatch.filter(files, pattern):
            yield os.path.join(path, filename)
 
def main():
    doc = minidom.Document()
    files = doc.createElement("files")
    doc.appendChild(files)
    comment = doc.createComment("Size attribute is reported in bytes.")
    files.appendChild(comment)
    
    for i, file in enumerate(locate("*.*", "\\\\SERVER\\Share")):
        try:
            item = doc.createElement("filename")
            item.setAttribute("id", "%s" % (i))
            item.setAttribute("path", file)
            item.setAttribute("ext", os.path.splitext(file)[1].lower())
            item.setAttribute("size", "%s" % os.stat(file).st_size)
            item.setAttribute("last_modified", time.ctime(os.stat(file).st_mtime))
            files.appendChild(item)
        except OSError, e:
            print "%s => %s" % (file, e.strerror)

    fp = open('myfiles.xml', 'w')
    doc.writexml(fp, "", "  ", "\n", "iso-8859-1")
    fp.close()
    
    return    

if __name__ == "__main__":
    main()

The locate function takes two parameters; the first is a file pattern to match and the second is the directory to start the crawl from.

Invenacom - Free Online Home Inventory Service

Invenacom Home Inventory Services, Inc. is now offering their online asset management program for free.

Although you can purchase a number of home inventory software programs in stores, Invenacom's system is unique in that it is web-based. The system will automatically keep track of the value of your assets so you can be sure your home insurance policy will cover everything in the event of a loss. You would be surprised how much you actually have. The program also allows you to upload a digital photo of each item and it will automatically resize it on the fly.

All your documented valuables are stored securely on their servers rather than locally on your computer so in the event of a fire or theft you simply log into their website and you have full access to your itemized inventory.

If you are curious about the company, I found a video interview with the founders, Gerald Lau and Terry Dhut on YouTube as well as a promotional video for their service.

It's a very simple to use and efficient system. If you get a chance, take a look at it.

Linuxfest Northwest 2007 - Bellingham, WA

PythonDog and I once again made the yearly trek to Bellingham, WA for Linuxfest Northwest. This year's Fest spanned two days rather than the usual one and, in my opinion, it continues to get better each time.

The presentations I attended were varied:

• Jabber with ejabberd - Aaron Klemm, DigiPen Institute of Technology
• Open sourcing Second Life - Rob Lanphier and Phoenix Linden, Linden Lab
• From HTML to XUL, Web to Desktop - Shane Caraveo, ActiveState
• Why Plone? (An Intro) - Brian Gershon and Andrew Burkhalter, Seattle Plone Users Group
• Up and Running (Fast) - Andy Carrel, Google
• One Laptop Per Child - Jesse Keating, Red Hat
• How Sites Scale Out - Brian Aker, MySQL AB

My favorite sessions were by Linden Lab (Second Life), Google (Up and Running), Red Hat (OLPC) and MySQL (How Sites Scale Out).

Although I do not consider myself much of a virtual socialite, the concept of what Linden Lab is doing with Second Life appeals to a part of me. It is a social medium and, like any medium, it allows the creative an outlet to express oneself and hopefully, in turn, reach a receptive audience.

Listening to what Andy Carrel had to say about Google and the daily issues they face with the vast amounts of hardware and data is mind boggling. One of the things he said that set me thinking was that of programmer effectiveness. Google engineers create services that run on building-sized computing platforms. Their computer is made up of thousands of CPUs, lots of DRAM, networking devices, and disk drives. I consider myself fortunate if I have a second server to help load balance a service.

Jesse Keating with Red Hat gave a great presentation on the One Laptop Per Child (OLPC) initiative. The little green laptop is a marvel of engineering and a testiment to what they are trying to accomplish considering who their target audience is - the third world, underdeveloped countries, kids. Each child gets their own laptop to take home and bring back to school. The short and long term implications of what this could mean for their development as individuals and a nation is awe inspiring.

Next year's Linuxfest is already on my calendar. I can't wait.

Flickr - Online Photo Management and Sharing

Flickr is an online photo management and sharing application. The service offers the ability to make photos available to others, both public and private, and collaborative ways of organizing images by allowing others to categorize photos by adding comments and tags.

I have uploaded a few images into my Flickr account. Some are digital and others are scans from my 35mm, Minolta X-700 that I still use as my primary camera.

Python - SOAPpy and HTTP Authentication

The other week I was wanting to use a SOAP web service that was protected by http basic authentication. I could not find a way to do the authentication with SOAPpy. I looked everywhere for an example before I stumbled upon a version of the below code in an archived newsgroup post.

from SOAPpy import Config, HTTPTransport, SOAPAddress, WSDL

class myHTTPTransport(HTTPTransport):
    username = None
    passwd = None
    
    @classmethod
    def setAuthentication(cls,u,p):
        cls.username = u
        cls.passwd = p
          
    def call(self, addr, data, namespace, soapaction=None, encoding=None,
             http_proxy=None, config=Config):
        
        if not isinstance(addr, SOAPAddress):
            addr=SOAPAddress(addr, config)
            
        if self.username != None:
            addr.user = self.username+":"+self.passwd
            
        return HTTPTransport.call(self, addr, data, namespace, soapaction,
                                  encoding, http_proxy, config)
    

if __name__ == '__main__':
    wsdlFile = 'http://localhost/soap/wsdl/'
    myHTTPTransport.setAuthentication('gollum', 'myprecious')
    server = WSDL.Proxy(wsdlFile, transport=myHTTPTransport)
    print server.ApiVersion()

It works because you can specify your own transport to the WSDL.Proxy using Python's **kw feature. The original author subclassed the default transport in Client.HTTPTransport and added a static class method to supply the basic authentication.

IBM's developerWorks - JavaScript and Ajax Tutorial Series

One of my often visited bookmarks is IBM's developerWorks site. The site is virtual library of technical information and tutorials.

In September 2006, Brett McLaughlin, Author and Editor with O'Reilly Media Inc, concluded his six part series on JavaScript, Ajax and the Document Object Model (DOM). Part one of the series starts with a quick-paced introduction to what Ajax is and how it works, follows with the use of the XMLHttpRequest object for Web requests and understanding the HTTP status codes it returns. The remaining parts of the series focus on how to mix JavaScript and the DOM to create interactive Ajax applications.

It is a great series that I still reference now and then when in the midst of a project.

• Mastering Ajax, Part 1: Introduction to Ajax
• Mastering Ajax, Part 2: Make asynchronous requests with JavaScript and Ajax
• Mastering Ajax, Part 3: Advanced requests and responses in Ajax
• Mastering Ajax, Part 4: Exploiting DOM for Web response
• Mastering Ajax, Part 5: Manipulate the DOM
• Mastering Ajax, Part 6: Build DOM-based Web applications

Overcast of Tag Clouds

With the proliferation of tag clouds appearing on the Web and providing a visual representation of both content tags and every other imaginable type of information I wonder if we will reach a saturation point. I wonder if we will become so inundated with our interpretive clouds that they will form a virtual overcast and obscure the clarity that comes on a cloudless day.

Fedora Core 6 - GeForce3 Ti 200

I upgraded my main home computer to Fedora Core 6 (FC6) this past weekend. It has faithfully run Fedora Core 3 for the last couple of years, however, I was so impressed with FC6 on my laptop I decided to update the main computer as well.

After completing the installation I usually look to The Unofficial Fedora FAQ for answers on the usual little quirks that come with each release, however, they appear to have not updated the site for FC6. Instead, I turned to Fedora Core 6 Tips and Tricks for quick references to installing the most popular free add-on software packages.

The first thing I always do after a fresh install is update my yum configuration to include the livna and freshrpms repositories,

rpm -ihv http://ayo.freshrpms.net/fedora/linux/6/i386/RPMS.freshrpms/freshrpms-release-1.1-1.fc.noarch.rpm
rpm -ihv http://rpm.livna.org/fedora/6/i386/livna-release-6-1.noarch.rpm

Installing Xine DVD Player
With the above repositories setup, installing the Xine Video Player is as simple as entering the following yum statement,

yum install xine xine-lib-extras-nonfree libdvdcss

and letting it resolve the other package dependencies. The libdvdcss library is what allows one to play commercial DVD movies.

Installing the Nvidia Drivers
One of my other motivations for upgrading to FC6 was the new OpenGL accelerated desktop effects provided by Compiz. Upon grabbing the latest drivers from Nvidia I discovered that my video card is now listed as "legacy" and is no longer supported in the most recent driver downloads. I read that Compiz needs at least version 96xx of the Nvidia drivers to work. I had to try several different ones until I found that the NVIDIA-Linux-x86-1.0-9631-pkg1.run package worked correctly.

My final modified xorg.conf configuration file is below which contains the necessary entries to add the Nvidia OpenGL drivers as well as the transparency effects.

# Xorg configuration created by system-config-display

Section "ServerLayout"
        Identifier     "single head configuration"
        Screen      0  "Screen0" 0 0
        InputDevice    "Keyboard0" "CoreKeyboard"
EndSection

Section "Module"
        Load  "dbe"
        Load  "extmod"
        Load  "type1"
        Load  "freetype"
        Load  "glx"
EndSection

Section "InputDevice"
        Identifier  "Keyboard0"
        Driver      "kbd"
        Option      "XkbModel" "pc105"
        Option      "XkbLayout" "us"
EndSection

Section "Device"
        Identifier  "Videocard0"
        Driver      "nvidia"
        VendorName  "NVIDIA Corporation"
        Option      "AddARGBGLXVisuals" "True"
EndSection

Section "Screen"
        Identifier "Screen0"
        Device     "Videocard0"
        DefaultDepth     24
        SubSection "Display"
                Viewport   0 0
                Depth     24
        EndSubSection
EndSection

I am still amazed it runs as well as it does on my aging Pentium III 866 Mhz.

Gas Leak!

Martin Luther King, Jr. Day was punctuated earlier this morning by a knock on our door. One of the sub-contractors working next to our neighbor's house had punctured a gas line while excavating. Whoops. They advised we might want to evacuate the house, just to be on the safe side, until the gas company could cap the broken line. It seemed like a reasonable suggestion so we headed a few doors down to the development's show home and office to wait.

All in all we were only out of the house for about 15 minutes before the gas company showed up and fixed the leak. I was pleased that my house was still where I left it and that it did not end up resembling a pile of popsicle sticks.