Python - Recursive Directory Crawl Using Generators

I was looking for an os.walk example to crawl through a file system and found the locate function below on ActiveState's Python Cookbook site. I incorporated it into a simple routine that dumps the output to an XML file that can then be transformed using XSLT to sort and tally the results.

#!/usr/bin/env python

import os
import fnmatch
import time
from xml.dom import minidom

def locate(pattern, root=os.curdir):
    for path, dirs, files in os.walk(os.path.abspath(root)):
        for filename in fnmatch.filter(files, pattern):
            yield os.path.join(path, filename)
 
def main():
    doc = minidom.Document()
    files = doc.createElement("files")
    doc.appendChild(files)
    comment = doc.createComment("Size attribute is reported in bytes.")
    files.appendChild(comment)
    
    for i, file in enumerate(locate("*.*", "\\\\SERVER\\Share")):
        try:
            item = doc.createElement("filename")
            item.setAttribute("id", "%s" % (i))
            item.setAttribute("path", file)
            item.setAttribute("ext", os.path.splitext(file)[1].lower())
            item.setAttribute("size", "%s" % os.stat(file).st_size)
            item.setAttribute("last_modified", time.ctime(os.stat(file).st_mtime))
            files.appendChild(item)
        except OSError, e:
            print "%s => %s" % (file, e.strerror)

    fp = open('myfiles.xml', 'w')
    doc.writexml(fp, "", "  ", "\n", "iso-8859-1")
    fp.close()
    
    return    

if __name__ == "__main__":
    main()

The locate function takes two parameters; the first is a file pattern to match and the second is the directory to start the crawl from.

XSLT to convert from a 24-hour timestamp

I had an XML file in which the timestamp for the file creation was in a 24-hour GMT format. I needed to convert the display date and time to be in the format of, "XXX as of 12:00 a.m., on Oct 17, 2005"

I found some of the code for converting the month name on the internet and the rest I hacked together to handle the 24-hour and GMT conversion.

Longer lines are split with a "\" character which should be removed before you use the code.

<!-- Start: XSLT date and time formatting template -->
<!-- e.g. 2005-07-18T17:59:30.187-08:00 -->

<xsl:template name="format-date-time">
  <xsl:param name="date" />
  <xsl:variable name="year" select="substring($date, 1, 4)" />
  <xsl:variable name="month" select="substring($date, 6, 2)" />
  <xsl:variable name="day" select="substring($date, 9, 2)" />
  <xsl:variable name="day2" select="translate($day, '0', '')" />
  <xsl:variable name="monthName" \ 
    select="substring('JanFebMarAprMayJunJulAugSebOctNovDec', \
    substring-before(substring-after($date,'-'),'-')*3-2,3)" />
  <xsl:variable name="hour24" select="substring($date, 12, 2) - 8" />
  <xsl:variable name="minute" select="substring($date, 15, 2)" />
  <xsl:variable name="second" select="substring($date, 18, 2)" />

  <xsl:variable name="hour12">
    <xsl:choose>
      <xsl:when test="$hour24 < 0">
        <xsl:value-of select="12 + $hour24" />
      </xsl:when>
      <xsl:when test="$hour24 = 0">
        <xsl:value-of select="12" />
      </xsl:when>
      <xsl:when test="$hour24 = 12">
        <xsl:value-of select="$hour24" />
      </xsl:when>
      <xsl:when test="$hour24 > 12">
        <xsl:value-of select="$hour24 - 12" />
      </xsl:when>
      <xsl:otherwise>
        <xsl:value-of select="$hour24" />
      </xsl:otherwise>
    </xsl:choose>
  </xsl:variable>
  
  <xsl:variable name="meridiem">
    <xsl:choose>
      <xsl:when test="$hour24 < 0">p.m.</xsl:when>
      <xsl:when test="$hour24 >= 12">p.m.</xsl:when>
      <xsl:otherwise>a.m.</xsl:otherwise>
    </xsl:choose>
  </xsl:variable>

  <xsl:value-of select="concat($hour12, ':', $minute, ':', $second, ' ', \
    $meridiem, ' on ', $monthName, ' ', $day2, ', ', $year)" />
  
</xsl:template>

<!-- End: XSLT date and time formatting template -->

C# XML data export from MySQL

I originally wrote this in C# at work under Visual Studio to connect to a Microsoft SQL Server on my workstation. One of the tables contained 117 records of archived articles for the web site. We needed to extract the data and output it into an XML file. Once I got it working I decided to see how well it would compile and run under Mono, an open source development platform based on the .NET framework, and have it connect instead to a MySQL database.

In order to have my .NET application connect to MySQL instead of the Microsoft SQL Server I needed to download and install the MySQL Connector/Net driver. The MySQL Connector/Net is a fully-managed ADO.NET driver written in 100% pure C#. It was then simply a case of adding the new MySQL Namespace to the code and substituting the different Microsoft specific classes for the MySQL ones.

This is the resulting code,

// Filename: ExportXMLData.cs

using System;
using System.Data;
using MySql.Data;
using MySql.Data.MySqlClient;

namespace ExportXMLData
{
  class ExportXML
  {
    static void Main(string[] args)
    {
      ExportXML exportXML = new ExportXML();
      exportXML.Run();
    }
                
    private void Run()
    {
      // Change the variables to reflect values needed for
      // your computer and database properties.
      string Database = "";
      string Server = "localhost";
      string User = "";
      string Pass = "";
      string TableName = "";
      string XMLRootNodeName = "Root";
      string OutputFileName = "output.xml";

      string conn = 
	"Database=" + Database + ";" + 
	"Server=" + Server + ";" +
	"Uid=" + User + ";" +
	"Pwd=" + Pass;

      MySqlConnection connection = new MySqlConnection(conn);
      MySqlDataAdapter adapter = new MySqlDataAdapter();
      adapter.TableMappings.Add("Table", TableName);
      connection.Open();
      MySqlCommand query = new MySqlCommand("SELECT * FROM "
					    + TableName, connection);
      query.CommandType = CommandType.Text;
      adapter.SelectCommand = query;
      DataSet ds = new DataSet(XMLRootNodeName);
      adapter.Fill(ds);
      connection.Close();

      ds.WriteXml(OutputFileName, XmlWriteMode.WriteSchema);
    }
  }
}

You will need to compile the code as follows so it finds the necessary libraries,

mcs ExportXMLData.cs -r System.Data -r MySql.Data

You can then run the C# program with,

mono ExportXMLData.exe

The result should be an XML file in your working directory containing your database table structure and data.

In a later post I will show you how you can use XSL Transformations (XSLT) to apply a stylesheet and change the display formatting of the XML file itself.