The Washington State Department of Transportation (WSDOT) operates Traffic Management Centers (TMCs) around the state to monitor traffic and identify problems. The TMCs use cameras located on the highway system and collect data from traffic detectors in the highways to get a real-time picture of traffic conditions. The most common detector WSDOT uses is the induction loop, a simple low-voltage wire coil buried in the roadway that sends an electrical pulse when a vehicle passes over it.

The collected traffic data is provided in a proprietary, yet simple, binary format from a WSDOT anonymous ftp site, ftp://webflow.wsdot.wa.gov/. The two main files we will be using are the WEBFLOW.DAT and WEBFLOW.STA files.

• WEBFLOW.DAT is a small, 6 KB binary file that contains the latest volume and occupancy data for the Seattle area TMS.
• WEBFLOW.STA is a text file database that contains the current list of detection locations for the Seattle area TMS. It is updated as new detection equipment is installed in the system.

The Python code below is heavily documented and includes the format of the binary and text files. The program retrieves the data files, parses and returns an xml stream containing the latest volume and occupancy information.

#!/usr/bin/env python

# Python WebFLOW data decoder example
# Author: Wayne Dyck

import datetime
import re
import struct
import urllib2
from xml.dom import minidom

class WebFLOW(object):
    """Returns traffic information from WSDOT's Traffic Management System.

    WEBFLOW.STA is a text file database that contains the current list of vehicle
    detection locations for the Seattle area TMS. It is updated as new detection
    equipment is installed in the system. Each detection location (also called
    a station) consists of one or more loops in a single roadway direction. The
    file is broken down as follows:
   
      Line 1: Version number
      Line 2: Station count (# of records)
      Line 3: Start of station info (records)
   
    Each record line contains a 16-byte record name, followed by the freeway
    and cross-street where the detector station is located. The record position
    is used by WEBFLOW.EXE to synchronize WEBFLOW.DAT with station definitions
    contained in the webflow maps. The file is in the same order as the data file.
    This order must be used to decipher the data file, WEBFLOW.DAT. The location
    names also appear in WebFLOW when the user clicks on a map segment.
    
    Last Line: END
   
    WEBFLOW.MSG is a text file that contains the latest text based messages from
    the TMS. These messages include current incident information, as well as
    bulletins that warn of upcoming construction closures. There are from one to
    four parts to this file:
   
    1st Byte = 0x02 Hexadecimal
      Body of a message containing Date, Heading and Message fields. The reader
      should disassemble the file with a good text editor to learn more about the
      message body format. The format of the message body does not change.
      
    Byte following message = 0x03 Hexadecimal
    
    The file can contain up to four messages in the format above, concatenated
    together.
   
    WEBFLOW.DAT is a binary file that contains the latest volume and occupancy
    data for the Seattle area TMS. Numbers given are zero based and in
    hexadecimal.
   
    Byte 0 is the version number of the WEBFLOW.STA file needed to decode it.
    Byte 1 has directional data for the I-5 and I-90 reversible express lanes.
    
      0x00 = Both Rev CLOSED
      0x01 = I-90 Rev CLOSED, I-5 Rev North
      0x02 = I-90 Rev CLOSED, I-5 Rev South
      0x10 = I-90 Rev East, I-5 Rev CLOSED
      0x20 = I-90 Rev West, I-5 Rev CLOSED
      0x11 = I-90 Rev East, I-5 Rev North
      0x12 = I-90 Rev East, I-5 Rev South
      0x21 = I-90 Rev West, I-5 Rev North
      0x22 = I-90 Rev West, I-5 Rev South
    
    Bytes 2 and 3 are reserved.
    
    The data is 1-minute data, for the time period ending on the time stamp
    given in bytes 4 through 9.
    
    Byte 4 = Year of time stamp, in hex (i.e.: 0x5F = 95)
    Byte 5 = Month of time stamp, in hex (i.e.: 0x07 = July)
    Byte 6 = Day of time stamp, in hex (i.e.: 0x19 = 25th)
    
      0x5F0719 = July 25, 95
    
    Byte 7 = Hour of time stamp, in hex (i.e.: 0x0D = 13)
    Byte 8 = Minute of time stamp, in hex (i.e.: 0x32 = 50)
    Byte 9 = Second of time stamp, in hex (i.e.: 0x2F = 47)
    
      0x0D322F = 13:50:47 hrs. Note: Clock is 24 hr
   
    Starting with byte 10, the data is in blocks of 5 bytes, the number of
    blocks is equal to the station count (see Line 2, WEBFLOW.STA).
   
    Each 5-byte block contains bit-mapped data in the following format:
    
    Bit positions are increasing right to left Byte positions are left to
    right, i.e., 1st data byte is first byte encountered in data file, etc.
   
    1st data byte:
      bit 6-7: Incident (0=No incident, 1=Tentative, 2=Occurred, 3=Continuing)
        (Note: at this time, the incident information is not active)
      bit 0-5: # of 20-second periods in data sample (3 = 1 minute, standard)
   
    2nd data byte:
      bit 7: Data validity flag (0=data NOT usable, 1=data OK)
      bit 4-6: # of loops in station (allows for several lanes per station, 0-7)
      bit 3: Reserved
      bit 0-2: High 3 bits of Scan Count (% Occupancy = Scan Count / 12)
   
    3rd data byte:
      bit 0-7: Low 8 bits of Scan Count
   
    4th data byte:
      bit 0-7: High 8 bits of Volume data (Volume is total count of vehicles)
   
    5th data byte:
      bit 0-7: Low 8 bits of Volume data
    
    The last four bytes of WEBFLOW.DAT are a 4-byte arithmetic checksum, low
    byte is first, which is the sum of all the other bytes in this file.
    
    """
    
    def getSeattle(self):
        """ Returns traffic volume, occupancy and speed data for the Seattle area. """
        
        sta = "WEBFLOW.STA"
        dat = "WEBFLOW.DAT"
        xmldata = self._getDataFiles(sta, dat)
        return xmldata

    def _getDataFiles(self, sta, dat):
        """ Retrieves the data files, parses and returns an xml stream. """
      
        doc = minidom.Document()
        stn = re.compile(r'_stn', re.IGNORECASE)
        url = "ftp://webflow.wsdot.wa.gov/"
        warning = """This program is using computer data originating from the Washington 
State Department of Transportation (WSDOT). WSDOT provides free software for public and 
private use of this data. The software is available at WWW.WSDOT.WA.GOV via the Internet. 
WSDOT may, at any time, change data formats, location, update rates or other features or 
aspects of the data. WSDOT changes to the data will adversely affect this software program. 
WSDOT is under no obligation to notify you or your software vendor of any changes to the 
data source."""

        webflow = doc.createElement("webflow")
        doc.appendChild(webflow)
        disclaimer = doc.createComment(warning)
        webflow.appendChild(disclaimer)
        
        request_sta = urllib2.Request(url + sta)
        request_dat = urllib2.Request(url + dat)

        try:
            response_sta = urllib2.urlopen(request_sta)
            response_dat = urllib2.urlopen(request_dat)
        except IOError, err:
            xml_error_message = self._getExceptionInfo(err)
            return xml_error_message
            
        lines = response_sta.read().splitlines()
        tokens = lines[0].split()
        version_sta = tokens[2]
        tokens = lines[1].split()
        stations = tokens[2]
        version_dat = ord(response_dat.read(1))
        express_lanes = ord(response_dat.read(1))
        response_dat.read(2) # Bytes 2 and 3 are reserved. Skip them.
        year = ord(response_dat.read(1)) + 2000
        month = ord(response_dat.read(1))
        day = ord(response_dat.read(1))
        hour = ord(response_dat.read(1))
        minute = ord(response_dat.read(1))
        second = ord(response_dat.read(1))
        timestamp = datetime.datetime(year, month, day, hour, minute, second)
        
        info = doc.createElement("info")
        info.setAttribute("version", str(version_dat))
        info.setAttribute("date", str(timestamp))
        webflow.appendChild(info)
        express = doc.createElement("express_lanes")
        express.setAttribute("status", str(express_lanes))
        webflow.appendChild(express)
        
        station_list = lines[2:] # skip the first two lines
        
        for i in range(int(stations)):
            tokens = station_list[i].split(',')
            byte0 = ord(response_dat.read(1))
            word0 = self._readWord(response_dat)
            word1 = self._readWord(response_dat)
            periods = byte0 & 0x3f
            incident = (byte0 & 0xc0) >> 6
            scans = word0 & 0x7ff
            loops = (word0 & 0x7000) >> 12
            valid = (word0 & 0x8000) != 0
            volume = word1 & 0xffff
            occupancy = scans / 12

            if not stn.search(tokens[0]): continue            
            
            sta = doc.createElement("station")
            sta.setAttribute("id", tokens[0])
            sta.setAttribute("valid", str(valid))
            sta.setAttribute("occupancy", str(occupancy))
            webflow.appendChild(sta)

        # Close resources
        response_sta.close()
        response_dat.close()

        return doc.toprettyxml(indent="  ")

    def _getExceptionInfo(self, err):
        doc = minidom.Document()
        webflow = doc.createElement("webflow")
        doc.appendChild(webflow)
        error = doc.createElement("error")
        webflow.appendChild(error)
        error_message = doc.createTextNode(str(err.strerror))
        error.appendChild(error_message)
        return doc.toxml("utf-8")
    
    def _readWord(self, stream):
        word = stream.read(2)     
        return struct.unpack('>H', word)[0]


def main():
    webflow = WebFLOW()
    xml = webflow.getSeattle()
    print xml


if __name__ == "__main__":
    main()