Parse inputs → Munge → Dump

I've been evolving a script and thought I would share one aspect of the design that helped in development and maintenance.

The Python script takes around five or six machine generated files as input, manipulates, checks and derives new data, then stuffs the new data into a database.

Parse

My input data files would all appear static in path-name so I have a parser for each file and pass it a file name but I have a default name that is correct. This leads to functions looking like

def read_all_colours(fname='all_colours.lst'):
    '''{A description of what is held in the file, 
    how it is organised, and what is to be extracted}'''

    # Parsing code
    with file(fname) as f:
        # Depending on the complexity of the task
        # parsing might include regexps (using kodos)
        # or not.
        …
        # Only elementary data checks on a single files 
        # data are done here.
        …
    return all_colours

def read_xyz_file(fname='xyz_file.lst):
    …
    # Additionally for dictionaries I adopt a naming 
    # scheme of '2' so a mapping
    # of kites to colours would be called kite2colour.

    return xyz2abc

…

I try and preserve those names returned from the read_* functions and call them from the top level. (Those read_* functions may call other functions of course but for the sake of this blog entry, I've simplified things).

My global calls might look like:

if __name__ == '__main__':
    all_colours = read_all_colours()
    xyz2abc = read_xyz_file()
    …
    all_kites, kite2colour = read_kite_info()

I can switch from vim and run this up in idle, checking and sampling the global variables getting the parsers debugged and also starting to write interactive code snippets for the next phase – cross checks.

Consistency checking

def check_consistency(all_colours, xyz2abc, …, all_kites, kite2colour):
    'Take all the parsed data apply consistency checks'
    # maybe all the colours in kite2colour must be a member of all_colours
    assert set(kite2colour.values()).issubset(set(all_colours)), \
        'Some kite has an unknown colour'
    # Further checks with associated assertions
    …

the individual consistency checks can be developed in the editor then the whole script executed or just the check_consistency function updates cut-n-pasted into idle or after the parsing is done, all the data is available globally in idle so I also just mess with the data interactively, being careful to not change the parsed data, and develop checks their then paste them to the check_consistency function in the editor.

When I'm done, I have added a global call to function check_consistency(...) after all the parsing steps.

Munging

This is the part where I take the parsed & checked input data and apply the business logic to create the derived data in a form that makes it easy to output. It may be split into several functions called from the global context or just one. If there is intermediate data generated that I think may be “key” then I like to return that to the global namespace too

Once the data munging function is created and the program run in idle intermediate and final data can be inspected and further tests done quite easily as most of what you need is available when the program ends in the idle REPL environment.

Output

Straight-forward call to a function(s) that does only simple data manipulations if necessary to dump the derived data in the correct output format.

First draft program structure

## namedtuple definitions are great for program structure
## if you can keep to short field names
…

## Parser functions
…

## Checker functions
…

## Business logic functions
…

## Output functions
def write_db(our_kites, our_kite_colours, fname='business_db.sql')
…

## Global environment

if __name__ == '__main__':
    ## Parse
    all_colours = read_all_colours()
    xyz2abc = read_xyz_file()
    …
    all_kites, kite2colour = read_kite_info()
    ## Check
    check_consistency(all_colours, xyz2abc, …, all_kites, kite2colour)
    ## Munge
    our_kites, our_kite_colours, best_colours = \
        munge(all_colours, xyz2abc, …, all_kites, kite2colour)
    ## Output
    write_db(our_kites, our_kite_colours)

Testing

I then have the first script that I can slot into my design flow – I have other programs and scripts that generate its inputs and tools ready to use the database it creates. I can then slot it into the design flow and use real world, historical, pre-computed and data generated using alternate tools that do the same thing to test it. Others can test it too and from the feedback I have to debug and maintain it.

Debugging/Maintenance

I have examples of what the parsers are set up to parse in comments in their read_* functions. This helps when a bug is due to some extra complication in the structure of an input file. Usually I have to first run the script in idle then cut-n-paste the parser function guts onto the command line to debug parser fails. I'm used to it. It works for me. From the nature of the problem reported, you can run the script using the problem data and then interrogate the global data on the idle command line to trace errors to their cause.

If I've used namedtuples then that makes getting back into the code much easier. Listing just one element of a compound homogenous variable such as a list/set/dict of items would then remind you of what data you have and its structure.

Because my source is under version control, I insert print statements for debug purposes and stick '0/0' on a line to stop the interpreter as well as set breakpoints (horses for courses and all that).

Production

For that I add a better command-line interface and documentation and ensure sufficient tests are under version control too.

End bit.

This is written because I've just been working on another way to do a task and wrote the new script in the same way as I wrote the last and realized that I had converged on a structure that allowed the previous, (and other), scripts to be easily maintained.

I regularly get to “eat my own dog food” i.e. maintain what I write over the years, and have to explain my code. 'Parse inputs → Munge → Dump to output(s)' is a recurring theme so I guess it is no surprise that I seem to have converged on a development template. Maybe its time to time to apply some heat to this simulated annealing :-)