Development Process with Git

I have been using various version control tools for years, however it has taken me a long time to make version control a core part of my work process. All of the ideas in this post come from other people... so as usual I am indebted to my colleagues for making me a more productive coder.

I now generally create a repo in my remote master, and then push the initial commit. This requires running the following on you local machine.

git init 

git add *
git commit -m "Initial Commit"
git remote add origin git@remote.com:project.git
git push -u origin master

If you want other people to work with you, then they can now clone the project.

git clone git@remote.com:project.git

Now for the interesting part: when you want to commit something into the repo but your branch has diverged from master. You can of course just run git pull and it will merge the results, unless there are drastic conflicts you need to resolve. But this creates a non-linear commit lineage, you can't look at this list of commits as one single history. To get a linear commit history you need to do the following.

git fetct 

git rebase

This will rewind your commit history to what it was before the changes in master were applied. It will apply the changes from the master branch, and then apply your sequence of changes on top. Voila, linear commit history.

There are a number of other key ideas in effectively using git.

Tag your releases

This is pretty simple, every time I push a major piece of code to production, release an App on the App Store etc, then I make sure to tag the contents of the repository. So that release can be recovered with minimal stuffing around.

git tag -a v1.2 -m 'Version 1.2 - Better than Version 1.1'

Create Branches

Branching can be a scary experience the first time you do it. However, if you want to do major refactorizations of code that is in production and needs to be supported while those changes are being made, then this is the only way to do it. In fact I don't know how people did these before tools like git.

Create a branch and check it out in a single command

git checkout -b refactor

Go back to master if you need to patch something quickly

git checkout master

When you are done go back to your branch

git checkout refactor

If you changed things in the master that you need inside your refactorization work, then merge master into it:

git merge master

Once you are done with all of your changes, run your tests etc, then your can merge that branch back into your production code.

git checkout master

git merge refactor

Once it is all merged and deployed, you don't need the branch so delete it.

git branch -d refactor

Using AWK for Data Science

Over the years I have become convinced that one of the essential tools needed by anyone whose job consists of working with data is the unix scripting language AWK. It will save you an awful lot of time when it comes to processing raw text data files.

For example, taking a large delimited file of some sort and pre-processing its columns to pull out just the data you want, perform basic calculations or prepare it for entry into a program that requires a specific format.

AWK has saved me countless hours over the years, so now I am writing a super brief primer that should not only convince you it is worth learning but show you some examples.

The first thing you need to know about AWK is that it is data driven, unlike most other languages for which execution is constrained largely by procedural layout of the instructions. AWK instructions are defined by patterns in the data to which actions should be applied. If you are familiar with the regular expression type control structures available in PERL then this should seem like a comfortable idea.

The programs are also data driven in the sense that the entire program is applied to every line of the file (as long as there are patterns that match) and furthermore the program has inbuilt access to the columns of data inside the file through the $0, $1, 2 ... variables: where $0 contains the entire line and $1 upwards has the data from individuals columns. By default the columns are expected to be TAB separated, but you can follow your AWK script with FS=',' to use a comma or any other field separator.

To run a simple AWK script type:

   >awk 'AWK_SCRIPT_HERE' FILE_TO_PROCESS

The basic syntax of the scripts themselves consists of mutiple pattern action pairs defined like this:
PATTERN {ACTION}

One need not include an PATTERN, in which case the action will be applied to every line inside the file to which the program is applied.

So for example, the following program will out the sum of columns 3 and 4

>awk '{print $3+$4}' FILENAME

If we only wanted this to happen when column 1 contained the value 'COSTS' we have a number of options. We could simply use the pattern equivalent of an IF statement as follows:

>awk '$1=="COSTS" {print $3+$4} FILENAME

Alternatively we could use a PATTERN expression as follows

>awk '/COSTS/ {print $3+$4} FILENAME

The problem with the second solution is that it if for some reason the word COSTS can appear in other fields or places in the file then may not get the results you are looking for. There can be a trade off for using the power and flexibility of the regular expression patterns and their ability to lull us into a false sense of security about what they are doing.

There are several special execution paths that can be included in the program. In place of the pattern you can include the reserved words BEGIN or END in order to execute routine before or after the file processing occurs. This is particularly useful for doing something like calculating a MEAN, shown below:

>awk '{sum+=$1; count+=1} END {print sum/count}' FILENAME

By now you should be seeing the appeal of AWK. You can manipulate your data quickly with small scripts that do not require loading an enormous file into a spreadsheet, or writing a more complicated JAVA or PYTHON program.

Finally here are a few of the kinds of tasks that I do with AWK all the time

1)  Convert some file with 10 or more columns into one with a sum of a few and reformating the others:

>awk '{print toupper($1) "," ($3/100) "," ($2+$4-$5)}' FILENAME


2) Calculate the Mean and Standard Deviation on a column. (The following is fora sample, just change the n-1 to n for a complete population.

> awk 'pass==1 {sum+=$1; n+=1} pass==2 {mean=sum/(n-1); ssd+=($1-mean)*($1-mean)} END {print sqrt(ssd/(n-1))}' pass=1 FILENAME pass=2 FILENAME


3) Calculate the Pearson correlation coefficient between a pair of columns. Again for a sample of the data. Change n-1 to n to do the calculation on the entire population data.

> awk 'pass==1 {sx+=$1; sy+=$2; n+=1} pass==2 {mx=sx/(n-1); my=sy/(n-1); cov+=($1-mx)*($2-my); ssdx+=($1-mx)*($1-mx); ssdy+=($2-my)*($2-my);} END {print cov / ( sqrt(ssdx) * sqrt(ssdy) ) }' pass=1 FILENAME pass=2 FILENAME

If you have any great tips for getting more out of AWK let me know, I am always looking for shortcuts.

Configuring Apache for a Local Site on Ubuntu

Introduction

This is a simple task, setting up my local machine so that I can browse directly to a hostname such as "mysite" and Apache will find the right project. This is a great way to test that paths will work as you expect when a site goes onto the production server. You will just need a config file in the site so it knows when it is on the development server (your local machine) and when it is live.

As simple as this is, I always have to look it up every time I do it.

So, in the interests of improving my own efficiency and maybe helping someone else I am blogging my process.

Dependencies

Ubuntu Lucid: 10.04(Check this with:  cat /etc/lsb-release )

Apache/2.2.14 (Ubuntu)
(Check this with: /usr/sbin/apache2 -v )

Process

First go add the new site to yours hosts file, edit

sudo vi /etc/hosts

Then change or add the line:

127.0.0.1       localhost mysite


Next you need to configure Apache to recognise the site. You need to create a config file for your site in the sites-available directory:

/etc/apache2/sites-available/mysite

with something like the following contents:

<VirtualHost *:80>
    ServerName mysite
    DocumentRoot /home/username/mysite/www
    <Directory /home/username/mysite/www>
        Options Indexes FollowSymLinks Includes
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>
    RewriteEngine On
    RewriteOptions inherit
</VirtualHost>


Then, you just need to enable the site with the apache script a2ensite, like thus:

sudo a2ensite mysite

Then reload apache

sudo /etc/init.d/apache2 reload

...and voila!!! You can now browse directly to http://mysite

Python Please

I have friends who swear by Python.

They rarely program with any other language, and I understand in theory the appeal. You have a language that forces you to write nicely formatted code just to make it work. You do away with redundant structure imposing syntax like braces and semicolons. No longer do you have to waste time formatting someone else's crappy code before you can work with it.

What I can't understand is how they manage to live with its horrendous approach to string processing. Python is a late bound weakly typed interpreted language with an approach to string processing that belongs in C or Assembly language. At that, all the purists are going to cry:

"Just because you don't understand encodings!!!"

Sigh. Yes. Every time I have to work with Python I go and reread Joel's fanatstic article: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!), just to make sure I haven't missed something. Every time, I confirm that I am not a complete idiot, and I come back to wrestle with python and try and work out where in the process of passing a string around I went wrong.

The problem is partly that I use Python predominantly to scrape webpages. This means that I am always loading up badly formatted text with incomplete or missing meta-data. So to be fair maybe programmers who do not engage in this process never see the problems I see. But it is not only me, look at this thread on Stackoverflow to see how ridiculous the situation is.

I want to propose something to Python enthusiasts. Just say you are right, and the problems are entirely mine (real python programmers love having to monitor string encodings continuously). Ok sure, then:

Why not have a mode for the language that will just force all strings to be a single encoding, say UTF-8 ?

The chorus will yell back, we do. You just do: X-Y-Z

Well people I have tried all of those X-Y-Zs and they do not work. Perhaps again it something to do with my approach. I use a bunch of libraries to process the data, maybe urllib library, or beautiful soup which I use to parse things. I don't know, I am not an expert, and I shouldn't need to be just to parse strings reliably.

I don't understand why it just doesn't work. I have never wasted so much time dealing with string coding problems with any other language than I have with Python.

It should not be so hard. It really shouldn't.