Using Pandas to stalk your neighbors

I picked up the book Python for Data Analysis as I've been seeing it mentioned in quite a few places. And so far, it's great. A very good high level overview of using Pandas. No, not the cute kind of pandas. I'm talking about the Python library for data analysis. Derp.

Anyhow, I decided to dive in and see what I could find out about my neighbors. Chapter 9 of the book goes into analyzing the 2012 Federal Election Commission Database so I loaded it up:

>>> import pandas as pd
>>> fec = pd.read_csv('P00000001-ALL.csv')

Looking into the data, there is some garbage rows. I grabbed all the Culver City zip codes (well, the zip codes I care about) at least:

>>> zips = fec.contbr_zip.unique()
>>> mask = np.array([str(x).startswith('90232') for x in zips])
>>> fec[fec.contbr_zip.isin(zips[mask])].contbr_city.value_counts()

I don't know if these come from bad data from the contributor or from the FEC so I'm just going to include everything based on zip code.

>>> culver = fec[fec.contbr_zip.isin(zips[mask])]
>>> culver.contb_receipt_amt.sum()

Fifty-eight grand! Nice going Culver City!

Now let's see who got the money:

>>> culver.pivot_table('contb_receipt_amt', rows='cand_nm', aggfunc=sum)
Huntsman, Jon                      4500
Obama, Barack                     50381
Paul, Ron                           500
Roemer, Charles E. 'Buddy' III      110
Romney, Mitt                       2850

That's kind of interesting...Huntsman got more money from the 90232 than Romney.

Now, let's check out the occupations that contributed the most:

>>> culver.pivot_table('contb_receipt_amt', rows='contbr_occupation',
... aggfunc=sum).order(ascending=False).head(10)
RETIRED                               7272.0
ACCOUNT MANAGER                       5000.0
PROFESSOR                             2800.5
PRESIDENT & C.E.O.                    2500.0
GALLERY OWNER                         2500.0
BOOKKEEPER                            2500.0
HOMEMAKER                             1971.0
INTERIOR DESIGNER                     1500.0
WRITER                                1410.0

Retirees going large. That's kind of interesting. Let's look at that.

>>> culver[culver.contbr_occupation == 'RETIRED'].pivot_table(
... 'contb_receipt_amt', rows='cand_nm', aggfunc=sum)
Obama, Barack                     7162
Roemer, Charles E. 'Buddy' III      10
Romney, Mitt                       100

Maybe I misunderstand our local retirees (at least the ones I've met) but this was surprising to me. I really expected Romney to come out on top.

I think that's enough peeking into my neighbors contributions habits for one night. I have to say Pandas makes this sort of thing really easy. I've only scratched the surface here. There's lots more that one can do (mathematically speaking) with Panads. Python for Data Analysis gives you a really good introduction to Pandas and then the webiste fills in the gaps.

Python for Data Analysis and Panads get two thumbs up from me. Thanks to O'Reilly and Wes McKinney.

Add a comment
Category: programming Tags:

Python and Spotify Together At Last

Here's a simple script to show the currently playing song in Spotify. All you have to do is put it on your $PATH and run:

> nowplaying
The Unsinkable Fats Domino by Guided By Voices

While this is handy in and of itself (to some people), if you are running GNU Screen, you can have it output the currently playing song in your statusline. Just add the following to your .screenrc:

backtick 101 5 5 /home/YOURNAME/bin/nowplaying
hardstatus string  '%101`'

If you already have a hardstatus (or caption) line, you'll just need to add %101 in there somewhere. Then, you'll have the currently playing song easily available.

For info about what the above does, see the GNU Screen manual about backtick.

Here's the entire script:

#!/usr/bin/env python
"""Spit out the currently playing song."""
import dbus
import sys

    bus = dbus.Bus(dbus.Bus.TYPE_SESSION)
    spotify = bus.get_object('com.spotify.qt','/')
    info = spotify.GetMetadata()
except dbus.exceptions.DBusException:
    print('Spotify is not running')

track = {}
trackMap = { 'artist'    : 'xesam:artist',
             'album'     : 'xesam:album',
             'title'     : 'xesam:title'

for key, value in trackMap.items():
    if not value in info:
    piece = info[value]
    if isinstance(piece, list):
        piece = ', '.join(piece)

    track[key] = piece.encode('utf-8')

if track.has_key('title') and track.has_key('artist'):
    print('%s by %s' % (track['title'], track['artist']))
    print('No song playing')

If you want Spotify to use the built-in notifier in Ubuntu, then by all means check out Spotify-notify. It also adds support for media keys.

Add a comment
Category: music Tags:

Books I Read in 2011

So, it's been over a year since my last "Books I Read" post so it's time I suppose. Only 28 books this year but this includes the behemoths that make up the first five books of the A Song Ice and Fire series so my page count is probably a bit higher this year. Hrm, that might be the job for a different script.

Looking over the list, books that stand out for me this year include The Woman in White, State of Wonder and A Visit from the Goon Squad.

If you want up to the minute updates, I suggest you follow me on Goodreads.

Full disclosure: Most of the book links on this page are Amazon affiliate links so I get some coin when you buy something.

Add a comment
Category: reading Tags:

The Big Easy 2011

TL;DR:I use quite a bit of software.

For no particular reason other than I need to post something to make up for all the times I didn't post (and because I love making lists), here's a list of the software that I use day to day.

  • Ubuntu: My main machine is running Ubuntu 11.04. I've got other boxes running older versions of Ubuntu (as well as OS X) but this is where my hat hangs for the time being. Speaking of hanging, Unity almost made me want to hang myself but I think I have it dialed into where I like it. Hidden inside the Compiz settings are some sweet keyboard shortcuts for windows management, which is the only reason I tried out xmonad for as long as I did.
  • GNU Emacs: Emacs is probably where I spend about 90% of my day. I've got it to the point where I hardly touch the configuration files anymore. People complain that emacs users spend tons of time fiddling with settings but you have to average that over the lifespan of you using the software. So the time setting it up isn't that much really. Emacs probably demands its own post about all the packages that I use come to think of it.
  • Google Chrome: Yep, I ditched Firefox. Chrome just seems a lot faster and the developer tools are built in and rock solid. Seeing as how Firebug was staring to cause me to stab my eyes out, I'm quite happy now.
  • Dropbox: This is a service that I actually pay for. At my last job, dropbox was blocked and it made my life kind hellish. Without it, all my ebooks, projects and personal wiki are inaccessible.
  • KeePassX: The older the get, the more I forget. And when it comes to the bazillions of passwords I need to remember for various sites, I rely on KeePassX. Open source and available on multiple platforms, it's a keeper. Har har! Get it? Keeper. KeePassX. I should move over to marketing.
  • Spotify: Another service that I pay for. The huge selection of songs and the integration with Facebook make it kind of hard to pass up. I mention Facebook because it's good fun to queue up weird songs so they end up on your wall and your Mom gets to see it.
  • Git: Yeah, I spend way too much time in git but I use it and it suits my software development needs. So much so, that I use git to interact with our Subversion repository at work.
  • Remeber the Milk: Another piece of software that I pay for. I know this is technically a web service but I have software installed for it on my phone and tablet so I'm lumping this into the software post.
  • urxvt: I'm using this less and less now that I've been using Emacs to run my shell but when I need to run complex commands, this is the termianal I turn to. My shell of choice is Zsh.
  • Oh My Zsh!: Speaking of Zsh, this is a great collection of very useful Zsh configurations and aliases which make working with Zsh very very nice.
  • homedir: I don't know how I exactly found homedir but it's very sweet. It's basically a small package manager for your home directory. I use to keep all configurations the same across machines. And where I need a machine specific configuration, homedir comes to the rescue. I don't really use this day to day but thought it was worth mentioning because it's saved my bacon a few times.

That about wraps up what I'm using day to day. I use other things on-and-off (like LibreOffice -- shudder) but didn't think they were worth putting in the post.

Add a comment
Category: personal Tags:

Moustaches For Good!

Sooo...hey! How ya' doing? Long time no see. Hope you've been well. Aside from being broken up over the breakdown of Google Reader, I've also been busy growing a moustache for Movember.

You can donate to my cause at Feel free to pass it around!

Add a comment
Category: personal Tags:

JSLint and git blame for fun and profit

For some unknown reason, we have a problem with superfluous trailing commas at work in our JavaScript. It's probably because we have a bunch of Perl developers writing JavaScript. Unfortunately, this doesn't play well in a world with Internet Explorer.

JSLint makes this easy enough to track down. But I wanted something more. I wanted to know who the culprits were. Thus, I whipped up this little script that'll take a list of files and tell you who committed a file with a trailing comma in it. Besides, JSLint, You'll need Rhino and a script to execute commands (named runtime.js). I'll try and post that later.

// blame_comma.js -- loop through the files passed in and see who has
// commas in them

// find ../htdocs/js/ECM -type f -name "*.js" -print | \
// xargs java -classpath \
//   /usr/share/yuicompressor-2.4.2/lib/rhino-1.6R7.jar \
// blame_comma.js


(function(args) {
    var bad_files = {};
    for (var i = 0; i < args.length; i++) {
        var file = args[i],
            js = readFile(file);
        var success = JSLINT(js, {
            browser : true,
            undef   : true,
            newcap  : false,
            indent  : 4,
            predef: ["Ext","ECM","ActiveXObject","window",
        if (!success) {
            var errors = JSLINT.errors;
            for (var j=0;j<errors.length;j++) {
                var e = errors[j];
                if (e && e.reason && e.reason.match(/Extra comma/)) {
                    var cmd = 'git blame -L' + e.line + ', ' +
                              e.line + ' -- ' + file;
                    var output = runtime.exec( cmd );
                    if (!bad_files[file]) {
                        bad_files[file] = [];
    for (var key in bad_files) {
        if (bad_files.hasOwnProperty(key)) {
            print("\n" + key);
            var errors = bad_files[key];
            for (var i = 0; i<errors.length;i++) {

Like most of the stuff I seem to do lately, this is available as a gist on GitHub. Let me know if you see anything you like.

Add a comment
Category: programming Tags:

Beautify your JavaScript in Emacs

I know you'll find this hard to believe dear reader but I'm a big fan of using Emacs to write JavaScript. One thing that irked me in the past is that none of the libraries got the indentation and other formatting how I wanted. Luckily, I recently stumbled onto js-beautify (via the most excellent jsFiddle.

Lo and behold, there is a command line interface to beautifying JavaScript! Good thing Emacs can call shell commands on text so easily. Thus, we have:

;;; js-beautify.el -- beautify some js code

(defgroup js-beautify nil
  "Use jsbeautify to beautify some js"
  :group 'editing)

(defcustom js-beautify-args "--jslint-happy --brace-style=end-expand
  "Arguments to pass to jsbeautify script"
  :type '(string)
  :group 'js-beautify)

(defcustom js-beautify-path "~/projects/js-beautify/python/"
  "Path to jsbeautifier python file"
  :type '(string)
  :group 'js-beautify)

(defun js-beautify ()
  "Beautify a region of javascript using the code from"
  (let ((orig-point (point)))
    (unless (mark)
    (shell-command-on-region (point)
                             (concat "python "
                                     " --stdin "
                             nil t)
    (goto-char orig-point)))

(provide 'js-beautify)
;;; js-beautify.el ends here

I like it so much I have bound to M-t (for tidy) in a mode hook:

(local-set-key "\M-t" 'js-beautify)

If you want to follow any updates, I've put this snippet up as a Gist on GitHub so feel free to clone and send me any pull requests.

Add a comment
Category: editor Tags:

Toggler in Ext JS

Since it has been awhile since I last posted and we started using Ext JS at work, I thought I'd post a little snippet JavaScript for setting up a link that toggles something.

// create a toggler there a better way?
var toggler = new Ext.BoxComponent({
    autoEl: {
        tag: 'a',
        href: '#',
        html: 'All'
    listeners: {
        'render': function(comp) {
            var el = comp.getEl();
                'click': {
                    scope: comp,
                    fn: function(event, element) {
                        var text = this.getEl().dom.innerHTML;
                        text = text.toggle('All', 'None');
                        // do your toggling here

Because it's an instance of Ext.Component it can easily be added to a Ext.Container so it's a tad more reusable than a straight up select. At least I think so.

Questions and comments welcome.

Add a comment
Category: programming Tags:

Books I Read in 2010

I was messing around with the Goodreads API and generated a list of books I read in 2010. Here they are in reverse order:

Only 32 books in a year! That's kind of low for me, even with my Kindle by my side. Such is life I suppose. I really enjoyed How to Live Safely in a Science Fictional Universe by Charles Yu, the Hunger Games series and Pragmatic Thinking and Learning: Refactor Your Wetware by Andy Hunt if anyone is looking for a "Best of."

Full disclosure: These are Amazon affiliate links so if you happen to buy one through a link, I'll get some shekels.

Add a comment
Category: reading Tags:

Git script to update Bugzilla

At my work we use git and Bugzilla for source code control and bug tracking respectively. We have a process where bugfixes are commited to a branch off of master with the name of bugfix/bz1234 where 1234 is the id of the issue in our bugzilla system.

After a fix is completed, a merge request is filed in our bugzilla system to merge this branch back into master (developers don't have write privileges to master). We are in the process of writing up a hook to parse commit messages to automate all of this but it's taking some time.

So, in the interest of making my life a bit easier until then, I took advantage of Bugzilla's excellent to parse incoming emails to edit tickets and wrote up a command to file fix and merge tickets for me.

I simply put git-bz on my $PATH and call it thusly:

> git bz fix
> git fix 1.2.3 hotfix | sendmail -t

Note: Our merge tickets require a milestone and a version for the merge ticket, thus the extra arguments for the merge.

Your system is probably radically different than ours but I thought it'd be interesting to post something here showing how powerful git is. Let me know what you think.

#!/usr/bin/env perl
# git-bz -- given a version and target, print out a merge
#           email for sending to bz

use warnings;
use strict;

use Carp;
use Git;

# map a shorthand version to the one bz expects for merges
my %versions = (
    'mne'    => 'Merge (M&E/Patch)',
    'rc'     => 'Merge (RC/Major)',
    'hotfix' => 'Merge (Hotfix)'

# get the action and config info
my $action   = shift || '';
my $bz_email = Git::config('');

if (! $bz_email) {
    croak 'No bugzilla email found';

# start doing the work
if ( $action eq 'fix' ) {
    my $user_email = get_user_email();
    my $bz_info = get_bz_info();

    print <<"MAIL";
To: $bz_email
From: $user_email
Subject: Issue $bz_info->{bug} fixed

\@bug_id = $bz_info->{bug}
\@bug_status = RESOLVED
\@resolution = FIXED

Fixed in $bz_info->{branch} pending merge.


} elsif ( $action eq 'merge' ) {
    my $target_milestone = shift || '---';
    my $merge_to = shift;
    my $version  = ( $merge_to && $versions{$merge_to} ) ||
    my $bz_info  = get_bz_info();

    my $user_email = get_user_email();

    if ( !defined $bz_info->{bug} ) {
        croak "No bug id found in branch name";

    my $bug_id = $bz_info->{bug};
    my $bz_branch = $bz_info->{branch};

 # get the remote branch, url and name
    my @remote_info
        = Git::command( 'config', '--get-regexp', '^branch\.'
                        . $bz_branch );
    my ( $remote, $remote_branch );
    for my $line (@remote_info) {
        if ( $line =~ /\.remote\s+(.+)/x ) {
            $remote = $1;
        } elsif ( $line =~ /merge\s+((refs\/)?heads\/)?(.+)/x ) {
            $remote_branch = $3;

    my $remote_url = Git::command_oneline( 'config', '--get-regexp',
        'remote.' . $remote . '.url' );
    if ( $remote_url =~ m/url\s+(.+)/x ) {
        $remote_url = $1;

 # get the commits to show
    my $commits = Git::command(
        'log',            "$remote/$remote_branch..",
        '--pretty=short', '--name-status'

 # finally print out the email message
    print <<"MAIL";
To: $bz_email
From: $user_email
Subject: Please merge $bz_branch to $remote_branch

\@blocked = $bug_id
\@product = Application Defects
\@component = Merge Request
\@version = $version
\@target_milestone = $target_milestone

Please merge $bz_branch to the remote branch $remote_branch

Merge: $bz_branch
To: $remote_branch
Repository: $remote_url


} else {
    croak <<"USAGE";
error: Unknon subcommand: $action
usage: git bz fix <target>
   or: git bz merge <merge_to> <target>


# get current branch and bug ID
sub get_bz_info {
    my $bz_branch = Git::command_oneline( 'symbolic-ref', 'HEAD' );
    $bz_branch =~ s{^refs\/heads/}{}x;
    my ($bug_id) = $bz_branch =~ m/bz(\d+)/x;
    return { branch => $bz_branch, bug => $bug_id };

sub get_user_email {
    my $email = Git::command_oneline( 'config', '--get', '' );
    return $email;

If you want to use it, you'll need to set up a config to point to your own bz email.

> git config

This script is available as a gist on github.

Add a comment
Category: programming Tags:

© Seth Mason 2014

Themed using aboutwilson

Powered by Pelican