Sometimes I think it is more trouble than it’s worth, a complicated mess.

Today, both.

Wikipedia and PyMC parameterize it differently, and it is a source of continuing confusion for me, so I’m just going to write it out here and have my own reference. (Which will match with PyMC, I hope!)

The important thing about the negative binomial, as far as I’m concerned, is that it is like a Poisson distribution, but “over-dispersed”. That is to say that the standard deviation is not always the square root of the mean. So I’d like to parameterize it with a parameter for the mean and for the dispersion. This is almost what PyMC does, except it calls the dispersion parameter instead of .

The slightly less important, but still informative, thing about the negative binomial, as far as I’m concerned, is that the way it is like a Poisson distribution is very direct. A negative binomial is a Poisson that has a Gamma-distributed random variable for its rate. In other words (symbols?), is just shorthand for

Unfortunately, nobody parameterizes the Gamma distribution this way. And so things get really confusing.

The way to get unconfused is to write out the distributions, although after they’re written, you might doubt me:

The negative binomial distribution is

and the Poisson distribution is

and the Gamma distribution is

Hmm, does that help yet? If and , it all works out:

But instead of integrating it analytically (or in addition to), I am extra re-assured by seeing the results of a little PyMC model for this:

I put a notebook for making this plot in my pymc-examples repository. Love those notebooks. [pdf] [ipynb]

[pdf] [ipynb]

There is copious, and growing documentation for pandas, but it assumes a level of familiarity with python and numpy. I thought I’d write some little examples calculations that I’ve done with pandas recently to complement the real docs with some “recipes”. You don’t really need to know python to use these, let alone numpy.

To begin, here are the creation and subset routines in pandas that do the same work that my last foray into this subject accomplished with the rec_array:

import pandas
a = ['USA','USA','CAN']
b = [1,6,4]
c = [1990.1,2005.,1995.]
d = ['x','y','z']
df = pandas.DataFrame({'country': a, 'age': b, 'year': c, 'data': d})

This is cooler than a rec_array because you don’t have to dig in the docs for the constructor, and you can use a dictionary to name each column.

You can select the subset of data relevant to a particular country-year-age thusly:

df[(df['country']=='USA') & (df['age']==6) & (df['year']==2005)]

This is not as cool as a rec_array, because writing df['age'] has more characters than df.age, but I feel churlish to complain about it.
It’s good that I complained about my uncool df['age'] business, because I learned that df.age works, too, as long as you are using an up-to-date pandas.

More substantial recipe to come. Is there already a cookbook out there?

How to embed the game in the blog?

From my quick look, it appears that pyMCMC has some advanced sampling methods (like Slice sampling) that are not yet implemented for PyMC. On the other hand, it seems like PyMC has a more flexible modeling language, which permits formulation of complex models without writing out likelihood functions explicitly.

Has anyone used PyMCMC? How did it go for you?

The paper led me to the bitcoin tools repository, which I’ll have to look into in more detail in the future.

Chun-Yuen Teng, Yu-Ru Lin, Lada A. Adamic, Recipe recommendation using ingredient networks
Yong-Yeol Ahn, Sebastian E. Ahnert, James P. Bagrow, Albert-László Barabási, Flavor network and the principles of food pairing

They both have wonderfully complex network graphics, although the lack of information in these beautiful figures is acknowledged:

I’d love to combine this sort of analysis with the work on nutritional risk factors that has been going on around here recently. Did either of these papers come with a dataset I can explore?

I’ve heard about this Khan Academy, and it seems like more and more course material is appearing as tiny web videos.

Also I recently found out that there is a free, online version of the Stanford Intro AI Class taught by Peter Norvig and Sebastian Thrun, for which 56,000 students signed up. I think I accidentally did their homework.

ABSTRACT–

Surveys are one of the principal means of gathering critical data from low-income regions. However, interviewer fabrication, or curbstoning, can threaten data quality. The existing literature lacks a set of general-purpose techniques to detect curbstoning; it does not capitalize on the potential of mobile data collection tools to help detect the phenomenon; and it provides few rigorous validations of the techniques that are developed. In this talk, I propose an anomaly detection framework to develop several general-purpose algorithms that identify curbstoning.

These algorithms can take advantage of the information in user traces from mobile data collection, a potential that I will evaluate rigorously. I also propose two studies to obtain high-quality labeled data sets with which I will validate my algorithms, thus partially filling the need for more rigorous evaluations.

Good job, Ben!  Also in attendance was Aram Harrow, who was reminded of this great story of the lying professor.  I wonder, could I could pull that off?

The book argues strongly for the meaningful popularization of scientific ideas. I love the popularizers of science, and was very influenced by books like Surely You’re Joking, Mr. Feynman and Gödel, Escher, Bach when I was a youth. The modern history sections in Unscientific America trace these popularizations to Carl Sagan’s book/television series Cosmos. I should check that out.