MCMC in Python: Fit a non-linear function with PyMC

Here is a recent q&a on stack overflow that I did and liked.

Leave a comment

Filed under statistics

Tabular Data in Python: Getting just the columns I want from pandas.DataFrame.describe

The Python Pandas DataFrame object has become the mainstay of my data manipulation work over the last two years. One thing that I like about it is the `.describe()` method, that computes lots of interesting things about columns of a table. I often want those results stratified, and `.groupby(col)` + `.describe()` is a powerful combination for doing that.

*But* today, and many days, I don’t want all of the things that `.describe()` describes. And the ones that I do want, I want as columns. Here is the recipe for that:

import pandas as pd

df = pd.DataFrame({'A': [0,0,0,0,1,1],
                   'B': [1,2,3,4,5,6],
                   'C': [8,9,10,11,12,13]})

df.groupby('A').describe().unstack()\
    .loc[:,(slice(None),['count','mean']),]

and out comes just what I wanted:

       B            C
   count  mean  count  mean
A
0      4   2.5      4   9.5
1      2   5.5      2  12.5

It took me a while to figure this out, and these docs helped:

http://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-by-stacking-and-unstacking

http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-xs

Here it is as a ipython notebook.

(Note: this requires Pandas version at least 0.14.)

Comments Off

Filed under software engineering

The one before that

Jake Vanderplas’s comparison of Python MCMC modules was preceded by a Bayesian polemic. In general, I find the stats philosophy war old-timey and distracting, but his comparison of confidence intervals and credible intervals is something I need to understand better.

http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/

Comments Off

Filed under statistics

MCMC in Python: a bake-off

While I’m on a microblogging spree, I’ve been meaning to link to this informative comparison of pymc, emcee, and pystan: http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4-bayesian-in-python/

Comments Off

Filed under statistics

MCMC in Python: Another thing that turns out to be hard

Here is an interesting StackOverflow question, about a model for data distributed as the sum of two uniforms with unknown support. I was surprised how hard it was for me.

http://stackoverflow.com/questions/24379868/estimate-the-parameters-of-a-random-variable-which-is-the-sum-of-uniform-random/24397044#24397044

I think the future of probabilistic programming should be to make a model for this easy to code.

Comments Off

Filed under statistics

MCMC in Python: Never… no… always check for convergence

I’ve had no teaching responsibilities over the last quarter, and I must miss it. I’ve found myself responding to PyMC questions on StackOverflow more than ever before. It is an interesting window into what is hard in Bayesian computation. Checking (and achieving) MCMC convergence is one thing that is hard. Here are some questions and answers that include it:

http://stackoverflow.com/questions/24294203/difference-between-bugs-model-and-pymc/24347102#24347102

http://stackoverflow.com/questions/24242660/pymc3-multiple-observed-values/24271760#24271760

http://stackoverflow.com/questions/24402834/fitting-power-law-function-with-pymc/24413323#24413323

Comments Off

Filed under statistics

Graduation Congratulations 2014

This is the season of graduation activities at IHME and UW. Congrats to my colleagues completing their fellowships!

Comments Off

Filed under Uncategorized