Oliver Sherouse Writes Occasionally

on Public Policy
and Python Programming

The Big Problem With Stata

29 Oct 2014

I use Python for almost all my data work, but both in my workplace and my field more generally Stata dominates. People use Stata for a reason1, and it provides a far wider range of advanced statistical tools than you can find with Python (at least so far), but I hate working in it.

I’ve always found it hard to explain to others just why I hate it so much. You can generally get your problem solved, the help files aren’t terrible, there’s lots of Google-able help online2, you can write functions if you want to learn how. And while I find lots of little things annoying (the way you get variable values, for example, or the terrible do-file editor), the big problem was the one other people didn’t understand.

Today, however, I was re-reading some pages about the Unix Philosophy, when I saw something that hit the nail on the head. It’s Rob Pike’s Rule 5:

Rule 5. Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming

Stata only has one data structure: the dataset. A dataset is a list of columns of uniform length. You can only have one dataset open at a time.

This is the right data structure for performing the actual analysis of data—say, a regression—and the wrong data structure for literally everything else. The problem is, 90 percent of doing data work is cleaning, aligning, adjusting, aggregating, disaggregating, and generally mucking around with your source data, because source data always comes from people who hate you. And because the data structure is wrong, you’re forced to use algorithms that look like they come from an H.P. Lovecraft story.

Never having seen anything better, most Stata users seem to be resigned to doing things like creating an entire column to store a single number and writing impenetrable loops for simple tasks. Or they use sensible tools to create their datasets (increasingly Python, but also even something like Excel) and then use Stata just for the analysis.

The latter is my approach when I can’t avoid Stata entirely. But I’m really looking forward to the day when I can avoid the fundamentally flawed design of Stata altogether.

  1. In my graduate program, we started learning econometrics with a different statistical program, called SAS. SAS is…SAS is rough.

  2. I’m looking at you, R.

Progressives Need Amazon to be a Problem

20 Oct 2014

A few weeks ago, Franklin Foer wrote an article at The New Republic arguing that Amazon is now a monopoly and therefore should be broken up. The difference between Amazon and what we used to think of as monopolies, he says, is that Amazon squeezes its producers, not its customers, and consumers are complicit in the squeezing, which is just kind of assumed to be a bad thing.

Foer didn’t offer very specific recommendations, but he did point to, say AT&T which was broken up using antitrust law in the 1970s as a good example.

“That’s silly”, I thought when I first read the piece, and I didn’t expect to hear much more about it.

Today, however, Paul Krugman followed up with an op-ed that correctly identified Amazon’s relationship to its producers as a monopsony, not a monopoly1, and argued that it is totally not ok, guys.

Krugman’s argument zeros in on Amazon’s fight with publisher Hachette. Hachette won’t agree to the revenue sharing that Amazon wants, so Amazon has disadvantaged their books.2

Like Foer, Krugman calls to mind the old progressive “victories” like the breakup of Standard Oil, saying, “The robber baron era ended when we as a nation decided that some business tactics were out of line. And the question is whether we want to go back on that decision.”

I think that line explains why suddenly we’re all supposed to be up in arms about Amazon. It’s certainly not out of deep concern for book publishers. Everyone hates book publishers, who squeeze authors as much as Amazon squeezes them (and, interestingly, more than Amazon squeezes authors, at least at present).

In fact, in a sane hour, Krugman et al. would probably have no trouble agreeing that what we’re really seeing here is publishers losing value because what they do is not nearly as valuable when you don’t need to physically print all your books. Certainly they would agree that, if the market were well and truly competitive, none of the publishers would be making money anyway because profits in a competitive market go to zero.

But Amazon is a BIG BUSINESS with MARKET POWER, and BIG BUSINESSES with MARKET POWER are bad and exploitative in the progressive view of the world. The breakup of Standard Oil is a part of the progressive identity the same way that, say, the Reagan tax cuts are part of the conservative identity.

If Amazon isn’t actually hurting real people, then maybe BIG BUSINESSES with MARKET POWER aren’t always bad. Maybe the breakup of Standard Oil wasn’t all that huge a victory for real people after all. So it’s important to the progressive view of the world that Amazon be perceived as hurting people.3

Now, there’s nothing wrong with having general rules for policy, like “monopoly is bad, let’s avoid that” or “let’s not try anything for the first time at the national level.” They’re especially good when they’ve been learned over time. But the hyper-dynamic technology-driven economy, where it’s has been harder and harder to preserve market power, has presented a powerful challenge to these old progressive beliefs, and those of us not wed to them should demand that they prove themselves again.

  1. A monopoly is when you are the only one selling, a monopsony is when you are the only one buying.

  2. Bizarrely, Krugman also veers into conspiracy theory territory when he argues that Amazon wants you to read Paul Ryan’s book, but not a book about the Koch Brothers because the shipping time is different. As he puts it, “Uh-huh.”

  3. A conservative analogy might be the insistence that the Bush tax cuts paid for themselves when they probably didn’t, because acknowledging that might undermine the popular understanding of Reaganomics.

Kirzner's Inclusive Austrianism

06 Oct 2014

On Thursday, the Mercatus Center celebrated the 40th anniversary of the Nobel Prize awarded to F. A. Hayek.1. The keynote speaker was Israel Kirzner, who Thomson Reuters predicts may this year’s Nobel Prize in economics. The question he posed for the talk was what role the Prize played in reviving the Austrian school of economics, but the more interesting aspect of his talk was the way he described what it means to be an Austrian.

The central thesis Kirzner proposed was that, following the socialist calculation debate, Mises and Hayek (the leaders of the Austrian school) made important original refinements to the theory of how markets work, but that these contributions were only recognized after Hayek’s Nobel Prize—awarded for earlier and quite separate work—gave younger economics newfound reason to examine Hayek’s entire career.

The socialist calculation debate, for those not versed in economic history, was a high-profile argument in the 1920s and 1930s about whether socialism—defined as state ownership of the means of production—could actually work.

Mises and Hayek argued that when the state owned the whole structure of production from raw materials to consumer goods, there would be no way to know which uses for goods—say, iron—were most valued. In a private property system, this task is accomplished by prices, but prices don’t exists when the state simply distributes resources to other parts of the state.

A decade and half after the first formulation of that argument, socialist economist Oskar Lange came up with what would be considered the final answer. Lange argued that you could just simulate free floating prices by assigning them randomly and then adjusting based on how quickly different resources are consumed.

Lange’s answer, with some revisions, won the debate as far as the economics profession was concerned. For this and a few other reasons, the Austrians lost their reputations in the economics field, and fell into obscurity until the Hayek won the Nobel.

Kirzner argued that, during that interlude, Mises and Hayek made important refinements to and elaborations on the theory of market processes that demonstrated why Lange’s answer—and the direction of the economics profession as a whole—missed the point. He called the key idea of this period “open-ended thinking.”

Lange’s problem was that, like the now-mainstream of economics, he relied on the assumption that every part of the economy was at equilibrium—that supply and demand was perfectly balanced at every point in the economy, and that this state of affairs simply needed to be replicated within socialism. This assumption entails others—perfect competition in every market, for example, and perfect knowledge of all individuals in the economy. Economists know that these assumptions are not true, but they will generally say that they are close enough to true as to not make a difference.

The problem, from the Austrian perspective, is that there are no new products, no new ideas, no opportunities for improvement anywhere. There are a fixed, closed set of choices—and all by initial assumption! The Austrian insight was that the set of choices in a market system is constantly in flux; competition and entrepreneurship allowed for the discovery of entirely new possibilities. And these possibilities are beyond the reach of normal research and development, because in many cases no-one knows to look for them. They are unknown unknowns.

In Lange’s model and all equilibrium model these possibilities disappear, and with them goes the tether to the real economy. That’s not to say that such models have no place—the mistake in mainstream economics was to give them the only place, and set aside all consideration of the process of markets.

It is this belief, according to Kirzner, that makes one an Austrian. This bar is far more ecumenical than some might like. You do not have to believe in Austrian business cycle theory, you don’t have to throw out GDP with the bathwater, and you don’t have to worry about radical a priori reasoning. Using this criterion, Kirzner was happy to label Julian Simon an Austrian even though he was also firmly in the Chicago school. Similarly, he absolved Russ Roberts, who confessed to finding equilibrium models helpful. The point, said Kirzner, is to remember that your models are tools and not reality, and that there are more things in heaven and earth than are dreamt of in your graduate economics seminar.

By this bar I can consider myself a Austrian, and even remember the sentence that turned me into one. It’s in Arnold Kling and Nick Shultz’s book, From Poverty to Prosperity. Kling and Shultz describe mainstream economics by saying that they can explain why it makes more sense to have your shirts ironed by a laundry than to do it yourself—specialization, trade, and so on. An open-ended economist says that all of that’s true. “But have you heard of permanent press?” Innovation completely changed the terms of the question, and I don’t have to iron my shirts at all.

I don’t know why, but when I read that example, it clicked. The world expands in unexpected ways, and our analyses are only related to the real world when they account for that.

Kirzner says that belief makes me an Austrian, and that’s an Austrianism I’m quite happy to be a part of.

Should the Results of Scotland's Independence Referendum Matter?

15 Sep 2014

This Thursday, Scotland will hold a referendum on independence. The conventional wisdom holds that the vote will be close, but that the “No” vote will carry the day. While the UK government does have final say on constitutional matters (and thus on independence), everyone seems to assume that it will honor the results of the referendum.

This entire situation highlights the absurdity of making important decisions through majority-rules direct democracy. Consider the possible outcomes. If the vote is 52 percent “Yes” to 48 percent “No”, then 48 percent of the populace, some two and a half million souls,1 will find their links to what they consider their home country severed. They will be forced, against their will, to either physically leave their homes to remain within the UK or try to mentally and emotionally locate themselves in a new political unit that they do not want or recognize as their own.

If, on the other hand, the vote is 52 percent “No” to 48 percent “Yes”, then you’ve got a similarly massive portion of the nation who actively wants not to be part of it—and not at some future point, but as soon as possible. No nation can tolerate that state of affairs for long with tranquility unless we assume that the no voters are just breezy and faddish. I’m not willing to believe that.

Surely, for such an important question, fifty-plus-one doesn’t cut it. Blindly following the referendum when public opinion is divided so evenly can only lead to a sense of disaffected displacement for the losing side. Whatever the economic effects,2 the social and psychological effects of feeling disconnected from your home country matter. A shared culture, a common historical heritage, a sense of community with your fellow citizens—these are all things that my fellow libertarians like to wave away as “tribalism,” but that are in fact necessary parts of forming a human identity.

As a matter of practicality the best result on Thursday is a “No” vote, not because I’m convinced the Scotland should remain part of the UK, but because it’s a push. The Scots can gain independence in the future much more easily than they can regain union. But that extra time only matters if the Scots use it to build a real consensus. Perhaps that means that the UK government can do a better job identifying and satisfying the concerns of those who favor independence. Maybe it means that the pro-Independence side can do a better job persuading their fellow Scots that an independent Scotland is not only attainable and advisable, but a place that will still be their home.

But that consensus has to be established before either side can claim a legitimate victory. Thursday’s referendum can only reveal the contours of the problem, not the wisest solution. In fact, it has probably already told us all it can.

  1. Yes, I’m assuming that the polls mirror the country exactly. If this bothers you, round down to one million out of five total. I don’t see how that makes things much better.

  2. Not that the economic effects aren’t important, but I haven’t studied them enough to make an argument either way, and they’re beside the point I’m making.