I use Python for almost all my data work, but both in my workplace and my field
more generally Stata dominates. People use Stata for a reason, and it provides
a far wider range of advanced statistical tools than you can find with Python
(at least so far), but I hate working in it.
I’ve always found it hard to explain to others just why I hate it so much.
You can generally get your problem solved, the help files aren’t terrible,
there’s lots of Google-able help online, you can write functions if you
want to learn how. And while I find lots of little things annoying (the way
you get variable values, for example, or the terrible do-file editor), the big
problem was the one other people didn’t understand.
Today, however, I was re-reading some pages about the Unix Philosophy, when
I saw something that hit the nail on the head. It’s Rob Pike’s Rule 5:
Rule 5. Data dominates. If you’ve chosen the right data structures and
organized things well, the algorithms will almost always be self-evident.
Data structures, not algorithms, are central to programming
Stata only has one data structure: the dataset. A dataset is a list of columns
of uniform length. You can only have one dataset open at a time.
This is the right data structure for performing the actual analysis of
data—say, a regression—and the wrong data structure for literally
everything else. The problem is, 90 percent of doing data work is cleaning,
aligning, adjusting, aggregating, disaggregating, and generally mucking around
with your source data, because source data always comes from people who hate
you. And because the data structure is wrong, you’re forced to use algorithms
that look like they come from an H.P. Lovecraft story.
Never having seen anything better, most Stata users seem to be resigned to
doing things like creating an entire column to store a single number and
writing impenetrable loops for simple tasks. Or they use sensible tools to
create their datasets (increasingly Python, but also even something like
Excel) and then use Stata just for the analysis.
The latter is my approach when I can’t avoid Stata entirely. But I’m really
looking forward to the day when I can avoid the fundamentally flawed design of
A few weeks ago, Franklin Foer wrote an article at The New Republic arguing
that Amazon is now a monopoly and therefore should be broken up. The
difference between Amazon and what we used to think of as monopolies, he says,
is that Amazon squeezes its producers, not its customers, and consumers are
complicit in the squeezing, which is just kind of assumed to be a bad thing.
Foer didn’t offer very specific recommendations, but he did point to, say AT&T
which was broken up using antitrust law in the 1970s as a good example.
“That’s silly”, I thought when I first read the piece, and I didn’t expect to
hear much more about it.
Today, however, Paul Krugman followed up with an op-ed that correctly identified
Amazon’s relationship to its producers as a monopsony, not a
monopoly, and argued that it is totally not ok, guys.
Krugman’s argument zeros in on Amazon’s fight with publisher Hachette. Hachette
won’t agree to the revenue sharing that Amazon wants, so Amazon has
disadvantaged their books.
Like Foer, Krugman calls to mind the old progressive “victories” like the
breakup of Standard Oil, saying, “The robber baron era ended when we as a
nation decided that some business tactics were out of line. And the question is
whether we want to go back on that decision.”
I think that line explains why suddenly we’re all supposed to be up in arms
about Amazon. It’s certainly not out of deep concern for book publishers.
Everyone hates book publishers, who squeeze authors as much as Amazon squeezes
them (and, interestingly, more than Amazon squeezes authors, at least at
In fact, in a sane hour, Krugman et al. would probably have no
trouble agreeing that what we’re really seeing here is publishers losing value
because what they do is not nearly as valuable when you don’t need to
physically print all your books. Certainly they would agree that, if the market
were well and truly competitive, none of the publishers would be making money
anyway because profits in a competitive market go to zero.
But Amazon is a BIG BUSINESS with MARKET POWER, and BIG BUSINESSES with
MARKET POWER are bad and exploitative in the progressive view of the world. The
breakup of Standard Oil is a part of the progressive identity the same way
that, say, the Reagan tax cuts are part of the conservative identity.
If Amazon isn’t actually hurting real people, then maybe BIG BUSINESSES with
MARKET POWER aren’t always bad. Maybe the breakup of Standard Oil wasn’t all
that huge a victory for real people after all. So it’s important to the
progressive view of the world that Amazon be perceived as hurting
Now, there’s nothing wrong with having general rules for policy, like “monopoly
is bad, let’s avoid that” or “let’s not try anything for the first time at the
national level.” They’re especially good when they’ve been learned over time.
But the hyper-dynamic technology-driven economy, where it’s has been harder and
harder to preserve market power, has presented a powerful challenge to these
old progressive beliefs, and those of us not wed to them should demand that
they prove themselves again.
This is how real economists release a
paper. Everything in the open,
downloadable for your convenience.
On Thursday, the Mercatus Center celebrated the 40th anniversary of the Nobel
Prize awarded to F. A. Hayek.. The keynote speaker was Israel Kirzner,
who Thomson Reuters predicts may this year’s Nobel Prize in economics. The
question he posed for the talk was what role the Prize played in reviving the
Austrian school of economics, but the more interesting aspect of his talk was
the way he described what it means to be an Austrian.
The central thesis Kirzner proposed was that, following the socialist
calculation debate, Mises and Hayek (the leaders of the Austrian school) made
important original refinements to the theory of how markets work, but that
these contributions were only recognized after Hayek’s Nobel Prize—awarded
for earlier and quite separate work—gave younger economics newfound reason to
examine Hayek’s entire career.
The socialist calculation debate, for those not versed in economic history, was
a high-profile argument in the 1920s and 1930s about whether
socialism—defined as state ownership of the means of production—could
Mises and Hayek argued that when the state owned the whole structure of
production from raw materials to consumer goods, there would be no way to know
which uses for goods—say, iron—were most valued. In a private property
system, this task is accomplished by prices, but prices don’t exists when the
state simply distributes resources to other parts of the state.
A decade and half after the first formulation of that argument, socialist
economist Oskar Lange came up with what would be considered the final answer.
Lange argued that you could just simulate free floating prices by assigning
them randomly and then adjusting based on how quickly different resources are
Lange’s answer, with some revisions, won the debate as far as the economics
profession was concerned. For this and a few other reasons, the Austrians lost
their reputations in the economics field, and fell into obscurity until the
Hayek won the Nobel.
Kirzner argued that, during that interlude, Mises and Hayek made important
refinements to and elaborations on the theory of market processes that
demonstrated why Lange’s answer—and the direction of the economics profession
as a whole—missed the point. He called the key idea of this period
Lange’s problem was that, like the now-mainstream of economics, he relied on
the assumption that every part of the economy was at equilibrium—that supply
and demand was perfectly balanced at every point in the economy, and that this
state of affairs simply needed to be replicated within socialism. This
assumption entails others—perfect competition in every market, for example,
and perfect knowledge of all individuals in the economy. Economists know that
these assumptions are not true, but they will generally say that they are close
enough to true as to not make a difference.
The problem, from the Austrian perspective, is that there are no new products,
no new ideas, no opportunities for improvement anywhere. There are a fixed,
closed set of choices—and all by initial assumption! The Austrian insight was
that the set of choices in a market system is constantly in flux; competition and
entrepreneurship allowed for the discovery of entirely new possibilities. And
these possibilities are beyond the reach of normal research and development,
because in many cases no-one knows to look for them. They are unknown unknowns.
In Lange’s model and all equilibrium model these possibilities disappear, and
with them goes the tether to the real economy. That’s not to say that such
models have no place—the mistake in mainstream economics was to give them the
only place, and set aside all consideration of the process of markets.
It is this belief, according to Kirzner, that makes one an Austrian. This bar
is far more ecumenical than some might like. You do not have to believe in
Austrian business cycle theory, you don’t have to throw out GDP with the
bathwater, and you don’t have to worry about radical a priori reasoning.
Using this criterion, Kirzner was happy to label Julian Simon an Austrian
even though he was also firmly in the Chicago school. Similarly, he absolved
Russ Roberts, who confessed to finding equilibrium models helpful. The point,
said Kirzner, is to remember that your models are tools and not reality, and
that there are more things in heaven and earth than are dreamt of in your
graduate economics seminar.
By this bar I can consider myself a Austrian, and even remember the sentence
that turned me into one. It’s in Arnold Kling and Nick Shultz’s
book, From Poverty to Prosperity. Kling and Shultz describe mainstream
economics by saying that they can explain why it makes more sense to have your
shirts ironed by a laundry than to do it yourself—specialization, trade, and
so on. An open-ended economist says that all of that’s true. “But have you
heard of permanent press?” Innovation completely changed the terms of the
question, and I don’t have to iron my shirts at all.
I don’t know why, but when I read that example, it clicked. The world expands
in unexpected ways, and our analyses are only related to the real world when
they account for that.
Kirzner says that belief makes me an Austrian, and that’s an Austrianism I’m
quite happy to be a part of.
This Thursday, Scotland will hold a referendum on independence. The
conventional wisdom holds that the vote will be close, but that the
“No” vote will carry the day. While the UK government does have final say on
constitutional matters (and thus on independence), everyone seems to assume
that it will honor the results of the referendum.
This entire situation highlights the absurdity of making important decisions
through majority-rules direct democracy. Consider the possible outcomes. If the
vote is 52 percent “Yes” to 48 percent “No”, then 48 percent of the populace,
some two and a half million souls, will find their links to what
they consider their home country severed. They will be forced, against their
will, to either physically leave their homes to remain within the UK or try to
mentally and emotionally locate themselves in a new political unit that they do
not want or recognize as their own.
If, on the other hand, the vote is 52 percent “No” to 48 percent “Yes”, then
you’ve got a similarly massive portion of the nation who actively wants not to
be part of it—and not at some future point, but as soon as possible. No
nation can tolerate that state of affairs for long with tranquility unless we
assume that the no voters are just breezy and faddish. I’m not willing to
Surely, for such an important question, fifty-plus-one doesn’t cut it. Blindly
following the referendum when public opinion is divided so evenly
can only lead to a sense of disaffected displacement for the losing side.
Whatever the economic effects, the social and psychological effects
of feeling disconnected from your home country matter. A shared culture, a
common historical heritage, a sense of community with your fellow
citizens—these are all things that my fellow libertarians like to wave away
as “tribalism,” but that are in fact necessary parts of forming a human
As a matter of practicality the best result on Thursday is a “No” vote, not
because I’m convinced the Scotland should remain part of the UK, but because
it’s a push. The Scots can gain independence in the future much more easily
than they can regain union. But that extra time only matters if the Scots use
it to build a real consensus. Perhaps that means that the UK government can do
a better job identifying and satisfying the concerns of those who favor
independence. Maybe it means that the pro-Independence side can do a better job
persuading their fellow Scots that an independent Scotland is not only
attainable and advisable, but a place that will still be their home.
But that consensus has to be established before either side can claim a
legitimate victory. Thursday’s referendum can only reveal the contours of the
problem, not the wisest solution. In fact, it has probably already told us all