<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Oliver Sherouse on technology, data, government, and culture</title>
  <updated>2026-11-18T04:34:58Z</updated>
  <author>
    <name>Oliver Sherouse</name>
    <email>oliver@oliversherouse.com</email>
  </author>
  <id>https://www.oliversherouse.com/feed.xml</id>
  <link rel="self" href="https://www.oliversherouse.com/feed.xml" />
  <generator>Unsitely</generator>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/cost-of-thriving.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/cost-of-thriving.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>The Cost of Thriving Index, Fixed</title>
    <updated>2020-03-01T00:00:00Z</updated>
    <content type="html">&lt;section id=&#34;background&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Background&lt;/h2&gt;
&lt;p&gt;Oren Cass thinks the middle class has a problem. Cass, a scholar at
the Manhattan Institute and head of the new organization American
Compass, argues that middle class families feel like they’re worse and
worse off, and furthermore, he argues that they’re right. The solution
he provides is for American conservatism to throw off the libertarian,
market fundamentalist who have controlled them like brain slugs since
the ’80s, and instead go for some good old fashioned industrial
planning.&lt;/p&gt;
&lt;p&gt;There’s a lot in there. But we’ll blow past most of it for now to
focus on the empirical argument that Cass has provided for his point
that the middle class is worse off than it used to be. He doesn’t think
that the official statistics (which show the opposite) are very good
because of the way they handle quality adjustments. Inflation measures
try to capture the fact that goods get more expensive over time both
because of inflation and because quality improves. The next iPhone will
be more expensive than the last one, but also better, so a lot of that
price increase doesn’t count as inflation.&lt;a href=&#34;#fn1&#34;
class=&#34;footnote-ref&#34; id=&#34;fnref1&#34; role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt; But
aha! says Cass. what happens if they stop selling the old iPhone? Sure,
the new one’s better, but it’s still more expensive to get a smartphone,
even if you don’t think the improved quality justifies the higher
price.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;the-cost-of-thriving-index&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;The Cost of Thriving Index&lt;/h2&gt;
&lt;p&gt;So to his credit, Cass has put forward an alternative. He picked four
categories that he argues are essential to feeling like you’ve got a
solid, middle class life. Those categories are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Housing&lt;/strong&gt;, as measured by the Department of
Housing and Urban Development’s estimate of a fair market rent for a
3-bedroom house in Raleigh, NC&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Health insurance&lt;/strong&gt;, as measured by the Kaiser
Family Foundation’s estimated average premium for employer-sponsored
health insurance for a family&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Transportation&lt;/strong&gt;, as measured by the Bureau of
Transportation Statistics’s estimate of the total cost of driving a car
15,000 miles per year&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Education&lt;/strong&gt;, as measured by the Department of
Education’s estimate of the cost of one semester of public
college&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This basket he compares to the last component,
&lt;strong&gt;Income&lt;/strong&gt;, as measured by the median wages for male
workers for one year. Why only males? Because, he reasons, it used to be
perfectly normal to be able to afford a middle-class life on one income,
so why shouldn’t it be so now?&lt;/p&gt;
&lt;p&gt;The combined price of those four expenses compared to the income
measure is what Cass calls the Cost of Thriving Index, which he lays out
in a report called &lt;a
href=&#34;https://www.manhattan-institute.org/reevaluating-prosperity-of-american-family&#34;&gt;The
Cost of Thriving&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And by that measure, the American middle class is, indeed, getting
screwed over pretty good. Here’s the chart of the number of weeks of
work needed to fund the four categories of major expenses for each year
from 1985-2018:&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;/img/coti/weeks_original.jpeg&#34;
title=&#34;It takes more than 52 weeks of work to pay for a middle class life on a middle class income, if this chart is right (which it isn&amp;#39;t).&#34;
alt=&#34;Cass Original Chart, Weeks of Work to Cover Major Expenses&#34; /&gt; &lt;/p&gt;
&lt;p&gt;Oh, no! It takes more than 52 weeks of work to afford a middle class
life on a middle class income! There’s not that many weeks in a year!
That sounds pretty bad!&lt;/p&gt;
&lt;p&gt;The chart that’s gone viral is this next one, showing the expenses
broken out vs. income over time.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;/img/coti/expenses_original.jpeg&#34;
title=&#34;Everything is so expensive if this chart is right (which it isn&amp;#39;t).&#34;
alt=&#34;Cass Original Chart, Income vs Major Expenses&#34; /&gt;.&lt;/p&gt;
&lt;p&gt;Pretty dramatic stuff. Wages have gone up over time, sure, but
they’ve just been dwarfed by the increased price of the major expenses.
Heck, after all that, there’s not even money to pay for food!&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;the-problem&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;The Problem&lt;/h2&gt;
&lt;p&gt;Again, there’s a lot in there. Almost every bit of it is
controversial on a theoretical level. See, for example, &lt;a
href=&#34;https://twitter.com/swinshi/status/1233034553796763654&#34;&gt;this&lt;/a&gt;,
&lt;a
href=&#34;https://www.bloomberg.com/opinion/articles/2020-02-27/men-can-still-support-their-families-on-one-income&#34;&gt;this&lt;/a&gt;,
and &lt;a
href=&#34;https://www.aei.org/carpe-diem/an-alternative-analysis-to-oren-casss-cost-of-not-thriving-index-using-bls-data/&#34;&gt;this&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;What I want to focus on is that health care component. You can see
from the second chart that it’s the biggest single driver of rising
costs. That’s not too surprising because we all know that health care
costs have been going up.&lt;/p&gt;
&lt;p&gt;But there’s a problem. As most folks with jobs are aware, an employee
rarely pays the whole cost of their health insurance; their employer
pays some, too. But Cass is using the total cost as the price of health
care. So in essence, his &lt;q&gt;cost&lt;/q&gt; of thriving includes costs that a
wage-earner &lt;em&gt;does not pay&lt;/em&gt;. &lt;a href=&#34;#fn2&#34; class=&#34;footnote-ref&#34;
id=&#34;fnref2&#34; role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;This mistake matters because it goes directly to what the COTI is
supposed to measure. We’re supposed to be looking at how long someone
has to work to pay for these four major categories. If we’re including
things that the wage-earner doesn’t pay for himself, then we’re going to
get the wrong picture—at least if the difference is large enough to
matter.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;what-the-coti-says-when-done-correctly&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;What the COTI Says When Done Correctly&lt;/h2&gt;
&lt;p&gt;I’m not the first person to point this out, but I was curious just
how big a problem this was, so I decided to reconstruct the index
myself. First, I included the employer-paid health insurance premiums to
make sure my number matched Cass’s. Then I recalculated it using using
only the employee-paid portion, which I got from the same source. It’s
worth pointing out that Cass gave clear enough indications of his
sources and methods that I was able to reproduce most of his numbers
exactly, and the ones that weren’t exact are close enough as makes no
difference.&lt;a href=&#34;#fn3&#34; class=&#34;footnote-ref&#34; id=&#34;fnref3&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;3&lt;/sup&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;So here’s what the weeks-of-work chart looks like with both with the
original index that incorrectly includes employer-paid premiums, and
with the recalculated index that only includes what employees actually
pay:&lt;/p&gt;
&lt;figure&gt;
&lt;img src=&#34;/img/coti/weeks.png&#34;
title=&#34;Hey, where&amp;#39;d all the rising costs go?&#34;
alt=&#34;Weeks of Work to Pay for Major Expenses, Original and Recalculated Index&#34; /&gt;
&lt;figcaption aria-hidden=&#34;true&#34;&gt;Weeks of Work to Pay for Major Expenses,
Original and Recalculated Index&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Oh. Instead of taking 53 weeks of work to pay for a middle class
life, it’s actually 39. Not only is that a good bit less than a year,
but it’s lower than where Cass’s original index had us in 1996. It’s
unclear whether Cass thinks 1996 were good times and the trend is still
up, but we see now that the four categories are less than
all-consuming.&lt;/p&gt;
&lt;p&gt;And what about the raw expenses chart? Here’s my recalculation of the
original, so you can see that even with a slightly different health
series, it’s the same story:&lt;/p&gt;
&lt;figure&gt;
&lt;img src=&#34;/img/coti/expenses_total_insurance.png&#34; title=&#34;Still Wrong.&#34;
alt=&#34;Income vs Major Expenses, Recalculation of Original&#34; /&gt;
&lt;figcaption aria-hidden=&#34;true&#34;&gt;Income vs Major Expenses, Recalculation
of Original&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;And here’s what it looks like when you count health insurance
properly:&lt;/p&gt;
&lt;figure&gt;
&lt;img src=&#34;/img/coti/expenses_employee_insurance.png&#34;
title=&#34;All that disastrous budget space! I blame libertarians.&#34;
alt=&#34;Income vs Major Expenses, Recalculation of Original&#34; /&gt;
&lt;figcaption aria-hidden=&#34;true&#34;&gt;Income vs Major Expenses, Recalculation
of Original&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;&lt;em&gt;Well, then.&lt;/em&gt; Looks like the middle class still has a good bit
of headroom, even after paying the price of thriving. And let’s not
forget: the things that Cass includes in his major expenses are the ones
where costs have gone up the most over time. The rest of that income can
go to things like food, leisure, technology, and travel that have gotten
cheaper, have improved in quality or convenience, and where a far
greater variety is available than used to be. Not only can a single
wage-earner still cover Cass’s expenses like they could in 1985, they
can go beyond in many ways that were unimaginable back then.&lt;/p&gt;
&lt;p&gt;A lot of folks have raised good criticisms of the Cost of Living
Index. I think it’s wrong to cherry-pick the fastest-growing categories
of expenditures. I think it’s wrong to ignore changes in quality over
time. I think it’s wrong to ignore the unpaid work that women used to do
more of as a group than they do now. But even if you ignore all of that,
what the Cost of Thriving Index shows, at least when calculated
properly, is that the middle class is doing just fine, thank you very
much. Would it be good if the price of health care insurance and college
were growing more slowly? Sure. Of course it would. But are they eating
up the middle class? No. No they are not.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;In theory, at least. I don’t have time for an Apple rant
in the middle of this chart rant.&lt;a href=&#34;#fnref1&#34; class=&#34;footnote-back&#34;
role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn2&#34;&gt;&lt;p&gt;Economically, you could argue that the employee bears
the cost of the employer share of insurance payments. In fact, I would
argue that. But if you want to look at it that way, you’d need to add
that to wages as income to make the index make sense. It wouldn’t really
make a difference to my argument, so I kept it this way for
simplicity.&lt;a href=&#34;#fnref2&#34; class=&#34;footnote-back&#34;
role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn3&#34;&gt;&lt;p&gt;All my work, including my source data is available on
GitHub. The repository is &lt;a
href=&#34;https://github.com/OliverSherouse/blog-coti-charts&#34;&gt;here&lt;/a&gt;, the
notebook with the calculations is &lt;a
href=&#34;https://github.com/OliverSherouse/blog-coti-charts/blob/master/cass_chart.ipynb&#34;&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Cass uses a Kaiser Family Foundation estimate for health insurance
costs; I looked at Kaiser and it pointed to the Agency for Healthcare
Research and Quality. The AHRQ numbers do not exactly match the one Cass
reports (they’re off by a few hundred dollars at both ends) and cover
more years than he says Kaiser does. The differences are small enough
that you can’t see them on the graphs except for the first few years,
and I’m confident that whatever series he’s using and the one I used are
very similar, especially in recent years.&lt;a href=&#34;#fnref3&#34;
class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/notebooks_arent_papers.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/notebooks_arent_papers.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>Jupyter Notebooks Can't Fix the Scientific Paper</title>
    <updated>2018-04-17T00:00:00Z</updated>
    <content type="html">&lt;p&gt;&lt;a
href=&#34;https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/&#34;&gt;This
piece&lt;/a&gt; in the Atlantic has been making the rounds, suggesting that
Jupyter notebooks are &lt;q&gt;what comes next&lt;/q&gt; as the scientific paper
goes the way of legal vellum and religious papyrus. The argument runs
something like this: most of the actual work of science is
computational. Notebook environments give an excellent environment for
doing replicable computation while also interspersing explanatory text,
and Jupyter notebooks are the best ones around. This clears away the
need for much of the gobbledygook for which scientific papers are so
widely criticized, and puts the work front and center.&lt;/p&gt;
&lt;p&gt;And some of that is right. Jupyter is amazing; my team and I are
actually working right now to build a Jupyter-based research platform
for our organization. And well-documented Jupyter notebooks are
certainly a step up over incomprehensible do files that we see with many
papers, especially when they’re augmented with simple controls to modify
parameters, for example.&lt;/p&gt;
&lt;p&gt;But the problem with this line of thinking is that it gets the basic
point of a scientific paper wrong. The computation is &lt;em&gt;not&lt;/em&gt; the
most important part of a paper. The most important part is the
writing.&lt;/p&gt;
&lt;p&gt;A paper should do three things:&lt;/p&gt;
&lt;ol type=&#34;1&#34;&gt;
&lt;li&gt;Orient the reader to the current state of knowledge&lt;/li&gt;
&lt;li&gt;Offer relevant evidence&lt;/li&gt;
&lt;li&gt;Integrate the evidence to produce a new current state of
knowledge&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Computations happen, if at all, in number 2. Jupyter’s great for
presenting that. Jupyter doesn’t help at all for numbers 1 and 3, and
they are equally as important, if not more so. What helps them—and
really the only thing that helps them—is clear thinking and writing.&lt;/p&gt;
&lt;p&gt;Now, obviously you can &lt;em&gt;have&lt;/em&gt; text in a Jupyter notebook. You
have Markdown and HTML, and that’s a lot! But Jupyter is not a writing
environment. There’s no formatting help if you forget the Markdown
syntax. You can’t track changes or get comments easily from your peers
or coauthors. You can’t manage your bibliography. And it’s just not a
pleasant interface for composing text.&lt;/p&gt;
&lt;p&gt;For that matter, it’s also not great for &lt;em&gt;reading&lt;/em&gt; text. The
lines are too long, for one thing. You can’t easily jump to the
footnotes and back, like you can in a well-formatted PDF. There aren’t
columns, and can’t be.&lt;/p&gt;
&lt;p&gt;That’s not really a criticism; that’s not what Jupyter is
&lt;em&gt;for&lt;/em&gt;. It has text support for writing short bits of text that
are secondary to the computational work. But let’s not pretend that
Jupyter will ever be an excellent replacement for &lt;del&gt;Word&lt;/del&gt; Vim on
the editing side, or a &lt;span class=&#34;math inline&#34;&gt;\(\LaTeX\)&lt;/span&gt; PDF
on the reading side.&lt;/p&gt;
&lt;p&gt;So, if not notebooks, is there something that can replace the crappy
scientific paper? At the risk of being dull, how about: the
&lt;em&gt;good&lt;/em&gt; scientific paper? Papers where introductions and
conclusions aren’t unnecessarily ponderous, where the evidence is
delivered with minimal gobbledygook and presented in a replicable way&lt;a
href=&#34;#fn1&#34; class=&#34;footnote-ref&#34; id=&#34;fnref1&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;, and implications are explored with
straightforwardness and intellectual humility?&lt;/p&gt;
&lt;p&gt;Of course, that’s what we’re supposed to be doing now. John H.
Cochrane has a &lt;a
href=&#34;http://tertilt.vwl.uni-mannheim.de/phd/phd_paper_writing.pdf&#34;&gt;wonderful
write-up&lt;/a&gt; on how to do it in economics&lt;a href=&#34;#fn2&#34;
class=&#34;footnote-ref&#34; id=&#34;fnref2&#34; role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;.
And there are good examples out there. Watson and Crick’s paper
introducing the double-helix structure of DNA? It’s &lt;a
href=&#34;http://www.sns.ias.edu/~tlusty/courses/landmark/WatsonCrick1953.pdf&#34;&gt;one
page long&lt;/a&gt;. Including a picture.&lt;/p&gt;
&lt;p&gt;Now, if we don’t do that in the future, it’ll be for the same reason
we don’t do it now. Journals and peers don’t make us. Publish-or-perish
puts more emphasis on producing papers than on producing good papers.
Papers are written to be published, not read.&lt;/p&gt;
&lt;p&gt;Fixing that technology—the institutional, meta-technology of
publishing, would fix the scientific paper. But I don’t think Jupyter
Notebooks, glorious powerful magic though they are, can do that.&lt;/p&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;Jupyter can help with this!&lt;a href=&#34;#fnref1&#34;
class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn2&#34;&gt;&lt;p&gt;&lt;em&gt;NOT SCIENCE LOL!&lt;/em&gt; I know, I know.&lt;a
href=&#34;#fnref2&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/peterson.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/peterson.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>Jordan Peterson and the Challenge of Statistics</title>
    <updated>2018-01-29T00:00:00Z</updated>
    <content type="html">&lt;p&gt;&lt;a href=&#34;https://youtu.be/aMcjxSThD54&#34;&gt;This&lt;/a&gt; contentious interview
of Jordan Peterson, a University of Toronto Psychology Professor, by
Cathy Newman of the UK’s Channel 4, has garnered a huge amount of
attention. While the interview was nominally to promote Peterson’s
upcoming book, Newman clearly believed that she was going to be able to
nail him as an ignorant bigot. Unfortunately for her, the general
consensus is that Peterson was able to avoid that outcome, and make her
look pretty silly in the process.&lt;/p&gt;
&lt;p&gt;Much of the conversation (&lt;a
href=&#34;https://www.theatlantic.com/politics/archive/2018/01/putting-monsterpaint-onjordan-peterson/550859/&#34;&gt;see
here, for example&lt;/a&gt; has focused on Newman’s interrogatory tactics and
how Peterson chose to respond to them, but I there are lessons to be
learned here about communicating with statistics. The first time I
watched the video my initial reaction was that Peterson clearly
understood the statistics he wanted to use to support his points, and
the interviewer did not. Those statistics are not all that
controversial, even among those who tend to disagree with Peterson’s
conclusions, but throughout the interview Newman consistently jumps from
his rather modest claims to extreme (and sometimes bizarre) conclusions
that she assigns to him.&lt;/p&gt;
&lt;p&gt;Even if, as some suggest, Newman’s ignorance here was deliberate, her
responses reflect the kind of intuitive interpretation of statistics
that I’ve seen many times. Statistics are not intuitive. They are
tricky. If you need to communicate with them to a non-statistician—and
you will—it’s important to help people understand what the statistics
you’re using do and do not imply.&lt;/p&gt;
&lt;p&gt;Let’s look at two sections where, with the help of hindsight, we
might be able to improve on Peterson’s presentation. First, let’s
examine the initial conversation about the pay gap.&lt;/p&gt;
&lt;iframe src=&#34;https://www.youtube.com/embed/aMcjxSThD54?start=317&amp;amp;end=426&#34; width=&#34;560&#34; height=&#34;315&#34; allowfullscreen&gt;
&lt;/iframe&gt;
&lt;p&gt;Peterson makes two mistakes here. First, in an uncharacteristically
imprecise use of language, he says that the pay gap &lt;q&gt;does not
exist,&lt;/q&gt; when that’s not what he means. Over a minute later, he
clarifies that he actually means &lt;q&gt;does not exist solely due to
gender&lt;/q&gt;, but by that point a minute of airtime has gone to waste.&lt;/p&gt;
&lt;p&gt;The more common mistake Peterson makes in the pay gap discussion,
though, is focusing on the &lt;em&gt;method&lt;/em&gt;. He starts talking about
multi-variate analysis, and the interviewer—and most home viewers—have
no idea what it means.&lt;a href=&#34;#fn1&#34; class=&#34;footnote-ref&#34; id=&#34;fnref1&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;. When challenged by Newman on why he
keeps talking about it, he enters into a mostly fine description of why
controls are important in regression (although he does make it sound
like he’s doing a series of one-to-one comparisons rather than a single
composite analysis). He’s not wrong, but he’s also not making his point;
the only thing that this part of the conversation does for him is make
it sound like he knows what he’s talking about, but the lay audience
won’t get anything out of it.&lt;/p&gt;
&lt;p&gt;Everyone who communicates about regression-type analysis needs to
have a stock phrase to describe what’s important about it and move on,
and I was a bit surprised Peterson didn’t have one ready. Here’s how I
might have phrased the point he was making in a way that could keep the
conversation focused on the point Peterson was driving at:&lt;/p&gt;
&lt;p&gt;&lt;q&gt;It does seem that way, but what repeated studies have reliably
found is that when you account for an person’s age and their personality
and their aptitude and their interest, then the difference their gender
makes to their salary is very small. So a man and a woman who are
similar in other ways should expect to make about the same amount of
money. So we know that the pay gap is not mostly due to gender
bias.&lt;/q&gt;&lt;/p&gt;
&lt;p&gt;I timed myself and that took 22 seconds to say, without getting the
methodology behind the point in the way of the point itself. Peterson
and Newman six times that on an unfruitful conversation about how
statistics work.&lt;/p&gt;
&lt;p&gt;The second difficulty that stood out to me about the interview was
the way that Peterson and Newman talked past each other on the subject
of population characteristics and individual characteristics.&lt;/p&gt;
&lt;p&gt;The best example picks up right where the last stopped:&lt;/p&gt;
&lt;iframe src=&#34;https://www.youtube.com/embed/aMcjxSThD54?start=426&amp;amp;end=459&#34; width=&#34;560&#34; height=&#34;315&#34; allowfullscreen&gt;
&lt;/iframe&gt;
&lt;p&gt;Again, Peterson makes an unforced error when he says &lt;q&gt;Women are
less agreeable than men,&lt;/q&gt; and again, the problem isn’t that he’s
wrong exactly but rather that what he’s saying will be taken differently
by the viewers than he means it. The natural implication of &lt;q&gt;woman are
less agreeable than men&lt;/q&gt; is that &lt;em&gt;all&lt;/em&gt; women are less
agreeable than &lt;em&gt;all&lt;/em&gt; men.&lt;/p&gt;
&lt;p&gt;This confusion is nicely demonstrated by the exchange that follows.
Newman accuses Peterson of &lt;q&gt;a vast generalization,&lt;/q&gt; by which she
means that he’s making a statement about all individual women. He says
that &lt;q&gt;it’s not a generalization,&lt;/q&gt; and what he means is that it’s a
statement about the distribution of that trait among the population of
all women. The disconnect is that the same words mean something slightly
different to the two because one is thinking statistically and the other
isn’t. And the onus has to be on Peterson to make his point clear.&lt;/p&gt;
&lt;p&gt;At first I thought the best phrase to do that would be
&lt;q&gt;agreeableness is more prevalent among women than men,&lt;/q&gt; but I don’t
think that’s quite right, because agreeableness is a continuous
variable. You could opt for something less precise like &lt;q&gt;more women
are highly agreeable than men,&lt;/q&gt; but that doesn’t quite fit right
either. I think the best solution here is a small modification: &lt;q&gt;Women
tend to be more agreeable than men.&lt;/q&gt; People understand the
non-universality of &lt;em&gt;tend&lt;/em&gt;, and that avoids the confusion.&lt;/p&gt;
&lt;p&gt;This one isn’t so much a question of wasting time as of avoiding
confusion. To their credit, Newman and Peterson reach consensus of what
they mean fairly quickly with the final exchange in that clip. They just
both get a bit annoyed doing it.&lt;/p&gt;
&lt;p&gt;Peterson warmed up as the interview went along, and I think he
handled a second go at much the same argument much better:&lt;/p&gt;
&lt;iframe src=&#34;https://www.youtube.com/embed/aMcjxSThD54?start=889&amp;amp;end=983&#34; width=&#34;560&#34; height=&#34;315&#34; allowfullscreen&gt;
&lt;/iframe&gt;
&lt;p&gt;In that exchange, Newman fires off a number of conclusions that she
claims are implied by Peterson’s arguments. All of them are predicated
on the idea that his population statistics determine what will happen
with &lt;em&gt;every&lt;/em&gt; woman. Instead of talking about how statistics work,
he goes to the concrete example of Newman herself. That allows him to
make his point without any confusion: she’s been successful precisely
because she’s pursued her career in the way that he says matters more
than gender. There’s no way to confuse &lt;q&gt;you, as a woman, are
successful because you have battled for it&lt;/q&gt; with &lt;q&gt;the need to
battle for success means women will never succeed.&lt;/q&gt; Sometimes when
you’re talking about statistical truths, the best way to do it is to
avoid discussing them statistically at all.&lt;/p&gt;
&lt;p&gt;Now, the point of this isn’t that Peterson’s dumb and I’m smart; I’ve
had time to consider and edit. The point is that communicating
statistics is incredibly difficult, even if you understand them well
yourself. It’s a separate skill, and takes practice. When you screw it
up, it’s easy to blame the ignorance of our listeners, but that’s too
easy; it’s far better in the long run to focus how you can be better at
communicating statistical facts. Then people might be more interested in
what you have to say.&lt;/p&gt;
&lt;p&gt;Some stray other thoughts about the interview:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p&gt;In general, the interview has been scored as a hands-down win for
Peterson. If it had ended after the first ten minutes or so, I’m not
sure that would have been the case. I think Newman and Channel 4 deserve
a bit of credit for resisting the urge to edit it down to that.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;That said, the utter &lt;a
href=&#34;https://youtu.be/aMcjxSThD54?t=22m11s&#34;&gt;blue screen&lt;/a&gt; that
happens to Newman is one of the most stunning things I’ve ever seen on
television.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;At one point Newman argues against the phrase &lt;q&gt;the typical
woman&lt;/q&gt; because &lt;q&gt;all women are different, to which Peterson replies
that&lt;/q&gt;they’re different in some ways and the same in others.&#34; I found
her comment utterly bizarre; if all women are totally different, what
makes them women?&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;As if to prove this point, both Newman and the Channel 4
caption-writer who worked on &lt;a
href=&#34;https://twitter.com/Channel4News/status/953598649178652672&#34;&gt;this
clip&lt;/a&gt; thought he was saying &lt;q&gt;multi-varied analysis.&lt;/q&gt;&lt;a
href=&#34;#fnref1&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/vim_zero.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/vim_zero.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>Vim Zero</title>
    <updated>2017-08-21T00:00:00Z</updated>
    <content type="html">&lt;section id=&#34;introduction&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Introduction&lt;/h2&gt;
&lt;p&gt;I’ve been using Vim as my editor for over ten years&lt;a href=&#34;#fn1&#34;
class=&#34;footnote-ref&#34; id=&#34;fnref1&#34; role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;.
That’s a long time to build up settings and plugins, and generally get a
lot of cruft into my &lt;code&gt;vimrc&lt;/code&gt;. These days, when I go in there,
I don’t always remember what a particular setting or plugin does or why
I put it there, and I rarely look to see if there are updated versions
of anything.&lt;/p&gt;
&lt;p&gt;So I thought it would be both advantageous and fun to clear out my
settings and start again: go Vim Zero, and build up from there. And it
was fun! Here’s what I came up with.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;requirements&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Requirements&lt;/h2&gt;
&lt;p&gt;The first question was: what do I really want an editor to do? I use
Vim for writing and for coding. The former is nearly always in markdown,
generally Pandoc-flavored but sometimes a different one. I write code
most frequently in Python, and a decent amount in HTML, JavaScript, CSS
and SCSS, &lt;a
href=&#34;http://www.oliversherouse.com/2016/04/07/makefiles.html&#34;&gt;Make&lt;/a&gt;.
That means I need good support for multiple languages—including syntax
checkers and completion—and I need a writing environment that feels
comfortable.&lt;/p&gt;
&lt;p&gt;I also write both code and text on multiple computers. I use Linux
when I can, but use Windows at work and occasionally find myself on a
Mac. That means I need to be able to sync everything using git, and all
my plugins and settings have to work in the same way. I want my
experience in GVim on Windows to be as close as possible to my
experience in the terminal on my Arch box.&lt;/p&gt;
&lt;p&gt;Finally, I want to keep things as simple and elegant as possible. I
want this to still be Vim when I’m done, which means I don’t want a
bunch of functionality I’m not using or a bunch of nonsense I don’t need
on my screen&lt;a href=&#34;#fn2&#34; class=&#34;footnote-ref&#34; id=&#34;fnref2&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;. In general, I want to prefer
built-in functionality to plugins, and simple, tightly focused plugins
to wide-ranging and powerful ones. With that in mind, I started with the
settings that made vanilla vim as pleasant as possible.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;built-in-functionality&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Built-in Functionality&lt;/h2&gt;
&lt;p&gt;Well, that’s &lt;em&gt;almost&lt;/em&gt; true. I knew I was going to be using Tim
Pope’s &lt;a href=&#34;https://github.com/tpope/vim-sensible&#34;&gt;vim-sensible&lt;/a&gt;
plugin simply because it sets a goodly number of the things you see in
almost every vimrc, like being able to backspace over anything and
setting incremental search. We’ll get back to that later.&lt;/p&gt;
&lt;section id=&#34;general-functionality&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;General Functionality&lt;/h3&gt;
&lt;p&gt;First I set the &lt;code&gt;mapleader&lt;/code&gt; to comma so that it applies
for all my mappings. I set &lt;code&gt;hidden&lt;/code&gt; to allow for unsaved
background tabs, and &lt;code&gt;spell&lt;/code&gt; so that I don’t reveal my
horrible spelling to the world. Turns out you can set new splits to be
on the left, so I turn that on (we live in a world of widescreen
monitors, how is this not the default?), and I turn on persistent undo
files. All that looks like this:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot; Built-In Functionality
&amp;quot;&amp;quot; General
let mapleader = &amp;#39;,&amp;#39;

set hidden &amp;quot; Allow background buffers without saving
set spell spelllang=en_us
set splitright &amp;quot; Split to right by default&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;text-wrapping&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Text-Wrapping&lt;/h3&gt;
&lt;p&gt;In general, I want things wrapped at 79 characters—enough, in fact,
that it’s easier for me to turn it off when I don’t want it than turn it
on when I do. I also like having a highlighted column at 80 characters
as a visual guide. I always want hard wraps, so I turn off
soft-wrapping.&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Text Wrapping
set textwidth=79
set colorcolumn=80
set nowrap&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;search-and-substitutions&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Search and Substitutions&lt;/h3&gt;
&lt;p&gt;I find I want the &lt;code&gt;g&lt;/code&gt; flag in my &lt;code&gt;s/&lt;/code&gt; commands
far more often than I don’t, so I set it to be on by default. I use
highlight searches because that’s half the point, and use the handy
combination of &lt;code&gt;ignorecase&lt;/code&gt; and &lt;code&gt;smartcase&lt;/code&gt; to
ignore case when I type in lowercase, but not when I type in capital
letters. I also have my first mapping here: comma-space for clearing the
highlighted searches. It makes a nice slapping noise, which I quite
enjoy, as if to say &lt;q&gt;get that out of here.&lt;/q&gt;&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Search and Substitute
set gdefault &amp;quot; use global flag by default in s: commands
set hlsearch &amp;quot; highlight searches
set ignorecase 
set smartcase &amp;quot; don&amp;#39;t ignore capitals in searches
nnoremap &amp;lt;leader&amp;gt;&amp;lt;space&amp;gt; :nohls &amp;lt;enter&amp;gt;&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;tabs&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Tabs&lt;/h3&gt;
&lt;p&gt;Because I am not a horrible human being who hates joy and love and
light, I use four spaces instead of tabs whenever I can&lt;a href=&#34;#fn3&#34;
class=&#34;footnote-ref&#34; id=&#34;fnref3&#34; role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;3&lt;/sup&gt;&lt;/a&gt;.
The following combo will do that, and should be required by law.&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Tabs
set tabstop=4
set softtabstop=4
set shiftwidth=4
set expandtab&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;backup-swap-and-undo&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Backup, Swap, and Undo&lt;/h3&gt;
&lt;p&gt;The next section might be a little controversial. Backup files, swap
files, and undo files are great features of vim, but I hate having them
clutter up my actual work directories. This isn’t so bad on Linux, where
hidden files are simple things, but on Windows, which will
incomprehensibly ignore leading dots when doing file completion, it’s
awful. So, after I turn on undo files for persistent undo across
sessions, I set folders inside my vim folder to hold all of these (see
implementation notes at the end to see how I make empty folders with
with Git).&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Backup, Swap and Undo
set undofile &amp;quot; Persistent Undo
if has(&amp;quot;win32&amp;quot;)
    set directory=$HOME\vimfiles\swap,$TEMP
    set backupdir=$HOME\vimfiles\backup,$TEMP
    set undodir=$HOME\vimfiles\undo,$TEMP
else
    set directory=~/.vim/swap,/tmp
    set backupdir=~/.vim/backup,/tmp
    set undodir=~/.vim/undo,/tmp
endif&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;netrw&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;NetRW&lt;/h3&gt;
&lt;p&gt;Some folks won’t like this section either, because it’s about NetRW,
vim’s file explorer. It gets more hate than it deserves, but I find it
useful&lt;a href=&#34;#fn4&#34; class=&#34;footnote-ref&#34; id=&#34;fnref4&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;4&lt;/sup&gt;&lt;/a&gt;. I set it to have the detail view
with human-readable file sizes. The hiding behavior is a little odd, so
I just tell the explorer to hide dotfiles, and to set them as hidden by
default (this can be toggled with &lt;code&gt;a&lt;/code&gt;). Then I turn off the
banner. I also add a mapping to start the explorer; the exclamation
point means that if the current buffer has unsaved changes, the Explorer
will split vertically instead of horizontally.&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot;&amp;quot; NetRW
let g:netrw_liststyle = 1 &amp;quot; Detail View
let g:netrw_sizestyle = &amp;quot;H&amp;quot; &amp;quot; Human-readable file sizes
let g:netrw_list_hide = &amp;#39;\(^\|\s\s\)\zs\.\S\+&amp;#39; &amp;quot; hide dotfiles
let g:netrw_hide = 1 &amp;quot; hide dotfiles by default
let g:netrw_banner = 0 &amp;quot; Turn off banner
&amp;quot;&amp;quot;&amp;quot; Explore in vertical split
nnoremap &amp;lt;Leader&amp;gt;e :Explore! &amp;lt;enter&amp;gt;&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;general-mappings&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;General Mappings&lt;/h3&gt;
&lt;p&gt;To wrap-up the built-in functionality, I have my general mappings.
Mapping a semicolon to the colon in normal mode is surprisingly useful.
I use control-H and -L to cycle through my buffers, because that feels
like moving left and right to me. I use comma-q to quit a buffer and
comma-w to save. Finally, I use comma x to access the copy buffer, which
allows me to copy and paste between vim and other programs. That last is
the only mapping which I have set to work in all modes, and I use it all
the time.&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Mappings
nnoremap ; :
nnoremap &amp;lt;C-H&amp;gt; :bp &amp;lt;enter&amp;gt;
nnoremap &amp;lt;C-L&amp;gt; :bn &amp;lt;enter&amp;gt;
nnoremap &amp;lt;Leader&amp;gt;w :w &amp;lt;enter&amp;gt;
nnoremap &amp;lt;Leader&amp;gt;q :bd &amp;lt;enter&amp;gt;

noremap &amp;lt;Leader&amp;gt;x &amp;quot;+&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;python-version&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Python Version&lt;/h3&gt;
&lt;p&gt;I use Python 3 more or less exclusively. Many of the libraries I use
most are (finally) &lt;a href=&#34;http://www.python3statement.org/&#34;&gt;moving to
require it&lt;/a&gt;, and I like it better anyway. So I have this little
autocommand group to set my omnicompletion to Python 3:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Python Version
augroup python3
    au! BufEnter *.py setlocal omnifunc=python3complete#Complete
augroup END&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;/section&gt;
&lt;section id=&#34;plugins&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Plugins&lt;/h2&gt;
&lt;p&gt;Plugins are a wonderful part of the Vim infrastructure, and they’re
what let you really make the editor your own. That said, folks tend to
go overboard; I see vimrc files floating around with dozens of plugins,
and it’s just not necessary. When I started this project, I decided to
only add plugins I didn’t want to live without, and I think I’ve kept it
to a reasonable number. A fantastic resource has been &lt;a
href=&#34;http://vimawesome.com/&#34;&gt;Vim-Awesome&lt;/a&gt;, which makes it easy to
find plugins by functionality, and also see which are popular, which are
maintained, and so on. I knew about some of these, but others I didn’t,
and so the site was a huge help.&lt;/p&gt;
&lt;section id=&#34;plugin-manager&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Plugin Manager&lt;/h3&gt;
&lt;p&gt;Once upon a time, I installed plugins manually. Then I used my
package manager and a script called &lt;code&gt;vim-plugin-manager&lt;/code&gt;.
Then Tim Pope wrote &lt;a
href=&#34;https://github.com/tpope/vim-pathogen&#34;&gt;Pathogen&lt;/a&gt;, and like the
rest of the world, I switched to it immediately. Then &lt;a
href=&#34;https://github.com/VundleVim/Vundle.vim&#34;&gt;Vundle&lt;/a&gt; came along
with its Git-driven management, and I happily used that until I started
this project.&lt;/p&gt;
&lt;p&gt;When I went to see what was out there, I found that Vundle was still
a good option, but I was charmed by the simplicity of &lt;a
href=&#34;https://github.com/junegunn/vim-plug&#34;&gt;VimPlug&lt;/a&gt;, which didn’t
need any &lt;code&gt;rtp&lt;/code&gt; manipulation in my &lt;code&gt;.vimrc&lt;/code&gt; and
could do parallel installations and updates. I decided it was worth
making the switch.&lt;/p&gt;
&lt;p&gt;This breaks my rule a little bit about preferring built-in
functionality; Vim 8 does have a built-in plugin manager of sorts.
Unfortunately it would mean taking a step back: there’s no way to keep
your plugins updated and I just don’t want to go back to doing it
manually and fiddling with submodules in my vimfiles repository. So
VimPlug it is! The entirety of my plugin installation section looks like
this:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot; Plugins 

&amp;quot;&amp;quot; Installation with VimPlug
if has(&amp;quot;win32&amp;quot;)
    call plug#begin(&amp;#39;~/vimfiles/plugged&amp;#39;)
else
    call plug#begin(&amp;#39;~/.vim/plugged&amp;#39;)
endif

&amp;quot;&amp;quot;&amp;quot; Basics
Plug &amp;#39;tpope/vim-sensible&amp;#39;
Plug &amp;#39;sheerun/vim-polyglot&amp;#39;
Plug &amp;#39;flazz/vim-colorschemes&amp;#39;

&amp;quot;&amp;quot;&amp;quot; General Functionality
Plug &amp;#39;lifepillar/vim-mucomplete&amp;#39;
Plug &amp;#39;scrooloose/syntastic&amp;#39;
Plug &amp;#39;sirver/ultisnips&amp;#39;
Plug &amp;#39;honza/vim-snippets&amp;#39;
Plug &amp;#39;tpope/vim-commentary&amp;#39;
Plug &amp;#39;chiel92/vim-autoformat&amp;#39;

&amp;quot;&amp;quot;&amp;quot; Particular Functionality
Plug &amp;#39;junegunn/goyo.vim&amp;#39;
Plug &amp;#39;junegunn/limelight.vim&amp;#39;
Plug &amp;#39;vim-pandoc/vim-pandoc-syntax&amp;#39;
Plug &amp;#39;godlygeek/tabular&amp;#39;

call plug#end()&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I’ll walk through each of those in more detail and give the
configuration I have for each. As you can see above, I group my plugins
into three groups: basics, which include simple settings, filetype and
syntax, and color schemes; general functionality plugins, which add
features that are generally useful when editing code or writing text,
and particular functionality plugins, when are only useful in particular
situations.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;basics&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Basics&lt;/h3&gt;
&lt;p&gt;I’ve already mentioned Tim Pope’s excellent &lt;a
href=&#34;https://github.com/tpope/vim-sensible&#34;&gt;Vim-Sensible&lt;/a&gt; plugin,
which there’s really no downside to installing. It just gives you a lot
of sane defaults, and the code is perfectly readable if you want the
details.&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://github.com/sheerun/vim-polyglot&#34;&gt;Vim-Polyglot&lt;/a&gt;
and &lt;a
href=&#34;https://github.com/flazz/vim-colorschemes&#34;&gt;Vim-Colorschemes&lt;/a&gt;
are both omnibus packages. Essentially, they’re curated lists. At first
this seemed like overkill to me—why not just install the ones I want?
But then I remembered just how many times I’ve switched to a new
language and found that either vim didn’t have a filetype for it, or
that the user community had a few fixes for the built-in version of
indentation or something. Vim-Polyglot collects all of the best of
those, and that just saves me having to do it later. Similarly,
Vim-Colorschemes has at least one color scheme you will like, even if
you’re as picky as I am&lt;a href=&#34;#fn5&#34; class=&#34;footnote-ref&#34; id=&#34;fnref5&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;. I turn on gui-style colors for the
terminal and use the Darth style:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Colors
set termguicolors
colorscheme darth&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;general-functionality-plugins&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;General Functionality Plugins&lt;/h3&gt;
&lt;section id=&#34;autocompletion&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Autocompletion&lt;/h4&gt;
&lt;p&gt;Vim isn’t an IDE, and shouldn’t be, but autocompletion is really,
really nice. That said, I lived without it for a long time because I
didn’t like my options. &lt;a
href=&#34;https://valloric.github.io/YouCompleteMe&#34;&gt;YouCompleteMe&lt;/a&gt; is a
pain on Windows. So is &lt;a
href=&#34;https://github.com/Shougo/neocomplete.vim&#34;&gt;NeoComplete&lt;/a&gt;, and
while it’s predecessor &lt;a
href=&#34;https://github.com/Shougo/neocomplcache.vim&#34;&gt;NeoComplCache&lt;/a&gt; is
more easily cross-platform, it can be slow and frustrating and isn’t
updated any more. &lt;a
href=&#34;https://github.com/ajh17/VimCompletesMe&#34;&gt;VimCompletesMe&lt;/a&gt; isn’t
bad, but has a few quirks I don’t like and is entirely tab-driven, when
I would rather just have my options pop up for me.&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://github.com/lifepillar/vim-mucomplete&#34;&gt;MuComplete&lt;/a&gt;
gives me what I want. It does omnicompletion, file completion, snippet
completion (see &lt;a href=&#34;#snippets&#34;&gt;below&lt;/a&gt;), pops up as I type and
doesn’t get in my way. And it’s fast. Here’s the configuration to make
it work:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Autocompletion
set completeopt=menuone,noinsert,noselect
set shortmess+=c &amp;quot; Turn off completion messages

inoremap &amp;lt;expr&amp;gt; &amp;lt;c-e&amp;gt; mucomplete#popup_exit(&amp;quot;\&amp;lt;c-e&amp;gt;&amp;quot;)
inoremap &amp;lt;expr&amp;gt; &amp;lt;c-y&amp;gt; mucomplete#popup_exit(&amp;quot;\&amp;lt;c-y&amp;gt;&amp;quot;)
inoremap &amp;lt;expr&amp;gt;  &amp;lt;cr&amp;gt; mucomplete#popup_exit(&amp;quot;\&amp;lt;cr&amp;gt;&amp;quot;)

let g:mucomplete#enable_auto_at_startup = 1 &lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;snippets&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Snippets&lt;/h4&gt;
&lt;p&gt;I’ve gone back and forth on snippets for years, but for the moment
I’m pro. They save a lot of time writing HTML and encourage me to write
docstrings&lt;a href=&#34;#fn6&#34; class=&#34;footnote-ref&#34; id=&#34;fnref6&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;. Here, &lt;a
href=&#34;https://github.com/SirVer/ultisnips&#34;&gt;Ultisnips&lt;/a&gt; has been around
for a long time, and while &lt;a
href=&#34;https://github.com/garbas/vim-snipmate&#34;&gt;SnipMate&lt;/a&gt; also exists
as a venerable option, Ultisnips still feels like the gold standard. A
solid compilation of snippets is available with the &lt;a
href=&#34;https://github.com/honza/vim-snippets&#34;&gt;Vim-Snippets&lt;/a&gt; plugin.
You don’t need any configuration for either as far as I’m concerned, but
you can configure MuComplete to take advantage of Ultisnips with this
line:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;call add(g:mucomplete#chains[&amp;#39;default&amp;#39;], &amp;#39;ulti&amp;#39;)&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;commenting&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Commenting&lt;/h4&gt;
&lt;p&gt;I’ve used &lt;a
href=&#34;https://github.com/scrooloose/nerdcommenter&#34;&gt;NerdCommenter&lt;/a&gt; for
a long time, but &lt;a
href=&#34;https://github.com/tpope/vim-commentary&#34;&gt;Commentary&lt;/a&gt;, another
from Tim Pope, gives a minimalist yet powerful implementation. You use
&lt;code&gt;gc&lt;/code&gt; to to toggle comments, and that’s about it. That’s all I
want here.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;syntax-checking&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Syntax Checking&lt;/h4&gt;
&lt;p&gt;Everything I write is perfect the first time, obviously, but
sometimes I read other people’s code. &lt;a
href=&#34;https://github.com/vim-syntastic/syntastic&#34;&gt;Syntastic&lt;/a&gt; is a
truly clever plugin for running syntax checking. Rather than write its
own rules it uses external checkers, like &lt;code&gt;flake8&lt;/code&gt; and
&lt;code&gt;tidy&lt;/code&gt;, which is a very Unix way of approaching the problem.
Hard to beat.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;autoformatting&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Autoformatting&lt;/h4&gt;
&lt;p&gt;Of course, if I can get a program to &lt;em&gt;fix&lt;/em&gt; my—er, other
people’s mistakes automatically, so much the better. The aptly named &lt;a
href=&#34;https://github.com/Chiel92/vim-autoformat&#34;&gt;Vim-Autoformat&lt;/a&gt;
solves this problem nicely. Like Syntastic, Autoformat uses external
programs to format your code. It doesn’t have support for as many
programs built-in as does Syntastic, but it’s very easy to define your
own.&lt;/p&gt;
&lt;p&gt;I set autoformat to run when I save a file, but not to do the default
vim sequence of autoindenting, retabbing, and removing trailing spaces.
Effectively, this means it only does anything if I have a formatter
installed.&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Autoformat
au! BufWrite * :Autoformat
let g:autoformat_autoindent = 0
let g:autoformat_retab = 0
let g:autoformat_remove_trailing_spaces = 0&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;One odd corner case I’ve had to be a bit clever to deal with is
markdown. Generally when I write, I’m writing in Pandoc’s syntax, but
for a few situations (primarily this blog), I’m using a slightly
different form of the language. Now, I can use Pandoc to auto-format in
either case, but I’ll need to vary the external call. The way I’ve
solved this is to use an autocommand to set a default markdown flavor in
a buffer-scoped variable, then setting a different flavor for specific
matches—in this case when I open a file with a full
&lt;code&gt;.markdown&lt;/code&gt; extension that has a directory named &lt;q&gt;blog&lt;/q&gt;
somewhere in its path. Then I use the value of the flavor in the call to
pandoc. That all looks like this:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;augroup markdown_flavor
    au! BufNewFile,BufFilePre,BufRead *.md 
                \ let b:markdown_flavor=&amp;quot;markdown&amp;quot;
    au! BufNewFile,BufFilePre,BufRead *.markdown
                \ let b:markdown_flavor=&amp;quot;markdown&amp;quot;
    au! BufNewFile,BufFilePre,BufRead */blog/*.markdown
                \ let b:markdown_flavor=&amp;quot;markdown_github&amp;quot;.
                \&amp;quot;+footnotes&amp;quot;.
                \&amp;quot;+yaml_metadata_block&amp;quot;.
                \&amp;quot;-hard_line_blocks&amp;quot;
augroup END

let g:formatdef_pandoc =
            \&amp;#39;&amp;quot;pandoc  --standalone --atx-headers --columns=79&amp;#39;.
            \&amp;#39; -f markdown -t &amp;quot;.b:markdown_flavor&amp;#39;
let g:formatters_markdown_pandoc = [&amp;#39;pandoc&amp;#39;]&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;/section&gt;
&lt;section id=&#34;particular-plugin-functionality&#34; class=&#34;level3&#34;&gt;
&lt;h3&gt;Particular Plugin Functionality&lt;/h3&gt;
&lt;section id=&#34;distraction-free-writing&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Distraction-Free Writing&lt;/h4&gt;
&lt;p&gt;Distraction-free writing is an interesting concept that has been
around for a while. The first implementation I remember hearing about
was &lt;a
href=&#34;http://www.hogbaysoftware.com/products/writeroom&#34;&gt;WriteRoom&lt;/a&gt;,
and since then the concept has even made its way a bit into recent
version of Word. I don’t always use it when I’m writing, but sometimes
it’s helpful. &lt;a href=&#34;https://github.com/junegunn/goyo.vim&#34;&gt;Goyo&lt;/a&gt; is
about as good an implementation as you could ask for, especially when
combined with the &lt;a
href=&#34;https://github.com/junegunn/limelight.vim&#34;&gt;Limelight&lt;/a&gt; plugin to
focus on individual paragraphs. You only need two lines to make this
work together nicely:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Goyo &amp;amp; Limelight
autocmd! User GoyoEnter Limelight
autocmd! User GoyoLeave Limelight!&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;pandoc-syntax&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Pandoc Syntax&lt;/h4&gt;
&lt;p&gt;Vim doesn’t have a built-in Pandoc filetype or syntax file, and
Pandoc really goes a long way beyond simple markdown. There’s a &lt;a
href=&#34;https://github.com/vim-pandoc/vim-pandoc&#34;&gt;Vim-Pandoc&lt;/a&gt; plugin,
but I found myself turning off an awful lot of the functionality because
it was either in my way or a re-implementation of something I already
had. Finally I decided just to use the syntax file, which is helpfully
separated into its own plugin named &lt;a
href=&#34;https://github.com/vim-pandoc/vim-pandoc-syntax&#34;&gt;Vim-Pandoc-Syntax&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;To get files to use the correct syntax, you have to use an
autocommand. The syntax plugin is also surprisingly powerful; I turn off
the &lt;q&gt;conceal&lt;/q&gt; functionality because I don’t like the way it looks,
but I’m very impressed by the ability to use the syntax of the embedded
language in fenced code blocks. Here’s my configuration:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;&amp;quot;&amp;quot; Pandoc
augroup pandoc_syntax
    au! BufNewFile,BufFilePre,BufRead *.md set filetype=markdown.pandoc
    au! BufNewFile,BufFilePre,BufRead *.markdown set filetype=markdown.pandoc
augroup END

let g:pandoc#syntax#conceal#use = 0
let g:pandoc#syntax#codeblocks#embeds#langs = [&amp;#39;python&amp;#39;, &amp;#39;vim&amp;#39;, &amp;#39;make&amp;#39;,
            \  &amp;#39;bash=sh&amp;#39;, &amp;#39;html&amp;#39;, &amp;#39;css&amp;#39;, &amp;#39;scss&amp;#39;, &amp;#39;javascript&amp;#39;]&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;table-formatting&#34; class=&#34;level4&#34;&gt;
&lt;h4&gt;Table Formatting&lt;/h4&gt;
&lt;p&gt;&lt;a href=&#34;https://github.com/godlygeek/tabular&#34;&gt;Tabular&lt;/a&gt; is one of
those wonderful little pieces of code that does one thing extremely
well. Tabular makes tables. That’s it. When you don’t need a table, you
don’t have to think about it. When you do, it saves you ten minutes of
fiddling around. It plays very well with Pandoc.&lt;/p&gt;
&lt;/section&gt;
&lt;/section&gt;
&lt;/section&gt;
&lt;section id=&#34;plugins-i-didnt-use&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Plugins I Didn’t Use&lt;/h2&gt;
&lt;p&gt;Obviously there are lots of plugins I didn’t install. Here are a few
that I know are popular, and why I didn’t use them:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/tpope/vim-fugitive&#34;&gt;Fugitive&lt;/a&gt;: See, I
don’t love &lt;em&gt;everything&lt;/em&gt; Tim Pope does. I had this installed for a
while, but never found myself using it. I’m happy on the command
line.&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/vim-airline/vim-airline&#34;&gt;Airline&lt;/a&gt;:
These kinds of plugins just strike me as a way to throw lots of
distracting information onto the screen. I don’t see the
attraction.&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/scrooloose/nerdtree&#34;&gt;NerdTree&lt;/a&gt;: I’m
happy with NetRW the way I have it.&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/majutsushi/tagbar&#34;&gt;Tagbar&lt;/a&gt;: I don’t
want to have to install ctags, and I can just search.&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/kien/ctrlp.vim&#34;&gt;CtrlP&lt;/a&gt;: Apparently
people have more trouble than I do finding things?&lt;/li&gt;
&lt;li&gt;&lt;a
href=&#34;https://github.com/terryma/vim-multiple-cursors&#34;&gt;Multiple-Cursors&lt;/a&gt;:
I almost went for this one, but &lt;a
href=&#34;https://medium.com/@schtoeffel/you-don-t-need-more-than-one-cursor-in-vim-2c44117d51db&#34;&gt;Christoph
Hermann’s article on it&lt;/a&gt; convinced me that it didn’t do anything you
can’t just do with built-in functionality.&lt;/li&gt;
&lt;/ul&gt;
&lt;/section&gt;
&lt;section id=&#34;gvim&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Gvim&lt;/h2&gt;
&lt;p&gt;My &lt;code&gt;gvimrc&lt;/code&gt; is simple: I turn everything off, and set my
fonts:&lt;/p&gt;
&lt;pre class=&#34;vim&#34;&gt;&lt;code&gt;set guioptions-=m &amp;quot; Turn off menubar
set guioptions-=T &amp;quot; Turn off toolbar
set guioptions-=r &amp;quot; Turn off right-hand scrollbar
set guioptions-=R &amp;quot; Turn off right-hand scrollbar when split
set guioptions-=L &amp;quot; Turn off left-hand scrollbar
set guioptions-=l &amp;quot; Turn off left-hand=scrollbar when split
set guicursor+=a:blinkon0 &amp;quot; Turn off blinking cursor

if has(&amp;quot;win32&amp;quot;)
    set guifont=Consolas:h11
else
    set guifont=Inconsolata\ 12
endif&lt;/code&gt;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id=&#34;implementation-notes&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Implementation Notes&lt;/h2&gt;
&lt;p&gt;Vim keeps its files in a &lt;code&gt;.vim&lt;/code&gt; folder on Linux, and a
&lt;code&gt;vimfiles&lt;/code&gt; folder on Windows. Happily, in new versions of
vim, the &lt;code&gt;vimrc&lt;/code&gt; and &lt;code&gt;gvimrc&lt;/code&gt; can live inside this
folder, which makes keeping everything in git easier.&lt;/p&gt;
&lt;p&gt;To ensure that my directories for undo, backup, and swap exist but
aren’t versioned, I put a &lt;code&gt;.gitignore&lt;/code&gt; in each with this
content:&lt;/p&gt;
&lt;pre class=&#34;gitignore&#34;&gt;&lt;code&gt;*
!.gitignore&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I also have a &lt;code&gt;.gitignore&lt;/code&gt; in the root directory, to
ignore both my netrw history, my spelling files, and my plugins.&lt;/p&gt;
&lt;pre class=&#34;gitignore&#34;&gt;&lt;code&gt;.netrwhist
spell/*
plugged/*&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I leave the autoload directory versioned, which means VimPlug itself
gets versioned. This saves me a step when I move to a different machine
and it’s not too terrible to update the repositiory when I occasionally
update VimPlug itself. So here’s the gitignore.&lt;/p&gt;
&lt;p&gt;With all that done, all I need to do to get set up on a new box (with
git installed) is clone my vimfiles repository, fire up vim or Gvim, run
&lt;code&gt;:PlugInstall&lt;/code&gt; and restart, which take almost no time at
all.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;final-thoughts&#34; class=&#34;level2&#34;&gt;
&lt;h2&gt;Final Thoughts&lt;/h2&gt;
&lt;p&gt;I should disclose that this post has taken me an absurd amount of
time to write. I’ve spent hours now comparing plugins, trying things
out, reading through documentation, and changing my mind. I know Vim far
better than I did when I started, no doubt about it. If your
&lt;code&gt;vimrc&lt;/code&gt; has gotten a bit stale, it might be a good time for
you to do a Vim Zero experiment yourself. Or, feel free to build off my
setup, which &lt;a href=&#34;https://github.com/OliverSherouse/vimfiles&#34;&gt;you
can find on GitHub here&lt;/a&gt;. If you decide to give it a go, be sure to
let me know on &lt;a
href=&#34;https://twitter.com/OliverSherouse&#34;&gt;Twitter&lt;/a&gt;.&lt;/p&gt;
&lt;/section&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;Yes, I’ve heard of Sublime and PyCharm and Atom and all
of them. No thank you. Too much noise. You have fun, though.&lt;a
href=&#34;#fnref1&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn2&#34;&gt;&lt;p&gt;I mean, just look at this website. I like things clean
and simple.&lt;a href=&#34;#fnref2&#34; class=&#34;footnote-back&#34;
role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn3&#34;&gt;&lt;p&gt;Makefiles are the exception, which is
&lt;em&gt;infuriating&lt;/em&gt;.&lt;a href=&#34;#fnref3&#34; class=&#34;footnote-back&#34;
role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn4&#34;&gt;&lt;p&gt;The only real problem I have with it is that if you try
to edit a directory path, you get a NetRW buffer that doesn’t go away.
So don’t do that.&lt;a href=&#34;#fnref4&#34; class=&#34;footnote-back&#34;
role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn5&#34;&gt;&lt;p&gt;My requirements here include: black background and
nothing else, not too much yellow, not too much orange, no neon-type
colors (the hot pink completion menu just &lt;em&gt;ruins&lt;/em&gt; the Ocean
family of schemes, and my old vimrc had code to manually replace it),
&amp;amp;c, &amp;amp;c.&lt;a href=&#34;#fnref5&#34; class=&#34;footnote-back&#34;
role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn6&#34;&gt;&lt;p&gt;Oh, don’t judge, you don’t write them either.&lt;a
href=&#34;#fnref6&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/unix-anaconda-m2.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/unix-anaconda-m2.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>Get a Decent Unix Toolkit on Windows With Anaconda</title>
    <updated>2017-01-03T00:00:00Z</updated>
    <content type="html">&lt;p&gt;I’m a Linux guy at heart, but like a lot of folks, I’m stuck on
Windows at work. But even on Windows, I spend a lot of time in the
command line. Now, the whole point of Windows was to get away from the
command line, so the default command prompt, &lt;code&gt;cmd.exe&lt;/code&gt;, has
never been much more than a glorified DOS shell. The alternative for
power users, PowerShell, is something I’ve always found to be confusing
and terrible.&lt;/p&gt;
&lt;p&gt;I want my Unix tools, dang it! And, open source being open source,
there have been a few attempts to make that happen, most notably &lt;a
href=&#34;https://www.cygwin.com/&#34;&gt;Cygwin&lt;/a&gt; and &lt;a
href=&#34;http://www.mingw.org/&#34;&gt;MinGW&lt;/a&gt;. But while those are certainly
impressive projects, they’re not something I want to try to keep updated
for my team, or document for researchers who I want to use my scripts;
the install and update tools are just too complicated for that.&lt;/p&gt;
&lt;p&gt;Happily, &lt;a href=&#34;http://www.mingw.org/&#34;&gt;Anaconda&lt;/a&gt; is there to
save the day. Again&lt;a href=&#34;#fn1&#34; class=&#34;footnote-ref&#34; id=&#34;fnref1&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt; . The Anaconda default channels
include a suite of tools built with M2, a project descended from MinGW.
That means that not only do you get to manage the tools you want with
the excellent conda package manager, but you also can easily reproduce
your environment elsewhere using conda requirement files. And no
administrator privileges needed!&lt;/p&gt;
&lt;p&gt;There’s only one drawback, which really isn’t a drawback: you can’t
run m2-based programs in the default anaconda environment. Why isn’t
that really a drawback? Because it keeps your path clean if you ever
need to be using the Windows built-in tools instead of Unix ones that
happen to have the same name.&lt;/p&gt;
&lt;p&gt;To install m2 (assuming you have Anaconda already installed), just
create a new conda environment or switch to one that already exists.
Then install the &lt;code&gt;m2-base&lt;/code&gt; package. I have one environment
called &lt;q&gt;main&lt;/q&gt; which I use when I’m not isolating requirements on a
project for release. You can create one by running this command
(replacing &lt;code&gt;main&lt;/code&gt; with the desired name):&lt;/p&gt;
&lt;div class=&#34;sourceCode&#34; id=&#34;cb1&#34;&gt;&lt;pre
class=&#34;sourceCode cmd&#34;&gt;&lt;code class=&#34;sourceCode dosbat&#34;&gt;&lt;span id=&#34;cb1-1&#34;&gt;&lt;a href=&#34;#cb1-1&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;conda create &lt;span class=&#34;at&#34;&gt;-n&lt;/span&gt; main m2&lt;span class=&#34;at&#34;&gt;-base&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Now, whenever you want to use your Unix tools, just run
&lt;code&gt;activate main&lt;/code&gt; (or whatever you called the environment).&lt;/p&gt;
&lt;p&gt;Or, if you have an environment you want to use already, just activate
it and &lt;code&gt;conda install m2-base&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;That will give you all the &lt;a
href=&#34;https://www.gnu.org/software/coreutils/manual/coreutils.html&#34;&gt;coreutils&lt;/a&gt;,
as well as bash if you want to use that. And, while this gives you a
perfectly good base system, there are plenty of more tools, like &lt;a
href=&#34;/posts/makefiles.html&#34;&gt;make&lt;/a&gt;. Just run
&lt;code&gt;conda search m2-.*&lt;/code&gt; to see them all.&lt;/p&gt;
&lt;p&gt;Is it as good as using Linux? No. No it is not. But as far as I can
tell, it’s the next best thing.&lt;/p&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;If you aren’t using Anaconda to manage your Python
environment on Windows, you really, really should be. Start &lt;a
href=&#34;http://anaconda-installer.readthedocs.io/en/latest/intro.html&#34;&gt;here&lt;/a&gt;.&lt;a
href=&#34;#fnref1&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/430_general.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/430_general.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>Outta Nowhere! 430 Projections for Clinton-Trump</title>
    <updated>2016-11-08T00:00:00Z</updated>
    <content type="html">&lt;p&gt;Back in January, I introduced the &lt;a
href=&#34;http://www.oliversherouse.com/2016/02/01/poll_forecast.html&#34;&gt;430
Model&lt;/a&gt; for using polls to forecast elections. The name comes from the
idea that you get 20 percent of your results from 80 percent of your
work, and I only wanted to do about 20 percent of what Nate Silver does
with FiveThirtyEight.&lt;/p&gt;
&lt;p&gt;Since then I’ve wanted to do a bigger and better version for the
general election. But since I’m the kind of guy who only wants to do 20
percent of the work, most of that didn’t happen. But I do have a
&lt;em&gt;slightly&lt;/em&gt; bigger and better version! So that’s something.&lt;/p&gt;
&lt;p&gt;As before, I’m using the poll data from the &lt;a
href=&#34;http://elections.huffingtonpost.com/pollster/api&#34;&gt;Pollster
API&lt;/a&gt;, combining the results and weighting by age to create a
super-poll. Then I’m running 10,000 simulations using the results of the
super-poll to figure out how likely each candidate is to win.&lt;/p&gt;
&lt;p&gt;The code for this version is in &lt;a
href=&#34;http://elections.huffingtonpost.com/pollster/api&#34;&gt;a Jupyter
notebook on GitHub&lt;/a&gt;, so you can play with it if you want to pretend
Jill Stein matters or something.&lt;/p&gt;
&lt;p&gt;Now to the results! Using the super-poll results since July, the race
has looked like this:&lt;/p&gt;
&lt;figure&gt;
&lt;img src=&#34;/img/Clinton-Trump.png&#34;
alt=&#34;Clinton is winning by a lot in the super-poll&#34; /&gt;
&lt;figcaption aria-hidden=&#34;true&#34;&gt;Clinton is winning by a lot in the
super-poll&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Those lines are the 95% confidence bands. The story here is that,
while things were pretty uncertain early in the race, Clinton began to
separate after the first debate in September, and Trump has never been
able to catch up. The recent revelations about the emails, for all the
sturm und drang they caused, don’t really show up here. She’s just
ahead. How ahead? This ahead:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align: left;&#34;&gt;Candidate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Mean&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Standard Deviation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Hillary Clinton&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;48.4%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;(0.2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Donald Trump&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;42.7%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;(0.2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Other&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;8.9%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;(0.1)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Well, she’s not over fifty. But five and a half points is a pretty
solid lead with as many polls as we’ve had. In 10,000 trial simulations,
she won every single one. That’s as comfortable as you get.&lt;/p&gt;
&lt;p&gt;So, any caution? If, as many suspect, there’s a weird Trumpian
version of the &lt;a
href=&#34;https://en.wikipedia.org/wiki/Bradley_effect&#34;&gt;Bradley Effect&lt;/a&gt;,
then he has a better chance than the model projects, which is none. But
with 0.2 percentage points standard error, he needs to get to swing
better than two and a half points if he’s drawing them all from Clinton,
and More if he’s getting them from the &lt;q&gt;Other&lt;/q&gt; category. In the
best case, that’s one secret supporter to every public one. That’s a
tough road to hoe.&lt;/p&gt;
&lt;p&gt;Still, this is a strictly poll-based model. If the polls are
&lt;em&gt;systematically&lt;/em&gt; wrong, then so is the model. We didn’t really
see systematically wrong polling in the primaries, though we have seen
it in votes abroad, such as Brexit. But that has to be the hope, if
you’re a Trump supporter: that the polls are almost all wrong and in the
same way.&lt;/p&gt;
&lt;p&gt;My gut says no. Clinton’s got this one.&lt;/p&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/makefiles.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/makefiles.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>Why aren't you using make for your data projects?</title>
    <updated>2016-04-07T00:00:00Z</updated>
    <content type="html">&lt;p&gt;Data work can take a long, long time. Once you’ve moved beyond small
scale projects, you just have to get used to doing something else while
your machine chugs away at cleaning, preparing, or analyzing your data.
And since most of the time you have to try a few things to see what
comes out, you’ll have to do sit and wait through multiple rounds of
each step.&lt;/p&gt;
&lt;p&gt;But a lot of people make it worse by doing one of two things: either
they have one big script which they re-run from start to finish every
time they make a change, or they run each step on-demand (say, in an
Jupyter notebook), and mentally keep track of what they need to do in
what order.&lt;/p&gt;
&lt;p&gt;The first is bad because every single run will take the whole run
time, and your life will be swallowed up in waiting. The second is bad
because you have to depend on yourself to keep what can be a pretty
intricate map of your analysis flow in your head.&lt;/p&gt;
&lt;p&gt;The Unix utility &lt;a
href=&#34;https://www.gnu.org/software/make/manual/html_node/index.html&#34;&gt;Make&lt;/a&gt;
provides a better solution. Make is the old standard build system,
designed to help with the compilation of large programs. Just like with
data projects, compiling large programs takes forever, has a variety of
smaller steps where you might need to go back and edit something, and
has a web of dependent steps you might need to re-run depending on what
you’ve changed. In other words: it’s a perfect fit.&lt;/p&gt;
&lt;p&gt;What’s more, Make helps you keep your steps separate computationally,
which helps &lt;em&gt;you&lt;/em&gt; keep them clear conceptually. You don’t even
have to use the same language with each step: I’ve had occasion to mix
Python, Scala, and &lt;em&gt;Stata&lt;/em&gt; in the same project,&lt;a href=&#34;#fn1&#34;
class=&#34;footnote-ref&#34; id=&#34;fnref1&#34; role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt; and
make made that much less painful.&lt;/p&gt;
&lt;p&gt;And here’s the good news: if you’re on a Mac or Linux, it’s probably
already installed. On windows, just install &lt;a
href=&#34;https://github.com/bmatzelle/gow/wiki&#34;&gt;Gow&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;So how does it work? The basic idea is, you write a &lt;em&gt;makefile&lt;/em&gt;
filled with &lt;em&gt;recipes&lt;/em&gt; that tell you how to &lt;em&gt;make&lt;/em&gt; a
&lt;em&gt;target&lt;/em&gt;. Each target can have &lt;em&gt;dependencies&lt;/em&gt;, and if a
dependency has been modified more recently than the target, the recipe
for the target gets re-run.&lt;/p&gt;
&lt;p&gt;So let’s say I want to classify some text in a folder, using some
trainers in another folder. Both have to be prepped for analysis, using
a script in my scripts subdirectory. I might put the following recipe in
my makefile:&lt;/p&gt;
&lt;pre class=&#34;make&#34;&gt;&lt;code&gt;data/clean/trainers.csv: scripts/clean_text.py data/raw/trainers
    python -o data/clean/trainers.csv scripts/clean_text.py data/raw/trainers&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;What’s going on here? The file before the colon is the target, the
thing we’re trying to make. The files after are dependencies, things
that need to be made before the target. The line beneath, which is
indented by a tab—yes, a &lt;strong&gt;tab&lt;/strong&gt;—is the command that makes
the file. Now, if we were to run &lt;code&gt;make data/clean/trainers&lt;/code&gt;,
make would check to see if either the clean_text.py script or the
trainers directory&lt;a href=&#34;#fn2&#34; class=&#34;footnote-ref&#34; id=&#34;fnref2&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt; had been modified more recently than
the output file, and if so, it would run the script to create the
file.&lt;/p&gt;
&lt;p&gt;We can write this recipe a simpler way:&lt;/p&gt;
&lt;pre class=&#34;make&#34;&gt;&lt;code&gt;data/clean/trainers.csv: scripts/clean_text.py data/raw/trainers
    python -o $@ $^&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In a makefile, &lt;code&gt;$@&lt;/code&gt; stands for the target, and
&lt;code&gt;$^&lt;/code&gt; stands for the list of dependencies in order. This means
if our dependencies are a script and a list of arguments to that script,
we can use them as a stand-in for the recipe.&lt;/p&gt;
&lt;p&gt;Now let’s say we use the same script to clean the unlabeled input. We
just need to add it as a new target:&lt;/p&gt;
&lt;pre class=&#34;make&#34;&gt;&lt;code&gt;data/clean/trainers.csv: scripts/clean_text.py data/raw/trainers
    python -o $@ $^

data/clean/unlabeled.csv: scripts/clean_text.py data/raw/unlabeled
    python -o $@ $^&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Easy! Now if we update clean_text.py, Make knows we need to remake
both those targets. But I hate repeating myself. Luckily, Make gives us
&lt;em&gt;canned recipes&lt;/em&gt;:&lt;/p&gt;
&lt;pre class=&#34;make&#34;&gt;&lt;code&gt;define pyscript
python -o $@ $^
endef

data/clean/trainers.csv: scripts/clean_text.py data/raw/trainers
    $(pyscript)

data/clean/unlabeled.csv: scripts/clean_text.py data/raw/unlabeled
    $(pyscript)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In fact, since I write all my scripts based off of the same &lt;a
href=&#34;/posts/boilerplate.html&#34;&gt;boilerplate&lt;/a&gt;, let’s fill out what the
whole project might look like:&lt;/p&gt;
&lt;pre class=&#34;make&#34;&gt;&lt;code&gt;define pyscript
python -o $@ $^
endef

data/results/output.csv: scripts/classify.py data/classifier.pickle data/clean/unlabeled.csv
    $(pyscript)

data/classifier.pickle: scripts/train_classifier.py data/clean/trainers.csv data/trainer_labels.csv
    $(pyscript)

data/results/analysis.csv: scripts/tune_classifier.py data/clean/trainers.csv data/clean/unlabeled.csv
    $(pyscript)

data/clean/trainers.csv: scripts/clean_text.py data/raw/trainers
    $(pyscript)

data/clean/unlabeled.csv: scripts/clean_text.py data/raw/unlabeled
    $(pyscript)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So that’s five scripts total: each reasonably separated and able to
be dropped into other projects with minimal modification. If we change
any of them, either because of a bug, or because we wanted to try
something different, we can use make to update only those parts that are
dependent on the chain. And if we just type the command
&lt;code&gt;make&lt;/code&gt;, it automatically makes the first recipe, so we can be
sure that our output.csv is using all the latest and greatest work we’ve
put in.&lt;/p&gt;
&lt;p&gt;There’s a lot more to Make, and I’ll focus in on a few features,
tips, and tricks in an occasional series here. It’s been a big help to
me, and if you find it helpful too, I’d love to &lt;a
href=&#34;http://twitter.com/OliverSherouse&#34;&gt;hear from you&lt;/a&gt;!&lt;/p&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;I said I did it, not that I’m proud of it.&lt;a
href=&#34;#fnref1&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li id=&#34;fn2&#34;&gt;&lt;p&gt;The directory itself, mind you, not the files inside
it!&lt;a href=&#34;#fnref2&#34; class=&#34;footnote-back&#34;
role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/430_nh.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/430_nh.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>You Asked For It: 430 Model Projections for Iowa</title>
    <updated>2016-02-09T00:00:00Z</updated>
    <content type="html">&lt;p&gt;Just before the Iowa Caucuses, I showed how you can make a
80-percent-of-the-way there poll-based election forecasting system with
surprisingly little work. I called it the &lt;a
href=&#34;/posts/poll_forecast.html&#34;&gt;430 Model&lt;/a&gt;, because 430 is about 80
percent of 538, and I was ripping off Nate Silver. &lt;a
href=&#34;/posts/after_iowa.html&#34;&gt;My model did OK&lt;/a&gt; on the Democratic
side, though (like the polls) it was off on the Republican side; Trump’s
support was overstated, or rather the opposition to Trump was
understated.&lt;/p&gt;
&lt;p&gt;But, since people have been interested, I’ve updated the model a bit
and made forecasts for New Hampshire! I’ve simplified the weighting
decay function to halve the weighted sample size for a poll each day&lt;a
href=&#34;#fn1&#34; class=&#34;footnote-ref&#34; id=&#34;fnref1&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;. I’ve also added limited my results
to polls with likely voter screens, and set the window of polls included
in the super-poll by number instead of by date. You can download and
play around with the new model as a Jupyter notebook &lt;a
href=&#34;http://nbviewer.jupyter.org/github/Oliversherouse/blog_notebooks/blob/master/430_nh/430%20Model%20-%20New%20Hampshire%20Edition.ipynb&#34;&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;So, let’s get to the fun!&lt;/p&gt;
&lt;p&gt;This is what the New Hampshire race has looked like over the last
three weeks on the GOP side, when polling has happened pretty
steadily:&lt;/p&gt;
&lt;figure&gt;
&lt;img src=&#34;/img/430_nh/nh-gop.png&#34;
title=&#34;Trump has a huge lead. Yes, huge. The joke&amp;#39;s been old for months&#34;
alt=&#34;New Hampshire GOP Chart&#34; /&gt;
&lt;figcaption aria-hidden=&#34;true&#34;&gt;New Hampshire GOP Chart&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;The picture’s been pretty stable the whole time. There’s three tiers:
the down-and-outs (Christie, Fiorina, Carson), the hope-for-seconds
(Rubio, Kasich, Cruz, Bush), and Trump. The superpoll results bear that
out:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align: left;&#34;&gt;Candidate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Estimate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Standard Error&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Donald Trump&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;31.2%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Marco Rubio&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;14.5%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;John Kasich&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;13.8%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Ted Cruz&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;11.7%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Jeb Bush&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;11.0%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Chris Christie&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;5.8%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Carly Fiorina&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;4.6%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Ben Carson&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.6%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Rubio’s bump has gotten him to the top of the second tier, but
realistically those guys are all tied, and each of them (except Cruz)
really, really wants a second place finish. First place seems utterly
out of reach for all of them; in my 10,000-trial simulation, Trump won
every single time.&lt;/p&gt;
&lt;p&gt;On the Democratic side, things seem equally fatalistic:&lt;/p&gt;
&lt;figure&gt;
&lt;img src=&#34;/img/430_nh/nh-dem.png&#34;
title=&#34;Sanders has a yuuuuuuuuuuuuuuuuuuuuuuuuuuuuge lead!&#34;
alt=&#34;New Hampshire Dem Chart&#34; /&gt;
&lt;figcaption aria-hidden=&#34;true&#34;&gt;New Hampshire Dem Chart&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Not even a story there. Bernie Sanders is winning. So also says the
superpoll:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align: left;&#34;&gt;Candidate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Estimate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Standard Error&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Bernie Sanders&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;55.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Hillary Clinton&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;40.9%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;As Trump did on the GOP side, Sanders wins every time in a
10,000-trial simulation.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;But.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Polls have &lt;a
href=&#34;https://twitter.com/bpmehlman/status/696699218187198464&#34;&gt;historically
been bad&lt;/a&gt; in New Hampshire. These projections are a great summary of
what the polls are telling us, but if the polls are consistently off by
a significant amount, we could still get some real surprises. If I were
Sanders or (especially) Trump, I would actually wish my numbers were a
little worse; in both cases, not dominating is going to look like a
loss. It’s entirely possible that one of those second-tier GOP
candidates is really going to break out, though I have no idea who it
would be or why.&lt;/p&gt;
&lt;p&gt;Or maybe they’ve finally learned how to poll New Hampshire and this
is all effectively a done deal. We’ll find out tonight!&lt;/p&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;This sounds more extreme than it is. A poll with 300
respondents has a margin of error of about 2.9 percent, one with 150
respondents about 4.1 percent, and one with 75 respondents about 5.8
percent. When you think about how many people can change their minds
every day during an election, that looks reasonable to me.&lt;a
href=&#34;#fnref1&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/after_iowa.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/after_iowa.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>After Iowa: How Did the 430 Forecast Perform?</title>
    <updated>2016-02-02T00:00:00Z</updated>
    <content type="html">&lt;p&gt;So how did my &lt;a href=&#34;/posts/poll_forecast.html&#34;&gt;430 model&lt;/a&gt; do?
On both sides the 30-day version was significantly worse than the 4-day.
That makes sense; there was a lot of polling in the final days, and a
good number of voters wait until the end of a campaign to make up their
minds already.&lt;/p&gt;
&lt;p&gt;On the Democratic side, the race looked like a coin flip, with the
odds very, very slightly in Clinton’s favor. As it turns out the race is
being called (by Sanders, at least) a &lt;a
href=&#34;http://www.huffingtonpost.com/entry/bernie-sanders-iowa_us_56b03247e4b057d7d7c7fcfa?dyytlnmi&#34;&gt;virtual
tie&lt;/a&gt;, and in fact several delegates &lt;em&gt;were&lt;/em&gt; awarded on &lt;a
href=&#34;http://www.theguardian.com/us-news/2016/feb/02/how-hillary-clinton-won-some-iowa-caucuses-with-a-coin-toss&#34;&gt;coin
flips&lt;/a&gt;&lt;a href=&#34;#fn1&#34; class=&#34;footnote-ref&#34; id=&#34;fnref1&#34;
role=&#34;doc-noteref&#34;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;. Martin O’Malley over-performed in
the model; seems like none of his people bothered to show up.&lt;/p&gt;
&lt;p&gt;On the Republican side, the 430 model far overestimated Trump’s
chances, and underestimated both Cruz and Rubio. It’s a great example of
&lt;q&gt;garbage in, garbage out.&lt;/q&gt; The polls overestimated how many of
Trump’s voters would turn out (and how few of Cruz’s would), so the
model did the same. With Rubio, I think, what happened was a little
different: he surged very late and very fast, and there just weren’t
many polls at the right time to catch it.&lt;/p&gt;
&lt;p&gt;So, should we change the model at all? Maybe we should assume all
Trump results are inflated (which I feel is probably true) and deflate
them by some amount. That feels pretty arbitrary, though, and we’d
probably overcompensate anyway. Or we could try to see which pollsters
are the best and weight accordingly. That also seems prone to
over-fitting, though, unless you want to go the full 538. So I don’t
think I’m going to try to add too much sophistication in it that
way.&lt;/p&gt;
&lt;p&gt;I &lt;em&gt;do&lt;/em&gt; think I may fool around both with the time-weighting
function, which was more or less pulled out of a hat, and with the
window. It may be that, instead of looking at a time-window (so, all
polls in the last 30 days), it would be more helpful to look at a
number-of-polls window, still weighted by time-delay. That way, as we
get closer to elections and more polling happens, we zero in
automatically. I’ll probably put out a new version for New Hampshire and
see what that looks like.&lt;/p&gt;
&lt;p&gt;And you should, too! The notebook is available &lt;a
href=&#34;http://nbviewer.jupyter.org/github/OliverSherouse/blog_notebooks/blob/master/430%20Model.ipynb&#34;&gt;here&lt;/a&gt;,
so feel free to make your own version and let me know on &lt;a
href=&#34;http://twitter.com/OliverSherouse&#34;&gt;Twitter&lt;/a&gt; if you come up with
anything interesting!&lt;/p&gt;
&lt;section id=&#34;footnotes&#34; class=&#34;footnotes footnotes-end-of-document&#34;
role=&#34;doc-endnotes&#34;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&#34;fn1&#34;&gt;&lt;p&gt;The Clinton campaign won 6 of 6 tosses, the odds of
which are only 1 in 64. But surely if the Clinton camp were sharp enough
to pre-supply unfair coins they would have won outright, right? Right?&lt;a
href=&#34;#fnref1&#34; class=&#34;footnote-back&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
    
  </entry>
  
  <entry>
    <id>https://www.oliversherouse.com/posts/poll_forecast.html</id>
    <link rel="alternate" href="https://www.oliversherouse.com/posts/poll_forecast.html" />
    <author>
      <name>Oliver Sherouse</name>
      <email>oliver@oliversherouse.com</email>
    </author>
    <title>How to Knock Off Nate Silver: the 430 Election Forecast Model</title>
    <updated>2016-02-01T00:00:00Z</updated>
    <content type="html">&lt;p&gt;It’s caucus day in Iowa, so what better time to rip off Nate Silver?
Silver has a well-respected &lt;a
href=&#34;http://projects.fivethirtyeight.com/election-2016/primary-forecast/&#34;&gt;election
forecasting model&lt;/a&gt; based on polls, polling firms’ historical house
effects and accuracy, and a few non-poll factors like endorsements.
Probably took a lot of hard work.&lt;/p&gt;
&lt;p&gt;But it’s actually pretty easy to build a simple poll-based model
yourself using Python. I’ve thrown one together, which I’m calling the
430 model. Why 430? Because of the 80/20 rule: you can get 80 percent of
the way there with 20 percent of the work, and 430 is about 80 percent
of 538.&lt;/p&gt;
&lt;p&gt;Everything below is available in a Jupyter notebook &lt;a
href=&#34;http://nbviewer.jupyter.org/github/Oliversherouse/blog_notebooks/blob/master/430%20Model.ipynb&#34;&gt;here&lt;/a&gt;
Let’s start with a little setup:&lt;/p&gt;
&lt;div class=&#34;sourceCode&#34; id=&#34;cb1&#34;&gt;&lt;pre
class=&#34;sourceCode python&#34;&gt;&lt;code class=&#34;sourceCode python&#34;&gt;&lt;span id=&#34;cb1-1&#34;&gt;&lt;a href=&#34;#cb1-1&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;im&#34;&gt;import&lt;/span&gt; collections&lt;/span&gt;
&lt;span id=&#34;cb1-2&#34;&gt;&lt;a href=&#34;#cb1-2&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;im&#34;&gt;import&lt;/span&gt; datetime&lt;/span&gt;
&lt;span id=&#34;cb1-3&#34;&gt;&lt;a href=&#34;#cb1-3&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;/span&gt;
&lt;span id=&#34;cb1-4&#34;&gt;&lt;a href=&#34;#cb1-4&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;im&#34;&gt;import&lt;/span&gt; numpy &lt;span class=&#34;im&#34;&gt;as&lt;/span&gt; np&lt;/span&gt;
&lt;span id=&#34;cb1-5&#34;&gt;&lt;a href=&#34;#cb1-5&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;im&#34;&gt;import&lt;/span&gt; pandas &lt;span class=&#34;im&#34;&gt;as&lt;/span&gt; pd&lt;/span&gt;
&lt;span id=&#34;cb1-6&#34;&gt;&lt;a href=&#34;#cb1-6&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;im&#34;&gt;import&lt;/span&gt; requests&lt;/span&gt;
&lt;span id=&#34;cb1-7&#34;&gt;&lt;a href=&#34;#cb1-7&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;/span&gt;
&lt;span id=&#34;cb1-8&#34;&gt;&lt;a href=&#34;#cb1-8&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;API_ENDPOINT &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;st&#34;&gt;&amp;quot;http://elections.huffingtonpost.com/pollster/api/polls&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span id=&#34;cb1-9&#34;&gt;&lt;a href=&#34;#cb1-9&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;/span&gt;
&lt;span id=&#34;cb1-10&#34;&gt;&lt;a href=&#34;#cb1-10&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;np.random.seed(&lt;span class=&#34;dv&#34;&gt;2016&lt;/span&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;So, first, let’s get our polling data, which we can do using the &lt;a
href=&#34;http://elections.huffingtonpost.com/pollster/api&#34;&gt;Pollster
API&lt;/a&gt;:&lt;/p&gt;
&lt;div class=&#34;sourceCode&#34; id=&#34;cb2&#34;&gt;&lt;pre
class=&#34;sourceCode python&#34;&gt;&lt;code class=&#34;sourceCode python&#34;&gt;&lt;span id=&#34;cb2-1&#34;&gt;&lt;a href=&#34;#cb2-1&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;kw&#34;&gt;def&lt;/span&gt; get_all_results(state&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;US&amp;#39;&lt;/span&gt;, party&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;gop&amp;#39;&lt;/span&gt;, start_date&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;2015-6-1&amp;#39;&lt;/span&gt;):&lt;/span&gt;
&lt;span id=&#34;cb2-2&#34;&gt;&lt;a href=&#34;#cb2-2&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    topic &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;st&#34;&gt;&amp;#39;2016-president-&lt;/span&gt;&lt;span class=&#34;sc&#34;&gt;{}&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;-primary&amp;#39;&lt;/span&gt;.&lt;span class=&#34;bu&#34;&gt;format&lt;/span&gt;(party)&lt;/span&gt;
&lt;span id=&#34;cb2-3&#34;&gt;&lt;a href=&#34;#cb2-3&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    params &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; {&lt;span class=&#34;st&#34;&gt;&amp;#39;state&amp;#39;&lt;/span&gt;: state,&lt;/span&gt;
&lt;span id=&#34;cb2-4&#34;&gt;&lt;a href=&#34;#cb2-4&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;              &lt;span class=&#34;st&#34;&gt;&amp;#39;after&amp;#39;&lt;/span&gt;: start_date,&lt;/span&gt;
&lt;span id=&#34;cb2-5&#34;&gt;&lt;a href=&#34;#cb2-5&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;              &lt;span class=&#34;st&#34;&gt;&amp;#39;topic&amp;#39;&lt;/span&gt;: topic&lt;/span&gt;
&lt;span id=&#34;cb2-6&#34;&gt;&lt;a href=&#34;#cb2-6&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;             }&lt;/span&gt;
&lt;span id=&#34;cb2-7&#34;&gt;&lt;a href=&#34;#cb2-7&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    page &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;dv&#34;&gt;1&lt;/span&gt;&lt;/span&gt;
&lt;span id=&#34;cb2-8&#34;&gt;&lt;a href=&#34;#cb2-8&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;cf&#34;&gt;while&lt;/span&gt; &lt;span class=&#34;va&#34;&gt;True&lt;/span&gt;:&lt;/span&gt;
&lt;span id=&#34;cb2-9&#34;&gt;&lt;a href=&#34;#cb2-9&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        params[&lt;span class=&#34;st&#34;&gt;&amp;#39;page&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; page&lt;/span&gt;
&lt;span id=&#34;cb2-10&#34;&gt;&lt;a href=&#34;#cb2-10&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        page_results &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; requests.get(API_ENDPOINT,&lt;/span&gt;
&lt;span id=&#34;cb2-11&#34;&gt;&lt;a href=&#34;#cb2-11&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                                    params&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;params).json()&lt;/span&gt;
&lt;span id=&#34;cb2-12&#34;&gt;&lt;a href=&#34;#cb2-12&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        &lt;span class=&#34;cf&#34;&gt;for&lt;/span&gt; poll &lt;span class=&#34;kw&#34;&gt;in&lt;/span&gt; page_results:&lt;/span&gt;
&lt;span id=&#34;cb2-13&#34;&gt;&lt;a href=&#34;#cb2-13&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;            subpop &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;bu&#34;&gt;next&lt;/span&gt;(i[&lt;span class=&#34;st&#34;&gt;&amp;#39;subpopulations&amp;#39;&lt;/span&gt;][&lt;span class=&#34;dv&#34;&gt;0&lt;/span&gt;]&lt;/span&gt;
&lt;span id=&#34;cb2-14&#34;&gt;&lt;a href=&#34;#cb2-14&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                          &lt;span class=&#34;cf&#34;&gt;for&lt;/span&gt; i &lt;span class=&#34;kw&#34;&gt;in&lt;/span&gt; poll[&lt;span class=&#34;st&#34;&gt;&amp;#39;questions&amp;#39;&lt;/span&gt;]&lt;/span&gt;
&lt;span id=&#34;cb2-15&#34;&gt;&lt;a href=&#34;#cb2-15&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                          &lt;span class=&#34;cf&#34;&gt;if&lt;/span&gt; i[&lt;span class=&#34;st&#34;&gt;&amp;#39;topic&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;==&lt;/span&gt; topic)&lt;/span&gt;
&lt;span id=&#34;cb2-16&#34;&gt;&lt;a href=&#34;#cb2-16&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;            &lt;span class=&#34;cf&#34;&gt;for&lt;/span&gt; response &lt;span class=&#34;kw&#34;&gt;in&lt;/span&gt; subpop[&lt;span class=&#34;st&#34;&gt;&amp;#39;responses&amp;#39;&lt;/span&gt;]:&lt;/span&gt;
&lt;span id=&#34;cb2-17&#34;&gt;&lt;a href=&#34;#cb2-17&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                &lt;span class=&#34;cf&#34;&gt;if&lt;/span&gt; response[&lt;span class=&#34;st&#34;&gt;&amp;#39;first_name&amp;#39;&lt;/span&gt;]:&lt;/span&gt;
&lt;span id=&#34;cb2-18&#34;&gt;&lt;a href=&#34;#cb2-18&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                    &lt;span class=&#34;cf&#34;&gt;yield&lt;/span&gt; {&lt;span class=&#34;st&#34;&gt;&amp;#39;poll&amp;#39;&lt;/span&gt;: poll[&lt;span class=&#34;st&#34;&gt;&amp;#39;id&amp;#39;&lt;/span&gt;],&lt;/span&gt;
&lt;span id=&#34;cb2-19&#34;&gt;&lt;a href=&#34;#cb2-19&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                           &lt;span class=&#34;st&#34;&gt;&amp;#39;date&amp;#39;&lt;/span&gt;: poll[&lt;span class=&#34;st&#34;&gt;&amp;#39;end_date&amp;#39;&lt;/span&gt;],&lt;/span&gt;
&lt;span id=&#34;cb2-20&#34;&gt;&lt;a href=&#34;#cb2-20&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                           &lt;span class=&#34;st&#34;&gt;&amp;#39;filter&amp;#39;&lt;/span&gt;: subpop[&lt;span class=&#34;st&#34;&gt;&amp;#39;name&amp;#39;&lt;/span&gt;].lower(),&lt;/span&gt;
&lt;span id=&#34;cb2-21&#34;&gt;&lt;a href=&#34;#cb2-21&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                           &lt;span class=&#34;st&#34;&gt;&amp;#39;obs&amp;#39;&lt;/span&gt;: subpop[&lt;span class=&#34;st&#34;&gt;&amp;#39;observations&amp;#39;&lt;/span&gt;],&lt;/span&gt;
&lt;span id=&#34;cb2-22&#34;&gt;&lt;a href=&#34;#cb2-22&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                           &lt;span class=&#34;st&#34;&gt;&amp;#39;candidate&amp;#39;&lt;/span&gt;: &lt;span class=&#34;st&#34;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&#34;sc&#34;&gt;{}&lt;/span&gt;&lt;span class=&#34;st&#34;&gt; &lt;/span&gt;&lt;span class=&#34;sc&#34;&gt;{}&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;&lt;/span&gt;.&lt;span class=&#34;bu&#34;&gt;format&lt;/span&gt;(response[&lt;span class=&#34;st&#34;&gt;&amp;#39;first_name&amp;#39;&lt;/span&gt;],&lt;/span&gt;
&lt;span id=&#34;cb2-23&#34;&gt;&lt;a href=&#34;#cb2-23&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                                                       response[&lt;span class=&#34;st&#34;&gt;&amp;#39;last_name&amp;#39;&lt;/span&gt;]),&lt;/span&gt;
&lt;span id=&#34;cb2-24&#34;&gt;&lt;a href=&#34;#cb2-24&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                           &lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;: response[&lt;span class=&#34;st&#34;&gt;&amp;#39;value&amp;#39;&lt;/span&gt;]}&lt;/span&gt;
&lt;span id=&#34;cb2-25&#34;&gt;&lt;a href=&#34;#cb2-25&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;/span&gt;
&lt;span id=&#34;cb2-26&#34;&gt;&lt;a href=&#34;#cb2-26&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        &lt;span class=&#34;cf&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;bu&#34;&gt;len&lt;/span&gt;(page_results) &lt;span class=&#34;op&#34;&gt;&amp;lt;&lt;/span&gt; &lt;span class=&#34;dv&#34;&gt;10&lt;/span&gt;:&lt;/span&gt;
&lt;span id=&#34;cb2-27&#34;&gt;&lt;a href=&#34;#cb2-27&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;            &lt;span class=&#34;cf&#34;&gt;break&lt;/span&gt;&lt;/span&gt;
&lt;span id=&#34;cb2-28&#34;&gt;&lt;a href=&#34;#cb2-28&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        page &lt;span class=&#34;op&#34;&gt;+=&lt;/span&gt; &lt;span class=&#34;dv&#34;&gt;1&lt;/span&gt;&lt;/span&gt;
&lt;span id=&#34;cb2-29&#34;&gt;&lt;a href=&#34;#cb2-29&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;/span&gt;
&lt;span id=&#34;cb2-30&#34;&gt;&lt;a href=&#34;#cb2-30&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;kw&#34;&gt;def&lt;/span&gt; get_polls(state&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;US&amp;#39;&lt;/span&gt;, party&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;gop&amp;#39;&lt;/span&gt;, start_date&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;2015-6-1&amp;#39;&lt;/span&gt;):&lt;/span&gt;
&lt;span id=&#34;cb2-31&#34;&gt;&lt;a href=&#34;#cb2-31&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    polls &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; pd.DataFrame(get_all_results(state&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;state,&lt;/span&gt;
&lt;span id=&#34;cb2-32&#34;&gt;&lt;a href=&#34;#cb2-32&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                                         party&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;party,&lt;/span&gt;
&lt;span id=&#34;cb2-33&#34;&gt;&lt;a href=&#34;#cb2-33&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;                                         start_date&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;start_date))&lt;/span&gt;
&lt;span id=&#34;cb2-34&#34;&gt;&lt;a href=&#34;#cb2-34&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;date&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; pd.to_datetime(polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;date&amp;#39;&lt;/span&gt;])&lt;/span&gt;
&lt;span id=&#34;cb2-35&#34;&gt;&lt;a href=&#34;#cb2-35&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;cf&#34;&gt;return&lt;/span&gt; polls&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Those two functions will allow us to specify a state, a party (either
&lt;q&gt;gop&lt;/q&gt; or &lt;q&gt;dem&lt;/q&gt;, per Pollster), and how far back in time we
want to go. What we get back is a Pandas DataFrame where each row has
the result for one candidate for one poll, plus some metadata about the
poll. All we really need for this model is the date, result, and number
of observations, but I’ve also included the population screen in case
you want to, say, restrict to only likely voters.&lt;/p&gt;
&lt;p&gt;Now we want to combine those poll results for a point-in-time
estimate of the mean, plus a standard deviation of the estimate. But not
all polls are equally good; we’ll want to be able to weight them
somehow.&lt;/p&gt;
&lt;p&gt;We’re going to be lazy, and just weight based on recency. For each
poll, we’ll set a weight of one over the square of the age of the poll
plus one (the plus one is so that we don’t divide by zero). Then we can
create a super-poll, in which we pool all the folks who said they’d vote
for each candidate in any poll, multiplied by the weight of that poll.
This allows us to calculate both the weighted estimate of the mean and
the standard deviation of the estimate:&lt;/p&gt;
&lt;div class=&#34;sourceCode&#34; id=&#34;cb3&#34;&gt;&lt;pre
class=&#34;sourceCode python&#34;&gt;&lt;code class=&#34;sourceCode python&#34;&gt;&lt;span id=&#34;cb3-1&#34;&gt;&lt;a href=&#34;#cb3-1&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;kw&#34;&gt;def&lt;/span&gt; get_distribution_for_date(polls, target_date&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;va&#34;&gt;None&lt;/span&gt;, window&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;dv&#34;&gt;30&lt;/span&gt;):&lt;/span&gt;
&lt;span id=&#34;cb3-2&#34;&gt;&lt;a href=&#34;#cb3-2&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;cf&#34;&gt;if&lt;/span&gt; target_date &lt;span class=&#34;kw&#34;&gt;is&lt;/span&gt; &lt;span class=&#34;va&#34;&gt;None&lt;/span&gt;:&lt;/span&gt;
&lt;span id=&#34;cb3-3&#34;&gt;&lt;a href=&#34;#cb3-3&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        target_date &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; datetime.datetime.today()&lt;/span&gt;
&lt;span id=&#34;cb3-4&#34;&gt;&lt;a href=&#34;#cb3-4&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    polls &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; polls[&lt;/span&gt;
&lt;span id=&#34;cb3-5&#34;&gt;&lt;a href=&#34;#cb3-5&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        (polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;date&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;&amp;lt;=&lt;/span&gt; target_date)&lt;/span&gt;
&lt;span id=&#34;cb3-6&#34;&gt;&lt;a href=&#34;#cb3-6&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        &lt;span class=&#34;op&#34;&gt;&amp;amp;&lt;/span&gt; (polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;date&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;&amp;gt;&lt;/span&gt; target_date &lt;span class=&#34;op&#34;&gt;-&lt;/span&gt; datetime.timedelta(window))&lt;/span&gt;
&lt;span id=&#34;cb3-7&#34;&gt;&lt;a href=&#34;#cb3-7&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    ]&lt;/span&gt;
&lt;span id=&#34;cb3-8&#34;&gt;&lt;a href=&#34;#cb3-8&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    weights &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;dv&#34;&gt;1&lt;/span&gt; &lt;span class=&#34;op&#34;&gt;/&lt;/span&gt; np.square(&lt;/span&gt;
&lt;span id=&#34;cb3-9&#34;&gt;&lt;a href=&#34;#cb3-9&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        (target_date &lt;span class=&#34;op&#34;&gt;-&lt;/span&gt; polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;date&amp;#39;&lt;/span&gt;]) &lt;span class=&#34;op&#34;&gt;/&lt;/span&gt;np.timedelta64(&lt;span class=&#34;dv&#34;&gt;1&lt;/span&gt;, &lt;span class=&#34;st&#34;&gt;&amp;#39;D&amp;#39;&lt;/span&gt;) &lt;span class=&#34;op&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;dv&#34;&gt;1&lt;/span&gt;)&lt;/span&gt;
&lt;span id=&#34;cb3-10&#34;&gt;&lt;a href=&#34;#cb3-10&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    weighted &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; polls[[&lt;span class=&#34;st&#34;&gt;&amp;#39;candidate&amp;#39;&lt;/span&gt;]].copy()&lt;/span&gt;
&lt;span id=&#34;cb3-11&#34;&gt;&lt;a href=&#34;#cb3-11&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;n&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; weights &lt;span class=&#34;op&#34;&gt;*&lt;/span&gt; polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;obs&amp;#39;&lt;/span&gt;]&lt;/span&gt;
&lt;span id=&#34;cb3-12&#34;&gt;&lt;a href=&#34;#cb3-12&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;votes&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;/&lt;/span&gt; &lt;span class=&#34;dv&#34;&gt;100&lt;/span&gt; &lt;span class=&#34;op&#34;&gt;*&lt;/span&gt; polls[&lt;span class=&#34;st&#34;&gt;&amp;#39;obs&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;*&lt;/span&gt; weights&lt;/span&gt;
&lt;span id=&#34;cb3-13&#34;&gt;&lt;a href=&#34;#cb3-13&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    weighted &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; weighted.groupby(&lt;span class=&#34;st&#34;&gt;&amp;#39;candidate&amp;#39;&lt;/span&gt;).&lt;span class=&#34;bu&#34;&gt;sum&lt;/span&gt;()&lt;/span&gt;
&lt;span id=&#34;cb3-14&#34;&gt;&lt;a href=&#34;#cb3-14&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;votes&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;/&lt;/span&gt; weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;n&amp;#39;&lt;/span&gt;]&lt;/span&gt;
&lt;span id=&#34;cb3-15&#34;&gt;&lt;a href=&#34;#cb3-15&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;std&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; np.sqrt(&lt;/span&gt;
&lt;span id=&#34;cb3-16&#34;&gt;&lt;a href=&#34;#cb3-16&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        (weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;] &lt;span class=&#34;op&#34;&gt;*&lt;/span&gt; (&lt;span class=&#34;dv&#34;&gt;1&lt;/span&gt; &lt;span class=&#34;op&#34;&gt;-&lt;/span&gt; weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;])) &lt;span class=&#34;op&#34;&gt;/&lt;/span&gt; weighted[&lt;span class=&#34;st&#34;&gt;&amp;#39;n&amp;#39;&lt;/span&gt;])&lt;/span&gt;
&lt;span id=&#34;cb3-17&#34;&gt;&lt;a href=&#34;#cb3-17&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;cf&#34;&gt;return&lt;/span&gt; weighted[[&lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;, &lt;span class=&#34;st&#34;&gt;&amp;#39;std&amp;#39;&lt;/span&gt;]].query(&lt;span class=&#34;st&#34;&gt;&amp;#39;mean &amp;gt; 0&amp;#39;&lt;/span&gt;).copy()&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;This function allows us to specify a target date, in case we want a
snapshot from earlier in the campaign, and also a window that gives us a
maximum age of polls. That’s so Scott Walker doesn’t show up in our
results even though he’s already dropped out of the race.&lt;/p&gt;
&lt;p&gt;Now we can run simulations! All we have to do is a draw from the
normal distribution for each candidate, and see who got the highest
percent of the vote:&lt;/p&gt;
&lt;div class=&#34;sourceCode&#34; id=&#34;cb4&#34;&gt;&lt;pre
class=&#34;sourceCode python&#34;&gt;&lt;code class=&#34;sourceCode python&#34;&gt;&lt;span id=&#34;cb4-1&#34;&gt;&lt;a href=&#34;#cb4-1&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;kw&#34;&gt;def&lt;/span&gt; run_simulation(dists, trials&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;dv&#34;&gt;10000&lt;/span&gt;):&lt;/span&gt;
&lt;span id=&#34;cb4-2&#34;&gt;&lt;a href=&#34;#cb4-2&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    runs &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; pd.DataFrame(&lt;/span&gt;
&lt;span id=&#34;cb4-3&#34;&gt;&lt;a href=&#34;#cb4-3&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        [np.random.normal(dists[&lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;], dists[&lt;span class=&#34;st&#34;&gt;&amp;#39;std&amp;#39;&lt;/span&gt;])&lt;/span&gt;
&lt;span id=&#34;cb4-4&#34;&gt;&lt;a href=&#34;#cb4-4&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;         &lt;span class=&#34;cf&#34;&gt;for&lt;/span&gt; i &lt;span class=&#34;kw&#34;&gt;in&lt;/span&gt; &lt;span class=&#34;bu&#34;&gt;range&lt;/span&gt;(trials)],&lt;/span&gt;
&lt;span id=&#34;cb4-5&#34;&gt;&lt;a href=&#34;#cb4-5&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        columns&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;dists.index)&lt;/span&gt;
&lt;span id=&#34;cb4-6&#34;&gt;&lt;a href=&#34;#cb4-6&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    results &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; pd.Series(collections.Counter(runs.T.idxmax()))&lt;/span&gt;
&lt;span id=&#34;cb4-7&#34;&gt;&lt;a href=&#34;#cb4-7&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;cf&#34;&gt;return&lt;/span&gt; results &lt;span class=&#34;op&#34;&gt;/&lt;/span&gt; results.&lt;span class=&#34;bu&#34;&gt;sum&lt;/span&gt;()&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Finally, here’s a little function to automate all the steps and print
out results:&lt;/p&gt;
&lt;div class=&#34;sourceCode&#34; id=&#34;cb5&#34;&gt;&lt;pre
class=&#34;sourceCode python&#34;&gt;&lt;code class=&#34;sourceCode python&#34;&gt;&lt;span id=&#34;cb5-1&#34;&gt;&lt;a href=&#34;#cb5-1&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;&lt;span class=&#34;kw&#34;&gt;def&lt;/span&gt; predict(state&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;us&amp;#39;&lt;/span&gt;, party&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;gop&amp;#39;&lt;/span&gt;, window&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;dv&#34;&gt;30&lt;/span&gt;, trials&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;dv&#34;&gt;10000&lt;/span&gt;,&lt;/span&gt;
&lt;span id=&#34;cb5-2&#34;&gt;&lt;a href=&#34;#cb5-2&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;            target_date&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;va&#34;&gt;None&lt;/span&gt;):&lt;/span&gt;
&lt;span id=&#34;cb5-3&#34;&gt;&lt;a href=&#34;#cb5-3&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    polls &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; get_polls(state&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;state, party&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;party)&lt;/span&gt;
&lt;span id=&#34;cb5-4&#34;&gt;&lt;a href=&#34;#cb5-4&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    dists &lt;span class=&#34;op&#34;&gt;=&lt;/span&gt; get_distribution_for_date(&lt;/span&gt;
&lt;span id=&#34;cb5-5&#34;&gt;&lt;a href=&#34;#cb5-5&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;        polls, window&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;window,target_date&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;target_date)&lt;/span&gt;
&lt;span id=&#34;cb5-6&#34;&gt;&lt;a href=&#34;#cb5-6&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;bu&#34;&gt;print&lt;/span&gt;(&lt;span class=&#34;st&#34;&gt;&amp;#39;Superpoll Results:&amp;#39;&lt;/span&gt;)&lt;/span&gt;
&lt;span id=&#34;cb5-7&#34;&gt;&lt;a href=&#34;#cb5-7&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;bu&#34;&gt;print&lt;/span&gt;(dists.sort_values(&lt;span class=&#34;st&#34;&gt;&amp;#39;mean&amp;#39;&lt;/span&gt;, ascending&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;va&#34;&gt;False&lt;/span&gt;)&lt;/span&gt;
&lt;span id=&#34;cb5-8&#34;&gt;&lt;a href=&#34;#cb5-8&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;          .applymap(&lt;span class=&#34;kw&#34;&gt;lambda&lt;/span&gt; x: &lt;span class=&#34;st&#34;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&#34;sc&#34;&gt;{:.1%}&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;&lt;/span&gt;.&lt;span class=&#34;bu&#34;&gt;format&lt;/span&gt;(x)))&lt;/span&gt;
&lt;span id=&#34;cb5-9&#34;&gt;&lt;a href=&#34;#cb5-9&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;bu&#34;&gt;print&lt;/span&gt;()&lt;/span&gt;
&lt;span id=&#34;cb5-10&#34;&gt;&lt;a href=&#34;#cb5-10&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;bu&#34;&gt;print&lt;/span&gt;(&lt;span class=&#34;st&#34;&gt;&amp;#39;Simulation Results:&amp;#39;&lt;/span&gt;)&lt;/span&gt;
&lt;span id=&#34;cb5-11&#34;&gt;&lt;a href=&#34;#cb5-11&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;    &lt;span class=&#34;bu&#34;&gt;print&lt;/span&gt;(run_simulation(dists, trials&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;trials)&lt;/span&gt;
&lt;span id=&#34;cb5-12&#34;&gt;&lt;a href=&#34;#cb5-12&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;          .sort_values(ascending&lt;span class=&#34;op&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;va&#34;&gt;False&lt;/span&gt;)&lt;/span&gt;
&lt;span id=&#34;cb5-13&#34;&gt;&lt;a href=&#34;#cb5-13&#34; aria-hidden=&#34;true&#34; tabindex=&#34;-1&#34;&gt;&lt;/a&gt;          .&lt;span class=&#34;bu&#34;&gt;map&lt;/span&gt;(&lt;span class=&#34;kw&#34;&gt;lambda&lt;/span&gt; x: &lt;span class=&#34;st&#34;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&#34;sc&#34;&gt;{:.1%}&lt;/span&gt;&lt;span class=&#34;st&#34;&gt;&amp;#39;&lt;/span&gt;.&lt;span class=&#34;bu&#34;&gt;format&lt;/span&gt;(x)))&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;So, now for the fun part, who wins? Here’s the superpoll results on
the Republican side:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align: left;&#34;&gt;Candidate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Estimate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Standard Deviation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Donald Trump&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;28.2%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Ted Cruz&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;23.6%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Marco Rubio&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;17.4%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Ben Carson&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;7.6%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Rand Paul&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;4.7%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Jeb Bush&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;4.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Mike Huckabee&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;3.3%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;John Kasich&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.8%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Carly Fiorina&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.4%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Chris Christie&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Rick Santorum&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.3%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Jim Gilmore&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;In the simulation, Donald Trump won 96 percent of the time, while Ted
Cruz won 4 percent.&lt;/p&gt;
&lt;p&gt;Here’s our superpoll results for the Dems:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align: left;&#34;&gt;Candidate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Estimate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Standard Deviation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Hillary Clinton&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;47.4%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.4%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Bernie Sanders&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;46.0%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.4%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Martin O’Malley&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;3.6%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Seems pretty close, but bad news, Bernie fans! In the simulation,
Hillary won 66 percent of the time, while Sanders only won 34
percent.&lt;/p&gt;
&lt;p&gt;Now, that’s all with a 30-day window. What if we keep it to just the
most recent polls?&lt;/p&gt;
&lt;p&gt;Here’s what we get with a 4-day window on the GOP side:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align: left;&#34;&gt;Candidate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Estimate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Standard Deviation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Donald Trump&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;27.5%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Ted Cruz&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;23.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Marco Rubio&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;18.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Ben Carson&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;7.5%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Rand Paul&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;5.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Jeb Bush&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;4.1%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Mike Huckabee&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;3.5%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;John Kasich&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.8%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Carly Fiorina&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.5%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Chris Christie&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.0%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Rick Santorum&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.3%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;0.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Trump still wins 93.6 percent of simulations, but Cruz is up to 6.4
percent, and Rubio is on the board, although with 0.0%.&lt;/p&gt;
&lt;p&gt;On the Democratic side, things get really interesting:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align: left;&#34;&gt;Candidate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Estimate&lt;/th&gt;
&lt;th style=&#34;text-align: right;&#34;&gt;Standard Deviation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Hillary Clinton&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;47.0%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Bernie Sanders&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;46.9%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;2.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align: left;&#34;&gt;Martin O’Malley&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;3.2%&lt;/td&gt;
&lt;td style=&#34;text-align: right;&#34;&gt;1.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;In the simulation, O’Malley stuns! Just kidding, Clinton wins 51
percent of the time, and Sanders wins 49 percent. Boy is that going to
be fun.&lt;/p&gt;
&lt;p&gt;Obviously we shouldn’t bet the house on these predictions. My
weighting model may be wrong (read: is wrong) or the polls themselves
may be wrong (read: are completely unreliable in recent elections). But
this shows you how simple it really is to get something like this off
the ground.&lt;/p&gt;
&lt;p&gt;If you decide to play around with this model (again, you can download
the Jupyter notebook &lt;a
href=&#34;http://nbviewer.jupyter.org/github/Oliversherouse/blog_notebooks/blob/master/430%20Model.ipynb&#34;&gt;here&lt;/a&gt;),
be sure to let me know on &lt;a
href=&#34;http://twitter.com/OliverSherouse&#34;&gt;twitter&lt;/a&gt;. It would be a lot
of fun to see what people come up with.&lt;/p&gt;
</content>
    
  </entry>
  
</feed>