🌲 Secondary sources are pretty great, actually

On the difficulty of learning from first principles, and how to pick what information to rely on

Conventional wisdom says that primary sources — which is to say, first-hand accounts, genuine artifacts of the ‘physical evidence’ type, and hard data from original research — are ‘best.’ Frankly, this is a great example of a time when I disagree with conventional wisdom.

Secondary sources are fantastic at distilling information, and while it is often useful to go back and confirm that a secondary source is in fact accurately representing a primary source, the fact of the matter is… a lot of the times laymen (and everyone is a layman in at least one field) do not have the expertise to evaluate that. Sometimes, the best way to confirm the reliability secondary source is to just read a lot of other secondary sources.

For a variety of reasons, teachers in a schoolhouse setting often have a hard time teaching students how to evaluate the reliability of sources, especially the modern stuff. This should not surprise anyone; teaching critical thinking is hard and so is teaching notetaking. One reason source evaluation is tricky is that teachers must navigate the political waters of their district and parents can get pretty angry if teachers criticize their preferred news outlet. “Herotodus was the first historian but you can’t trust everything” is easy. “The Atlantic doesn’t typically outright lie but does occasionally shade the truth” can get you into trouble. Another is because we’re required to teach fine-grained distinctions between primary, secondary, and tertiary sources … but due to time and cognitive constraints it gets oversimplified down to “primary sources are best!” even though primary sources are harder to interpret without context. A third reason is because students doing formal research projects are highly incentivized to “play it safe” by sticking to obviously reliable sources and therefore get very few chances to learn by experimenting with edge cases, because then they might “fail” and failure has consequences; lost time, lost prestige, and shame, among others.

But where does that leave folks after graduation and they’re no longer required to write research papers for a grade?

Most people lack sufficient context for quality interpretation

Back when I was still teaching middle school, in my district there was a shift from teaching kids how to pass multiple choice tests to grading them on how well they analyzed provided documents (a la the AP history exam’s document based question). Leaving aside for now whether it’s appropriate to train sixth graders how to take a test meant to demonstrate that advanced high schoolers had completed college-level material, the transition was a pretty wild ride.

Sixth grade students in my district tend to have a very shaky grasp of history — most of them only get an inconsistent smattering of history and science in elementary school. So the stuff they come up with, without the sufficient background context to understand what they're actually reading, is wild speculation. As you might imagine, an eleven-year-old, given a short excerpt from Herodotus, tends to misinterpret the situation.

The damsels sent by the Hyperboreans died in Delos; and in their honour all the Delian girls and youths are wont to cut off their hair. The girls, before their marriage-day, cut off a curl, and twining it round a distaff, lay it upon the grave of the strangers. This grave is on the left as one enters the precinct of Diana, and has an olive-tree growing on it. The youths wind some of their hair round a kind of grass, and, like the girls, place it upon the tomb. Such are the honours paid to these damsels by the Delians.

Your average adult barely knows what a ‘distaff’ is, or what ‘precinct’ means in this context. They’d probably be lucky to be able to identify which ancient civilization the Delians belonged to. Kids who mostly haven’t even read Percy Jackson yet are not going to put “Diana” and “olives” and get “written by a Greek guy who lived under the Persians.” But more importantly, without broader context, it’s really easy to over-index on tiny details — given this quote, a lot of my students would have assumed Delos was a plains culture, not an island one.

If you give somebody a primary source that is a snippet of text and expect them to come up with something that makes sense, instead of a tertiary source that has written to educate and explain, they’re going to have a rough time.

Yet everything I’ve seen in education is structured to push for primary sources as being ‘more reliable.’ It drives me nuts.

Here’s another example, this time from Mesopotamia.

Let’s say I am looking at a primary source from Mesopotamian history, an artifact. I am basically looking at a sculpture, possibly with some cuneiform writing on it. Since I only speak English and a tiny smattering of Spanish, to actually engage with that source, I would first need to learn how to read cuneiform like what was used in ancient Mesopotamia. That’s — I don’t know, let’s say a year of my life to just learn the language, and then to learn what the words mean I also need to understand what was going on in the context they were written — and to figure that out by relying on first principles means I am somehow supposed to make all of these inferences independently.

Even if it's not taken out of context, even if it is the entire entirety of Hammurabi’s Code, students lack the context — heck, most lack the life experience — to comprehend its impact, its value, its meaning and its implications, without secondary and tertiary sources — like, at the end of the day, a teacher’s explanation… to help guide them. One of the most infuriating parts of teaching DBQs to sixth graders was being required to give a good grade to any “plausible and well-defended answer” and knowing that my students were often walking out of class with wildly inaccurate understandings of historical events because of how hard we were being pushed to let them draw their own conclusions.

I suppose I could provide them all the journal articles that come directly from academics doing archeological digs, since some definitions (and don’t get me started on how oversimplified source categorization is at the secondary school level) consider those primary sources.

But I think it is a lot faster and more efficient — and only lose a tiny bit of “truth” — to read a well-written substack article series, or attend an Oxford lecture series, which summarizes useful information well enough for someone to actually engage with the Code itself… without losing five years (at best!) to getting a PhD.

What about modern stuff that matters?

Now let’s take American laws for an example. The internet is rife with arguments about whether something is illegal or not, but what not everybody realizes because the plain text of the law does not usually mean what a normal person would naturally think it means.

Once, when I was still a teenager, I had to decide where I wanted to go to law school. The dean of the one I had my eye on gave an introductory welcome lecture thing, and during the Q&A session afterward, a parent asked what major was best for law school. The dean surprised me at first: he said it wasn't political science, it wasn't philosophy.

It was wait for it,

wait for it,

linguistics. Or take a language like Spanish or whatever.

His opinion was that learning American law is basically like learning a second language — because not only is there so much jargon, there’s also jargon that isn't jargony because it's some crazy technical term (lookin’ at you, deodand)… it's jargon because of hundreds of years of common law twisting the meaning of a word.

Negligence has a very specific definition that is not obvious on the face of it, because it has been defined through years and years and years of evolution of case law. And you know what? If you try to figure it out just reading case law on your own, it will take you years to understand what the law is in practice well enough to even mostly predict what things really truly are illegal on the margins.

Textbooks — which I tend to think of as tertiary sources — are fine for an overview, but if you want to genuinely understand the nuances, you’re probably gonna need to read a bunch of articles by people arguing about it. I have certainly gotten a better understanding of newsletter host content policies reading articles about the recent brouhaha than from reading the plain language of the Substack and Beehiiv and Ghost terms of service!

What to do?

Let’s say you’re taking your first steps into the wide world of learning stuff independently, and are being inundated with all sorts of “fake news” and propaganda, questionably accurate pop-sci books and outright lies, clickbait headlines and the inevitable result of memetic games of telephone played out on social media. How do you find sources you can trust? How can you evaluate information without devoting your entire life to becoming an expert in every subject you have any interest in whatsoever?

To keep beating this drum — sometimes you need secondary and tertiary sources. I would argue that starting with Wikipedia is actually the most logical and sensible starting place, despite the average educator’s habit of declaring Wikipedia absolutely verboten. It’s to the point where I’ve had students straight-faced tell me Wikipedia is obviously worse than whatever they get from the first result on Google. Wikipedia, however, offers a decent overview of the state of play — which allows you to decide how to spend your time and whether to chase it all the way down to primary sources. I don’t agree with everything in Expecting Better by

Emily Oster

but it’s a great starting point for finding out where to even begin looking into what the major controversies are with regards to pregnancy & parenting research.

A lot of smart folks recommend writing and notetaking and learning in public because it’s easier to keep yourself honest when you’re doing stuff in front of people whose opinions you care about. It is immensely helpful when readers point out inconsistencies in your logic or related information you weren’t already aware of (thank you!). It’s the same reason that it’s a good idea to sign up for a couple of competitions if you’re learning ballroom dance; deadlines and social pressure are better at forcing you to improve than vague goals.

But since nobody likes embarrassing themselves unnecessarily, here’s my advice for evaluating the reliability of stuff you read on the internet.

Does the reference cite its sources?

My biggest recommendation is to look for media (articles, videos, whatever) that cite their sources, or are at least willing to do so when asked. Plenty of accessible writing references more academic texts and even if you have difficulty parsing them, the fact that they’re actually basing their information off of real scholasticism is a good (although not sure-fire) sign.

My pop history books are liberally annotated with things like “ok WHICH scholars say that? Don’t just say scholars says!” and “ORLY” and “double-check.” I don’t think this makes them bad books, it just means that it is a book, not a research paper. It would render the book unreadable to cite every claim even if you didn’t count how much it would increase the wordcount—the book would balloon in width way past what bookbinders can handle. Technically this is speculation, but I’m basing it off of how liberally cited my education research papers were during my master’s program, and how often I have to wade through lengthy inline citations when I read journal articles. Most people don’t enjoy having to skip a third of the line for every sentence.

For that matter, I analyze pop books differently than I analyze pop articles. Pretty often pop articles don’t bother with citations and references at all, so it’s an easy top-level way to separate the stuff worth reading from the crap that’s not even worth my time. The rest of the time, they’ll link to the original study and I’ll try to read that instead. That’s what we mean by “primary sources are more reliable” — a lot of science journalists have a shaky understanding of what reports are actually saying, or go out of their way to sensationalize things. This is particularly bad in nutrition science reporting, I’ve noticed. It is also very true for politics, which is why I like to read actual proposed laws myself.

The danger, of course, is that often you will lack the expertise to correctly analyze such a primary source. Non-experts lack context. I’m a critical consumer of historian-created media, not a historian myself—despite teaching history for years, the credentialing process is a lot more focused on the teaching pedagogy than my actual knowledge of history, which was tested with… literally one test that I hope an average high-schooler could have passed.

Are the sources reliable?

You can usually get a sense of how reliable a pop science book (or news article, or tweet, or whatever) is by looking at the size and reputation of the publisher. Big 5 Publishing Houses and major universities can’t always be relied on (they will sometimes ‘chase that dollar’ or let a mistake slip through) but you’re less likely to publicly embarrass yourself citing a book put out by Harvard University Press than a small publisher known for pushing, say, holocaust denialism.

Media sources can often be evaluated on the basis of what their incentives are. The replication crisis uncovered in the early 2010s was not usually the result of bad-faith liars, but it is still a pretty big problem. Journals aren’t typically punished for publishing bad papers, universities sometimes outright refuse to investigate blatant frauds, and academics are pressured to publish regardless of publication quality. Journalists are incentivized to publish sensational things. Propaganda outlets are trying to get people to buy into a particular worldview, for whatever reason. This doesn’t automatically mean they’re lying or that the information is wrong (a man going into space can be sensational and true), but it does raise the bar for a double-check.

This is one reason I tend to stick to learning about relatively obscure eras of history & neat animal facts. Reasonable people might disagree about how horseback riding got invented, but it’s usually pretty easy to figure out what perspective my sources are coming from. At a certain point, you just have to decide whether the author is acting in good faith and then take responsibility for confirming any claims that you are planning to make.

Don’t repeat claims you haven’t confirmed

If someone makes a claim that you feel inclined to repeat, triple-check it. There aren’t enough man-hours to confirm everything you read, whether it matches your pre-existing biases (and everybody has them, they’re a natural consequence of having a knowledge base) or not. But if you’re going to repeat a claim or integrate it into your worldview in such a way that you might reference it in a discussion, it behooves you to be able to explain with dignity where you learned that little tidbit, and there are only some cases where a good response is “well I read it from some guy in a Reddit thread.”

That said, just because a claim was published in a respected textbook that was recommended by an actual historian doesn’t mean that every other historian agrees with it. Especially if the source is older, go see if there have been any updates in the literature. Just because multiple sources repeat a claim doesn’t mean you’ve confirmed it, by the way — kinda the opposite. If every source you look at repeats the exact same claim in the exact same way, chances are you’ve run afoul of some kind of SEO article farm that’s just regurgitating stuff from Wikipedia or random other articles. More on this later.

Can you track the claim back to its origin?

I recommend trying to read widely, because it will help hone your instincts. If somebody makes a claim that doesn’t “feel” right, track it down. But even if you have no context to evaluate the truth of a particular claim, you can evaluate how broad the claim is. Extremely broad claims like “every single culture has a 10% rate of mothers lying about paternity” are almost certainly false.

Find out what other people say. This is good practice anyway, because even people who aren’t trying to peddle bigotry and nonsense are often wrong because of perfectly normal biases that aren’t, like, awful. Robert Heinlein once said, “if everyone knows thus-and-such, then it ain’t true, but at least a thousand to one” but if all the comments on a Reddit thread are pointing out why the original post is incorrect and they’re all providing links to journal articles that seem to accurately support their claims (double-check this, because sometimes people will post links to “support” that actually directly contradicts their claims!) — well, maybe there’s something to what they’re saying.

Also, pay attention to who created the source. Google them. Are they a professor at a respected institution? Are they at the center of any Twitter controversies (this doesn’t immediately disqualify them from being reliable, but it should give you a sense of their positions)? What does their “About” page say; is it a satire account or a widely published Yale professor? What does their purpose for writing seem to be? Do they come across as someone who loves a particular time period, or do they seem to be really interested in making a polarizing point (which again doesn’t automatically mean they’re wrong, but it is a factor to consider). For example, if a particular article is dead set on talking about Vikings in a particular way… is that person writing a whole bunch of articles making the same point, and really hyper focused on that point, or are the rest of their articles looking at other facets of Viking life that don’t make that point?

Who agrees with them? Are the people agreeing with them engaging in something that feels academically or intellectually rigorous, or are they just happy to have “someone who sounds smart” agreeing with their pre-existing biases? Do they generally seem willing to engage with alternative viewpoints and admit they are wrong?

In Practice

So, let’s use an example as a practice exercise.

Pretend I’m looking for information about when waterwheels were invented. I’m going to start by googling it, because I’m not writing a dissertation or anything I’m just looking to learn something about a history thing I found interesting after having visited the Netherlands on vacation.

I’m not a Medievalist or anything, and I’m definitely not an expert on whatever you need to be an expert on for waterwheels to fall under your purview, so let’s say I get hits for 3 books and 4 websites. I skip the books for now because I’m looking for an immediate answer to a simple question.

One article is from what looks like an encyclopedia website with a bunch of articles, but, drat! It’s paywalled because I already read 4 great articles from there. I skip it for now — although if I hit this paywall often enough, I’ll buy a membership.

The next article doesn’t have academic references, but it has a couple of hyperlinks to what look like sources and seems to be made by a hobbyist. Paydirt!

The third article is from a website trying to sell landscaping rocks that used to be millstones, so it’s definitely not academic and they’re definitely trying to convince me to spend money, so I know that whatever they’re claiming, I’m going to need to double-check, but I still read it because maybe I’ll get some useful search terms out of it.

The 4th article reads like a ranty screed trying to convince me that a particular ethnic group is the best because they invented waterwheels and a bunch of other early technologies. I stop reading after the third paragraph and make sure to click back as fast as possible, so Google knows it was a crappy result it should de-prioritize in SEO.

Then, because I’m interested now, I go back to the books. First, I look at the cover. Does it look accessible, or does it look like somebody’s dissertation? I like to think I’m pretty smart and a fast reder, but dissertations are long, filled with a bunch of stuff I don’t usually care about, and not usually written to a layman audience even when they’re about something interesting. I skip the one that looks like it’s intended for a super dry academic audience. (Note: I do sometimes read dissertations — The Life Cycle of Disability in Ancient Greece by Deborah Sneed was pretty good, and a buddy of mine wrote a really interesting one about international copyright.)

The next one looks interesting and is written at my level and is even filled with citations, but the reviews are a mix of creepy and scathing. I leave it off my wishlist.

The third one looks more promising. The reviews seem enthusiastic about how this book addresses new archaeological finds, it’s making a claim I haven’t heard before, and it’s been published by a reputable publisher. The sample pages appear heavily footnoted. It even has a neat cover! I put it on my wishlist and get it for Christmas and start reading it.

But whoa, some of the claims it’s making are not things I’ve ever heard before. The author addresses the previous state of scholarship and even sounds convincing about why they’re right and their predecessors are wrong. (I know of no new scholarship about waterwheels, but this sort of thing does happen — The Amazons by Adrienne Mayor and The Golden Thread by Kassia St. Claire are good examples).

So now what do I do?

First, I make sure to read critically. I annotate heavily, making liberal use of the ORLY owl and notes-to-self to follow up on because honestly, even the absolute best, most well-considered, most thoughtful and well-intentioned books is going to say stuff that’s outside of the expertise of the author.

My husband once listened to a podcast that goes super into depth about I think WWII (I don’t recall, but it was definitely an American war) and the guy was clearly an expert on that war. I believe it was recommended by

Lawfare

but I don’t remember the details. Regardless, he made some throwaway comment about dragoons that my husband took on faith and like, I knew it was wrong, dragoons were not invented by Napoleon or whatever outlandish thing he claimed. I wound up in a long debate with my husband about it because he then felt like he couldn’t trust anything the guy said and was really upset that a source he thought was reputable had recommended this podcast and like now he felt like he couldn’t trust anything from that source and…

… look, nobody’s perfect. So before you go getting excited about some hot new thing you learned, even if it’s from a source that really seems great, double and triple check it. Because that waterwheel story I was telling?

Every single one of the sources I found online claimed that they were invented in 4000 BCE. Which is insane. The wheel was invented in 3500 BCE in Mesopotamia and I know that off the top of my head because I happen to teach a unit on Mesopotamia (I’m a social studies teacher, but everyone will have different bases of knowledge). The waterwheel is attributed to one of Alexander the Great’s generals in the same breath as the 4000 BCE date.

So because I am terrible with dates, I google Alexander the Great, who it turns out was active around 400 BCE, and get my answer: somebody typo’d, and from thence began the game of telephone.

Anyway, once I’ve found a claim that I think I might ever want to repeat, I go looking for verification. Half the time I don’t bother with the actual source cited by the book, because that stuff is usually behind a paywall I can’t access, unless my Google Fu fails me. Then I try to get access to the source, or email the author (authors are often thrilled to respond!) for more information, or ask over on a heavily moderated subreddit like r/AskHistorians. Last week’s article about legal tender & currency was at least partially based on a question I once asked about the origins of coinage — apparently people like to erroneously claim that their ethnic group is responsible for that invention.

To Recap

If you come across a claim you’d like to integrate into your worldview or repeat, try to track it back to its source and evaluate the original claimant. This goes doubly for claims that seem surprising or unintuitive but can’t be dismissed out-of-hand. Consider: does the original claim match the one you read? If so, try to figure out if the original claimant is a crackpot or a respected institution, and whether or not their claim is considered controversial. How likely is it that they are incentivized to publish something that isn’t true?

How embarrassed will you be if you get corrected in public? You can’t do this kind of rigorous fact-checking for everything, so a good rule of thumb is to consider the return on investment. If you’re wrong about whether horses were first ridden before or after domestication, probably no one is going to be harmed by you being wrong in public. But it might be pretty dangerous to accidentally spread misinformation about, say, safe food preservation practices.

Be careful out there & good luck!