Scientific American falsely links homeschooling to abuse

Neglecting the base rate is something SciAm should check

Art for The Intrinsic Perspective is by Alexander Naughton

In June’s edition of Scientific American the editors of the magazine collectively called for more legislation around homeschooling, including background checks of parents. Their argument was that homeschooling commonly functions as a cover for child abuse (a similar argument was recently made by the editors of The Washington Post, and in ProPublica too).

Some might say such an op-ed by the editors of Scientific American is part of a broader trend: SciAm has been increasingly criticized for taking political stances that go beyond science. On the other hand, this particular op-ed is not some barn-burning screed filled with jargon. Its content includes sources linking homeschooling to abuse, things worthy of addressing in a serious data-based manner. Although it turns out that when one does address them in a serious data-based manner, they disappear. Simply because SciAm embarrassingly neglected one of the most fundamental numbers in statistical analysis: the base rate.

Before getting into the details, it’s worth noting the broader context. Why is this string of op-eds happening? Homeschooling’s increasing popularity has taken it from niche to a battleground over the future of education, simply because…

Homeschooling is the fastest growing education sector.

Source: The Washington Post

Somewhere around ~6% of US school-age children are now homeschooled (~3 million).¹ That’s about twice as popular as Catholic schools. The secular version is growing fastest: homeschooling primarily for religious purposes is a falling minority and new entrants to the category are just as likely to be democrats as republicans.

source: The Washington Post

As even The Washington Post itself points out:

Homeschoolers conform less and less to the stereotype of mom working one-on-one with her children at the kitchen table… About half of homeschool parents said their children would receive at least some instruction from a teacher or tutor this year, much higher than the 22 percent who said the same in 2019. Nearly 6 in 10 said their kids would take live online classes, and about 1 in 5 plan to participate in a homeschool co-op.

Yet, despite its popularity, research on the outcomes of homeschooling is actually quite limited. For example, it’s hard to figure out:

Do homeschoolers underperform or overperform their peers?

The most commonly circulated numbers, those that come up first when you Google, are derived from homeschooling advocacy organizations. Like:

source

The editors at SciAm have something to say about that:

Home­school­ing advocacy organizations promote studies that claim to show equal or higher levels of academic achievement among home­schooled students. But these studies often are conducted by home­school­ing advocates and are methodologically flawed.

Here the SciAm editors actually seem correct. They’re referring to basically one guy: Brian Ray. He’s been running the National Home Education Research Institute for decades (that’s where that “78% outperform” statistic comes from if you look closely). He’s published so many papers advocating for homeschooling that he’s essentially flooded the literature to the point it’s hard to get an objective take on the subject.

Where to turn? How about a 2020 survey of the existent academic literature: “Homeschooling: An Updated Comprehensive Survey of the Research.” An 84-page monograph, it draws from ~2,000 sources and is written by what seem like relatively unbiased university researchers. There, Ray’s efforts are described as hindering objective research into homeschooling’s effectiveness:

Happily, there is a move away from the advocacy disguised as research that characterized many of the most influential studies of the 1990s and early 2000s (e.g., Ray, 1997a; Ray, 1997b; Ray 2004a; Ray2004b; Ray, 2010; Rudner, 1999)…. Brian D. Ray’s National Home Education Research Institute (NHERI), which for decades was the dominant player in homeschooling research and the most visible proponent of research-based homeschooling advocacy, has become far less active in recent years even as university-based research has increased.

Essentially, Brian Ray would publish survey results (self-reported) based on outreach and parents volunteering. The homeschoolers, perhaps unsurprisingly, came out as amazingly ahead. Yet people still regularly cite the results online (this is despite criticism of Ray’s work from other more empirically-minded homeschool advocates).

So what does this new university-based research tell us about homeschooling? The main result is that homeschooling is a hugely variable form of education. Even the SciAm editors were forced to give the possible upsides of homeschooling a nod, admitting that:

Home­schooled students have won the National Spelling Bee; one was the most prolific mathematician in history. Many are well-rounded and well-adjusted children who go on to thrive as adults.

By the way, the “most prolific mathematician” SciAm mentions is Paul Erdős, who was homeschooled by his mother, herself a mathematics teacher.² Quelle surprise.

What’s interesting is that homeschoolers don’t seem drastically different from their peers—maybe a little bit ahead in verbal scores, maybe a similarly small amount behind in math scores. But despite the average often being around the same, the variability of outcomes at the tails is one of the few hypotheses education researchers actually credit (in turn, variability makes other blanket statements about outperforming or underperforming difficult). Some examples the authors of the large-scale survey give are that:

At the high end, Yusof (2015) identified a subset of informally-educated, high-achieving homeschoolers who enjoyed and were very good at math. Wilkens, Wade, Sonnert, and Sadler (2015), likewise, using data from the 2009-2010 Factors Influencing College Success in Mathematics survey (N=10,492 representing 134 institutions), found that homeschooled students outperformed their demographic peers in Calculus I as first-year students….

At the other end of the achievement spectrum, Green-Hennessy (2014), in a powerful analysis of data from the National Survey on Drug Use and Health for the years 2002-2011 (N=182,351), found that homeschoolers age 12 and up were two to three times more likely than their public-school equivalents to report being behind grade level….

While much more research needs to be conducted, these studies at least hint at the possibility that the normal curve of homeschooler academic achievement may have a wider distribution spread than that of students attending institutional schools…

One thing that likely biases homeschooling results is how it can act as a last resort for troubled or academically-behind children.

As Green-Hennessy (2014) notes, homeschooled children behind grade level “may well be struggling academically before homeschooling commences.” Green-Hennessy’s speculation was given empirical validation.... In Kentucky and other states, a significant number of families and schools are using homeschooling as a “dropout loophole” for at-risk children (Coleman, 2019).

So if I had to sum up the official university-based research into homeschooling, it would be pretty much what I expected before looking into it: You get out of homeschooling what you put in. Some results are amazing. Some are terrible. What contributes substantially to the negative side is that homeschooling is used to avoid the stigma / record of dropping out of school in tough childhood situations. Which brings me to…

The accusations about abuse by the editors of Scientific American.

The SciAm op-ed (along with the Post and ProPublica) heavily associates homeschooling with child abuse.

In 2020 an 11-year-old boy in Michigan was found dead after his stepmother used homeschooling to conceal years of torture… In these cases, homeschooling was a farce—a hole in children’s social safety net for abusers to exploit.

They go on to say:

Although it’s impossible to say how commonly homeschooling conceals abuse, data from Connecticut paint a concerning picture. Following the abuse and 2017 death of an autistic teenager whose mother had removed him from school, Connecticut’s Office of the Child Advocate found that 36 percent of children withdrawn from six nearby districts to be home­schooled lived in homes that had been subject to at least one report of suspected abuse or neglect.

This Connecticut investigation emphasized by the SciAm editors is by far the strongest existent link between homeschooling and abuse (it is mentioned in the large education survey as well). The investigation was spurred by the death of the nonverbal autistic teenager, Matthew Tirado, whose mom horrifically starved him to death (he was 84 pounds at the end). The story made national news.

Matthew was not homeschooled. He was still registered in public school. The Department of Children and Families (DCF) didn’t take the many reports of abuse seriously, ignoring a number of previous red flags (e.g., the mother missed 5 scheduled court dates). According to the New Haven Register:

… after nine months of not being allowed to see Matthew Tirado or verify his whereabouts, DCF recommended the court terminate the neglect case. Transcripts from the December 2016 court proceeding indicate the hearing lasted less than a minute.

So why is homeschooling involved at all? Because (a) leading up to his death, the mother prevented Matthew from attending normal school, where he was registered (SciAm calls this being “removed from school”), and (b) two months prior to Matthew’s death, the mother did request that the sister be taken out of school to be officially homeschooled.

Rather than investigating why the Connecticut DCF closed an investigation into a situation with tons of red flags, homeschooling became the center of attention in the case. As part of this, the Office of the Child Advocate in Connecticut examined six public school districts and tracked all students pulled out to be homeschooled during the 2013-2016 period (total found: 380). They then cross-checked that with the child abuse reports database kept by the DCF for allegations ever made against the parents for abuse or neglect (total found: 138). That’s where the 36% having parents subject to reports of alleged abuse comes from.

When I initially read the 36% number I thought: Wow, maybe the SciAm editors have a point. Isn’t that incredibly high? And the stories highlighted by the Connecticut investigation are indeed tragic. If you read the case reports (probably the worst ones), they portray lives in downward spirals, mostly due to things like drug use by the parents. In some situations, homeschooling appears to be a way to stop the bullying associated with being a neglected kid (by far the predominate form of abuse).

But the sheer human tragedy of some situations shouldn’t prevent us from asking really basic statistical questions like:

What’s the base rate?

I’m no Scientific American editor, but when you’re giving a statistic, like rates of kids withdrawn to be homeschooled whose parents were at some point reported for alleged abuse, it’s pretty important to think about the base rate. A base rate in statistics comes into play when you’re comparing some measured number (e.g., how many at Harvard come from rich families?) to some other broader context (what percent of the US population comes from rich families?). Here, a sensible base rate to check would be how often parents in general, all across the nation, get reported for suspected abuse.

Well, it turns out researchers have an answer to that question. Here’s from a 2017 paper, “Lifetime Prevalence of Investigating Child Maltreatment Among US Children:”

We estimate that 37.4% of all children experience a child protective services investigation by age 18 years.

First of all, holy crap. 37%? That’s so much higher than I would have guessed. More than one third of Americans get Child Protective Services (CPS) called on them? I mean, look at these cumulative risk assessments!

Age in years of average child against risk of report

Why is this number so high? It’s important to note that registered (sometimes unintuitively called “accepted”) reports do not indicate guilt. It merely means that the agency found the alleged situation could rise to the legal level of abuse or neglect. Only a minority of these reports are further counted as being “substantiated.” So it appears reports of abuse that agencies find worth keeping are relatively common and vary widely in severity. According to a former CPS investigator on Reddit doing an AMA, reasons to make reports include:

Unexplained injuries, injuries inconsistent with caregivers explanation, children that appear to have a lack of clean clothing/water/food, discipline objectively inconsistent with the child's age or developmental level - whether it's physical or not, threats or name calling, child outcries of maltreatment, etc. are all prime examples, but it's by no means a complete list. Drugs were responsible for the majority of my cases, but there are so many other ways to mistreat a child and I'm big believer in "see something, say something.”

Personally, 37% of Americans abusing their kids to the degree that they should be investigated by the government sounds high to me. But maybe my slice of life and experience is more provincial than I thought.

Regardless of how or why that number is so shockingly high, either from actual abuse or from over-reporting, 36% of withdrawn-to-be-homeschooled kids having parents subject to reports of alleged abuse in Connecticut looks to be an identical percent of reports of alleged abuse in the general population itself. In other words, you could just randomly sample the national population and get the exact same numbers. Put in that context, the Connecticut investigation actually shows zero link between homeschooling and likelihood of abuse reports.

Despite having on offer such a knock-down comparison (36% vs. 37%), unlike the SciAm editors, I’ll openly admit to some fuzziness. In one case we are using reports to CPS. In the other, “child protection reports” that I think are kept by the DCF. These two types of reports likely have similar standards but there’s a chance the two differ in average severity. Additionally, the cumulative nationwide risk, 37%, is by age 18. The Connecticut investigation lacks things that would be in a standard scientific paper, like average age of their sample, but it’s definitionally younger than 18. However, as you can see from the cumulative risk scores, even by age 12 (halfway through school) the nationwide probability of a report is at or above 30%.

Yet there are actually far more prior reasons to expect the Connecticut sample just being at the nationwide average to actually contraindicate an abuse link (i.e., give weight to the hypothesis homeschooling parents are less likely to abuse than the general pop). These include:

1. The Connecticut number is for entire families (the alleged report didn’t have to be about the withdrawn-to-be-homeschooled child). Unlike the CPS nationwide number, which is uniquely per kid. Since it sums between children and across parents, we should expect it to be higher than the national unique per-kid risk (yet it is not higher).

2. While Connecticut might be below the national average for reports, Hartford itself (where the investigation occurred) is likely not, at least judging from demographics and socioeconomic status. Additionally, a sample from public school districts is biased away from the top of the socioeconomic spectrum to begin with.

3. The Connecticut investigation focuses solely on withdrawals and therefore is overly sensitive to how researchers say homeschooling can act as a “dropout loophole.” The numbers don’t include those who homeschool from the beginning, which might select out families for which the act is more consciously thought out instead of a reaction to failures and troubles.

These likely at minimum balance out any potential age difference. Alternatively, as I suspect, they might even shift the investigation’s finding to being that homeschoolers are at less risk of abuse.³ Certainly, it may be that in some extreme cases homeschooling is used as a veil for abuse (although since only ~20% of alleged abuse reports come from education professionals, it’s not exactly a surefire way to avoid reports).

Regardless, merely being around the national average for reports of abuse is a far cry from being tied specifically to abuse, which is the line of attack these editorials have all used. Nothing changes the fact that…

Homeschooling rates of abuse look, at worst, about identical to the national average.

Overall, it’s hard for me to look at this entire investigative saga without seeing some bias at work. Matthew Tirado, whose death spurred the Connecticut investigation, was still technically registered in public school. His case looks more like DCF dropping the ball, indicating it was not secrecy of abuse that was the problem but follow-through on reports. Then neither the resultant Connecticut investigation, nor the SciAm editors, bothered to ask really basic questions like “Is this anything different than what we’d expect from a random sample?” before smearing homeschooling with the abuse link and calling for legislation.

I expect that some proposed piece of official legislation will soon cite both the Connecticut investigation and calls like the SciAm editorial. At that time, I hope whoever is in power knows something about base rates.

1. Full disclosure: This number of total children homeschooled in the US will likely include, eventually, my own kids. The decision is still years out, given the oldest only just turned three (I have an ongoing series on how to teach reading to toddlers). Options are open. If I ever found a formal school with an exciting enough program I’d consider moving across the country for it. But in the short-term, despite our town itself being great, the local public schools are… not great. Like “someone was attacked in a bathroom, there was a viral video, and it made national news” not great (no, I won’t link and spread the video further).

2. Paul Erdős was so prolific that scientists and mathematicians compete to have the lowest “Erdős number,” which is how far away you are from being a paper co-author of Erdős (i.e., the shortest path of steps of co-authorship between you to him). My Erdős number is five. That’s better than the lowly peon-like sixes, but worse than the aristocratic and judgy fours.

3. Unfortunately, we don’t have the obvious base rate for the Connecticut investigation, which is “Connecticut children of the same year and in the same districts who weren’t withdrawn to be homeschooled” (a number a peer-reviewed study would be tasked with supplying to make the claim the official investigation does).