The Dangers of Analytic Journalism

This past summer, the Wall Street Journal published an article revealing that Medicare Advantage insurance programs send nurses into members’ homes to assess their risk of developing costly medical issues. Medicare compensates these insurance companies based on the health risk profiles of their beneficiaries, with higher payments for higher-risk patients. By conducting in-home visits, these companies are attempting to adjust risk profiles by adding new diagnoses—diagnoses that the patient's regular healthcare providers may not have documented.

 For the article, the authors obtained data from the Centers for Medicare and Medicaid Services (CMS) on Medicare Advantage patients from 2018 to 2021, which included clinical visits and diagnoses. Their analysis revealed that these in-home visits often resulted in additional diagnoses, which in turn led to higher reimbursements. The authors concluded that these diagnoses were inappropriate and unwarranted, citing two main reasons: first, that nurses were adding new diagnoses that had not been identified during routine care; and second, that nurses had, in some cases, misdiagnosed patients. The article implied that diagnoses made by regular healthcare providers were more accurate, while any misdiagnosis by home-visiting nurses was intentional. 

 However, a plausible explanation could also be that the regular providers missed certain diagnoses, and any misdiagnoses by the nurses were simple human errors. In fact, misdiagnosis is one of the most common types of medical errors, and it is possible that both sets of providers made mistakes. The authors’ conclusions seemed to have overlooked this possibility.

 This story is an example of analytic journalism, a growing trend that combines traditional investigative reporting with “big data” analysis. While this approach can be powerful, it also requires caution and new safeguards.

 Historically, investigative journalism has focused on uncovering hidden problems through in-depth exploration. With the addition of analytics, journalists can now leverage large datasets to reveal hidden facts and better understand complex issues. However, investigative journalism can sometimes be biased towards sensationalism, driven by the need for attention-grabbing headlines, readership, and accolades. Traditional investigative reporting is often qualitative and transparent, allowing readers to critically evaluate the methodology. Analytic journalism, however, does not always offer the same level of transparency.

Big data analysis, as used in analytic journalism, is valuable for identifying trends, distributions and averages, similar to how sociologists analyze demographic data. But a key problem arises when correlation is mistaken for causation. This can lead to misleading conclusions, as seen in the Wall Street Journal article, where updates to patients' risk profiles were framed as potential misconduct.

 The Wall Street Journal might argue that sharing its analytic approach ensures transparency and allows readers to judge the findings. However, this is misleading. Most readers lack the expertise to evaluate the quality of complex analyses, and therefore, they cannot critically assess the conclusions presented.

 In academic science, peer review allows experts to evaluate the validity of a study's methods and findings. Reviewers can identify issues for correction or question the analysis itself. The authors of this Wall Street Journal article, like so many of their industry peers performing this kind of journalism, assessed the validity of additional diagnoses based on how patients were treated after the diagnosis. They assumed that if no prescription was filled for an appropriate drug, the diagnosis was problematic. This reasoning, however, involves assumptions that would likely raise red flags for peer reviewers. Reviewers might suggest an alternative explanation: the diagnosis was correct, but poor communication between the insurance company and healthcare providers, a common issue, led to a lack of proper treatment.

 As investigative journalism increasingly relies on big data, it may benefit from a similar external review process to assess the appropriateness of its analyses. This could enhance trust and ensure that actions based on the findings are justified.