Science Must be Reproducible: Three Parts

 By Dr. Jay Lehr |May 9th, 2022|Regulation, Science|10 Comments

The National Association of Scholars (NAS) is a non profit organization of academics and independent scholars intent on recapturing the essence of scholarship which was so well respected in the past. We once respected all doctors without questioning their level of knowledge. Perhaps you have heard the joke beginning with the question “what do you call an individual who graduated medical school with a D average” the answer is “doctor”. The same was true for academics, college professors, all those that taught us at a college or university. Sadly as government slowly took over 80% of all academic research the standard of excellence declined . The by-words of too much research became “as the fear increases so does the money” and government involvement.

In hopes of bringing back the level of excellence among teachers and researchers NAS was formed. It does research itself into how schools are performing in the modern era. While there remains much that is good, there is a great deal that is bad.

This is the first of three essays taken from their new publication aptly titled Shifting Sands. It focuses on the failing efforts to reproduce scientific research that too often ends up supporting unnecessary or inappropriate government regulations. Much of their book uses the tremendous flaws in EPA’s effort to tighten the already unsupportable air quality regulation of Particulate Matter smaller than two and a half microns (millionth of a meter) which is called their PM2.5 rule. I wrote about the PM2.5 hearing they held by Zoom on February 25 on these pages in the weeks of March 27 and April 4, 2022. All but two of those who testified opposed EPA’s effort to tighten the current rule. There were 15 of us testifying against their plan. The EPA panel on the conference call did not ask a single question of the 15 people giving testimony in opposition. We suspect they had no interest in even listening to us. We all agreed that none of the evidence EPA was using to tighten the PM 2.5 rule could be reproduced even if their data could be obtained.

An irreproducibility crisis afflicts a wide range of scientific and social scientific disciplines from epidemiology to social psychology. Science has always had a layer of untrustworthy results published in respectable places and experts who are eventually shown to be sloppy, mistaken or untruthful. Herman Muller even won the Nobel Prize for his fraudulent studies of the fruit fly, which is now known to have resulted in the unsupportable Linear No Threshold model that has handicapped work on medical radiation for more than a half century. But the irreproducibility crisis is something new. It’s magnitude has brought many scientists confidence in other’s research to a very skeptical position. And most of today’s work is performed on the public’s dollar. A majority of modern research may well be wrong. How much government regulation is actually built on irreproducible science.

In the NAS text Shifting Sands the authors included 8 sources of misdirection leading to irreproducibility. They include:

• Malleable research plans
• Legally inaccessible data sets
• Opaque methodology and algorithms
• Undocumented data cleansing
• Inadequate or non existent data archiving
• Flawed statistical methods
• Publication bias hiding negative results
• Political or disciplinary group think (political correctness)

Government regulation is meant to clear a high barrier of proof. Regulations should be based on a large body of scientific research, the combined evidence of which provides sufficient certainty to justify reducing American’s liberty with a government regulation.

The justifiers of regulations based on flimsy or inadequate research often cite the “precautionary principle”. They would say that instead of being a regulation on rigorous science, they base the regulation on the possibility that a scientific claim is accurate. They do this with the logic that it is too dangerous to wait for the actual validation of a hypothesis, and the lower standard of reliability is warranted when dealing with matters that might involve severely adverse outcomes. The invocation of the precautionary principle is not only non-scientific, but is also an inducement to accepting poor science and even fraud.

We are living with this right now as the government wants to stop the use of fossil fuels because of a belief that it could lead the earths’ temperature to an unwanted level. No such proof of this exists.

The political consequences have unavoidably had the affect of tempting political activists to skew scientific research in order to impact the manner in which the government weighs evidence. Any formal system of assessment inevitably invites attempts to game it.

To all this we must add the distorting effects of massive government funding of scientific research. Our federal government is the largest single funder of research in the world. It’s expectations affect not only the research it directly funds, but also all research done in hopes of receiving federal funding. Government experts therefore have it in their power to create a skewed body of research which they can use to justify regulations.

A 2020 report prepared for the Natural Resource Defense Council estimates that American air pollution regulations cost $120 billion per year, and we may take that estimate provided to us by an environmental advocacy group to be the lowest plausible number.

It is time for US citizens to know all this and react to it in a manner that begins swinging the pendulum back toward more reliable research conclusions.

Note: Portions of this essay were excerpted from the book Shifting Sands with permission of the National Association of Scholars (NAS) and its authors Peter Wood, Stanley Young, Warren Kindzierski, and David Randall.

Irreproducible science – Part two

By Dr. Jay Lehr |May 16th, 2022 | Science | 80 Comments @ CFACT 

The empirical scientist conducts controlled experiments and keeps accurate, unbiased records of all observable conditions at the time the experiment is conducted. If a researcher has discovered a genuinely new or previously unobserved natural phenomenon, other researchers -with access to his or her notes and some apparatus of their own devising- should be able to reproduce or confirm the discovery. If sufficient corroboration is forthcoming the scientific community eventually acknowledges that the phenomenon is real and adapts existing theory to accommodate the new observations.

The validation of scientific truth requires replication or reproduction. Replicability most commonly refers to obtaining an experiment’s result in an independent study, by different investigator with different data, while reproducibility refers to different investigators using the same data, methods, and/or computer code to reach the same conclusions.

Yet today the scientific process of replication and reproduction has ceased to function properly. A vast proportion of the scientific claims in published literature have not been replicated or reproduced. Estimates are that a majority of these published claims that cannot be replicated or reproduced are in fact false.

An extraordinary number of scientific and social-scientific disciplines no longer reliably produce true results, a state of affairs commonly referred to as the Irreproducibility Crisis. A substantial majority of 1500 active scientists, recently surveyed by Nature magazine coined the urgent situation a Crisis. The scientific world’s completely inappropriate professional incentives bear much of the blame for this catastrophic failure.

Politicians and bureaucrats commonly act to maximize their self-interest rather than acting as disinterested servants of the public good. This applies specifically to scientists, peer reviewers and government experts. The different participants in the scientific research system all serve their own interests as they follow the systems incentives.

Well published university researchers earn tenure, promotion, lateral moves to more prestigious universities, salary increases, grants, professional reputation, and public esteem-above all, from publishing exciting new positive results. The same incentives affect journal editors who receive acclaim for their journal, and personal awards by publishing what may be viewed as exciting new research-even though the research has not been thoroughly vetted.

Grantors want to fund exciting research, and government funders possess the added incentive that exciting research with positive results supports the expansion of their organization’s mission. American university administrations want to host grant -winning research, from which they profit by receiving overhead costs- frequently the majority of the amount in the grant. As one who has experienced and viewed this first hand it will boggle the readers mind as to the huge portions of most research grants that goes to the university as overhead rather than to support actual research costs.

All these incentives reward published research with new positive claims but not necessarily reproducible research. Researchers, editors, grantors, bureaucrats , university administrations, each has an incentive to seek out what appears to be exciting new research that draws money, status, and power. There are few if any incentives to double check their work. Above all, they have little incentive to reproduce the research, to check that the exciting claim holds up because if it does not, they will lose money status and prestige.

The scientific world’s incentives for new findings rather than reproducible studies, drastically affects what becomes submited for publication. Scientists who try to build their careers on checking old findings, or publishing negative results are unlikely to achieve professional success. The result is that scientists do not submit negative results for publication. Some negative results go to the file drawer. Others somehow turn into positive results as researchers consciously or unconsciously massage their data and their analyses.(As a science modeler we call this “tuning”, a technical word for cheating). Neither do they perform or publish many replication studies, since the scientific world’s incentives do not reward those activities either.

The concept of statistical significance is being tortured to the point that literally hundreds if not thousands of useless papers claiming that significance, appear everywhere.

Researchers try to determine whether the relationships they study differ from what can be explained by chance alone by gathering data and applying hypothesis tests, also called tests of statistical significance. Most commonly they start by testing the chance that there is no actual relationship between two variables which is called the “null hypothesis”. If that fails and it is likely their is relationship they go on to other hypothesis. How well the data supports a “null hypothesis” (no relationship) is a statistic called a p-value. If the p-value is less that 5% or .05 it is assumed there may be a relationship between the variables being studied.

The governments central role in science, both in funding scientific research and in using scientific research to justify regulation, adds tremendously to the growth of flimsy statistical significance throughout the academic world. Within a generation statistical significance went from a useful shorthand that agricultural and industrial researchers used to judge whether to continue their current procedures or switch to something new, to a prerequisite for regulation, government grants, tenure and every other form of scientific prestige and also essential for publication.

Many more scientists use a variety of statistical practices with more or less culpable carelessness including:

• improper statistical methodology
• biased data manipulation that produces desired results
• selecting only measures that produce statistical significance and ignoring any that do not
• using illegitimate manipulations of research techniques

Still others run statistical analyses until they find a statistically significant result and publish the one result. This is called “p-hacking”. Far too many researchers report their methods unclearly and let the uninformed reader assume they actually followed a rigorous scientific process.

The most insidious of all scientific cheating is p-HARKING. That is when a scientist chooses a hypothesis only after collecting all the data that produces a desired result. A more obvious word for it is CHEATING. Irreproducible research hypotheses produced by HARKING sends whole disciplines chasing down rabbit holes.

Publication bias and p-harking collectively have degraded scientific research as a whole. In addition, for decades surveys show that researchers are unlikely too publish any negative results their studies uncover.

A false research claim can become the foundation for an entire body of literature that is uniformly false and yet becomes an established truth. We cannot tell exactly which pieces of research have been affected by these errors until scientists replicate every piece of published research. Yet we do possess sophisticated statistical strategies that does allow us to diagnose specific claims that support government regulation. One such method- an acid test for statistical skullduggery is p-value plotting described in detail in the the National Association of Scholars handbook, SHIFTING SANDS. A brief paper back I can not recommend too strongly

Note: Portions of this essay were excerpted from the NAS book SHIFTING Sands with permission of the National Association of Scholars and its authors Peter Wood, Stanley Young, Warren Kindzierski, and David Randall.

The National Association of Scholars (NAS) recognition of a scientific duplication crisis – Part 3

By Dr. Jay Lehr May 23rd, 2022 | Science | 23 Comments @ CFACT Part One, Part two

EPA regulations rely on environmental epidemiological literature, without applying rigorous tests for reproducibility, and without considering the environmental epidemiology discipline’s general refusal to take account of the need for Multiple Testing and Multiple Modeling. Such rigorous tests are needed not least because earlier generations of environmental epidemiologists have already identified the low hanging fruit.

These include massive statistical correlations between risk factors and health outcomes such as the connection between smoking and lung cancer. Modern environmental epidemiologists habitually seek out small but significant risk factors and health outcome associations. These practices render their research susceptible to false positives as real results. They risk mistaking an improperly controlled co-variable for a positive association.

Environmental epidemiologists are aware of these difficulties, but regardless have made their discipline into exercises in applied statistics. They do little to control for bias, p-hacking and other well known statistical errors. The intellectual leaders of their discipline have positively counseled against taking measures to avoid these pitfalls. But environmental epidemiologists, and the bureaucrats who depend on their work to support regulations, proceed as a field with unwarranted self-confidence. They have an insufficient sense of the need for humble awareness of how much statistics remains an exercise in measuring uncertainty rather than establishing certainty. Their results, do not possess an adequate scientific foundation. Their so-called “facts” are built on Shifting Sands, not on the solid rock of transparent, and critically reviewed scientific inquiry.

A NAS study showed how one particular set of statistical techniques simply counting and p-value plots, can provide a severe test for environmental epidemiology. Meta analyses must be used to detect p-hacking and other frailties in the underlying scholarly literature. We have used these techniques to demonstrate that meta-analyses associating PM 2.5 and other air quality components with mortality, heart attacks and asthma attacks fail this severe test.

The NAS study also demonstrates negligence on the part of both environmental epidemiologists and the EPA. The discipline of environmental epidemiology has failed to adopt a simple statistical procedure to test their research. The EPA failed to require that research justifying regulation be subjected to such a test. These persistent failures undercut confidence in their professional capacities as researchers and as regulators.

Both environmental epidemiology as a discipline, including journals , foundations and tenure committees and the EPA must adopt a range of reforms to improve the necessary reproducibility of their research. However, NAS directs its recommendations to the EPA and more broadly to federal regulatory and granting agencies.

They have reluctantly come to the conclusion that scientists will not change their practices unless the federal government credibly warns them it will withhold government grant dollars until they adopt stringent reproducibility reforms. NAS has also come to the conclusion that federal regulators will not adopt stringent new tests of science underlying regulation unless they are explicitly required to do so.

The National Association of Scholars recommend the following eleven actions be taken by the EPA in order to bring their methodologies up to the level of Best Available Science which is mandated in The Information Quality Act of 2019.

The EPA should adopt resampling methods as part of its standard battery of tests applied to environmental epidemiology research/

1. The EPA should adopt resampling methods as part of its standard battery of tests applied to environmental epidemiology research.
2. The EPA should rely for regulation exclusively on meta-analyses that use tests to take account of endemic questionable research procedures, p-hacking and HARKing.
3. The EPA should redo its assessment of base studies more broadly to take account of endemic questionable research procedures, p-hacking and HARKing.
4. The EPA should require preregistration and registered reports of all research that informs regulation.
5. The EPA should also require public access to all research data used to justify regulation.
6. The EPA should consider the more radical reform of funding data set building, and data set analysis separately.
7. The EPA should place greater weight on reproduced research.
8. The EPA should constrain the use of “weight of evidence” to take account of the irreproducibility crisis.
9.  The EPA should report the proportion of positive results to negative results in the research it funds.
10.  The EPA should not rely on research claims of other organizations until these organizations adopt sound statistical practices
11.  The EPA should increase funding to investigate direct causal biological links between substances and health outcomes.

NAS has used the phrase “irreproducible crisis” throughout this essay, and they note that distinguished meta-researchers prefer to regard the current state of affairs as a challenge rather than a crisis.

You do not need to believe it to be a crisis. These current scientific practices are simply not the best available science. We should use the best scientific practices simply because they are the best scientific practices. Mediocrity ought not be acceptable.

If this is the first article you have read in this series please go back to the past two weeks at cfact.org to read the even more extensive parts 1 and 2 or click on my name at the beginning of this article and all my previous article titles will pop up on a list. Click on any title and the full article will appear.

There is no doubt that all CFACT readers question many EPA regulations. After you read this series of articles extracted from the National Association of Scholars booklet, SHIFTING SANDS, you will question even more.

Note: Portions of this essay were excerpted from the book Shifting Sands with permission of the National Association of Scholars (NAS) and its authors Peter Wood, Stanley Young, Warren Kindzierski, and David Randall.

Author

• Dr. Jay Lehr

  CFACT Senior Science Analyst Jay Lehr has authored more than 1,000 magazine and journal articles and 36 books. Jay’s new book A Hitchhikers Journey Through Climate Change written with Teri Ciccone is now available on Kindle and Amazon.