By
Dr. Jay Lehr
|May 9th, 2022|Regulation, Science|
The
National Association of Scholars (NAS) is a non profit organization of
academics and independent scholars intent on recapturing the essence of
scholarship which was so well respected in the past. We once respected
all doctors without questioning their level of knowledge. Perhaps you
have heard the joke beginning with the question “what do you call an
individual who graduated medical school with a D average” the answer is
“doctor”.
The same was true for
academics, college professors, all those that taught us at a college or
university. Sadly as government slowly took over 80% of all academic
research the standard of excellence declined . The by-words of too much
research became “as the fear increases so does the money” and government
involvement.
In hopes of bringing
back the level of excellence among teachers and researchers NAS was
formed. It does research itself into how schools are performing in the
modern era. While there remains much that is good, there is a great deal
that is bad.
This is the first of
three essays taken from their new publication aptly titled Shifting
Sands. It focuses on the failing efforts to reproduce scientific
research that too often ends up supporting unnecessary or inappropriate
government regulations. Much of their book uses the tremendous flaws in
EPA’s effort to tighten the already unsupportable air quality regulation
of Particulate Matter smaller than two and a half microns (millionth of
a meter) which is called their PM2.5 rule. I wrote about the PM2.5
hearing they held by Zoom on February 25 on these pages in the weeks of
March 27 and April 4, 2022. All but two of those who testified opposed
EPA’s effort to tighten the current rule. There were 15 of us testifying
against their plan. The EPA panel on the conference call did not ask a
single question of the 15 people giving testimony in opposition. We
suspect they had no interest in even listening to us. We all agreed that
none of the evidence EPA was using to tighten the PM 2.5 rule could be
reproduced even if their data could be obtained.
An irreproducibility
crisis afflicts a wide range of scientific and social scientific
disciplines from epidemiology to social psychology. Science has always
had a layer of untrustworthy results published in respectable places and
experts who are eventually shown to be sloppy, mistaken or untruthful.
Herman Muller even won the Nobel Prize for his fraudulent studies of the
fruit fly, which is now known to have resulted in the unsupportable
Linear No Threshold model that has handicapped work on medical radiation
for more than a half century. But the irreproducibility crisis is
something new. It’s magnitude has brought many scientists confidence in
other’s research to a very skeptical position. And most of today’s work
is performed on the public’s dollar. A majority of modern research may
well be wrong. How much government regulation is actually built on
irreproducible science.
In the NAS text Shifting Sands the authors included 8 sources of misdirection leading to irreproducibility. They include:
- Malleable research plans
- Legally inaccessible data sets
- Opaque methodology and algorithms
- Undocumented data cleansing
- Inadequate or non existent data archiving
- Flawed statistical methods
- Publication bias hiding negative results
- Political or disciplinary group think (political correctness)
Government regulation is
meant to clear a high barrier of proof. Regulations should be based on a
large body of scientific research, the combined evidence of which
provides sufficient certainty to justify reducing American’s liberty
with a government regulation.
The justifiers of
regulations based on flimsy or inadequate research often cite the
“precautionary principle”. They would say that instead of being a
regulation on rigorous science, they base the regulation on the
possibility that a scientific claim is accurate. They do this with the
logic that it is too dangerous to wait for the actual validation of a
hypothesis, and the lower standard of reliability is warranted when
dealing with matters that might involve severely adverse outcomes. The
invocation of the precautionary principle is not only non-scientific,
but is also an inducement to accepting poor science and even fraud.
We are living with this
right now as the government wants to stop the use of fossil fuels
because of a belief that it could lead the earths’ temperature to an
unwanted level. No such proof of this exists.
The political
consequences have unavoidably had the affect of tempting political
activists to skew scientific research in order to impact the manner in
which the government weighs evidence. Any formal system of assessment
inevitably invites attempts to game it.
To all this we must add
the distorting effects of massive government funding of scientific
research. Our federal government is the largest single funder of
research in the world. It’s expectations affect not only the research it
directly funds, but also all research done in hopes of receiving
federal funding. Government experts therefore have it in their power to
create a skewed body of research which they can use to justify
regulations.
A 2020 report prepared
for the Natural Resource Defense Council estimates that American air
pollution regulations cost $120 billion per year, and we may take that
estimate provided to us by an environmental advocacy group to be the
lowest plausible number.
It is time for US
citizens to know all this and react to it in a manner that begins
swinging the pendulum back toward more reliable research conclusions.
Note: Portions of this essay were excerpted from the book Shifting Sands
with permission of the National Association of Scholars (NAS) and its
authors Peter Wood, Stanley Young, Warren Kindzierski, and David
Randall.
Irreproducible science – Part two
By
Dr. Jay Lehr
|May 16th, 2022 | Science |
The empirical scientist conducts
controlled experiments and keeps accurate, unbiased records of all
observable conditions at the time the experiment is conducted. If a
researcher has discovered a genuinely new or previously unobserved
natural phenomenon, other researchers -with access to his or her notes
and some apparatus of their own devising- should be able to reproduce or
confirm the discovery. If sufficient corroboration is forthcoming the
scientific community eventually acknowledges that the phenomenon is real
and adapts existing theory to accommodate the new observations.
The validation of scientific truth requires
replication or reproduction. Replicability most commonly refers to
obtaining an experiment’s result in an independent study, by different
investigator with different data, while reproducibility refers to
different investigators using the same data, methods, and/or computer
code to reach the same conclusions.
Yet today the scientific process of
replication and reproduction has ceased to function properly. A vast
proportion of the scientific claims in published literature have not
been replicated or reproduced. Estimates are that a majority of these
published claims that cannot be replicated or reproduced are in fact
false.
An extraordinary number of scientific and
social-scientific disciplines no longer reliably produce true results, a
state of affairs commonly referred to as the Irreproducibility Crisis. A
substantial majority of 1500 active scientists, recently surveyed by
Nature magazine coined the urgent situation a Crisis. The scientific
world’s completely inappropriate professional incentives bear much of
the blame for this catastrophic failure.
Politicians and bureaucrats commonly act to
maximize their self-interest rather than acting as disinterested
servants of the public good. This applies specifically to scientists,
peer reviewers and government experts. The different participants in the
scientific research system all serve their own interests as they follow
the systems incentives.
Well published university researchers earn
tenure, promotion, lateral moves to more prestigious universities,
salary increases, grants, professional reputation, and public
esteem-above all, from publishing exciting new positive results. The
same incentives affect journal editors who receive acclaim for their
journal, and personal awards by publishing what may be viewed as
exciting new research-even though the research has not been thoroughly
vetted.
Grantors want to fund exciting research,
and government funders possess the added incentive that exciting
research with positive results supports the expansion of their
organization’s mission. American university administrations want to host
grant -winning research, from which they profit by receiving overhead
costs- frequently the majority of the amount in the grant. As one who
has experienced and viewed this first hand it will boggle the readers
mind as to the huge portions of most research grants that goes to the
university as overhead rather than to support actual research costs.
All these incentives reward published
research with new positive claims but not necessarily reproducible
research. Researchers, editors, grantors, bureaucrats , university
administrations, each has an incentive to seek out what appears to be
exciting new research that draws money, status, and power. There are few
if any incentives to double check their work. Above all, they have
little incentive to reproduce the research, to check that the exciting
claim holds up because if it does not, they will lose money status and
prestige.
The scientific world’s incentives for new
findings rather than reproducible studies, drastically affects what
becomes submited for publication. Scientists who try to build their
careers on checking old findings, or publishing negative results are
unlikely to achieve professional success. The result is that scientists
do not submit negative results for publication. Some negative results go
to the file drawer. Others somehow turn into positive results as
researchers consciously or unconsciously massage their data and their
analyses.(As a science modeler we call this “tuning”, a technical word
for cheating). Neither do they perform or publish many replication
studies, since the scientific world’s incentives do not reward those
activities either.
The concept of statistical significance is
being tortured to the point that literally hundreds if not thousands of
useless papers claiming that significance, appear everywhere.
Researchers try to determine whether the
relationships they study differ from what can be explained by chance
alone by gathering data and applying hypothesis tests, also called tests
of statistical significance. Most commonly they start by testing the
chance that there is no actual relationship between two variables which
is called the “null hypothesis”. If that fails and it is likely their is
relationship they go on to other hypothesis. How well the data supports
a “null hypothesis” (no relationship) is a statistic called a p-value.
If the p-value is less that 5% or .05 it is assumed there may be a
relationship between the variables being studied.
The governments central role in science,
both in funding scientific research and in using scientific research to
justify regulation, adds tremendously to the growth of flimsy
statistical significance throughout the academic world. Within a
generation statistical significance went from a useful shorthand that
agricultural and industrial researchers used to judge whether to
continue their current procedures or switch to something new, to a
prerequisite for regulation, government grants, tenure and every other
form of scientific prestige and also essential for publication.
Many more scientists use a variety of statistical practices with more or less culpable carelessness including:
- improper statistical methodology
- biased data manipulation that produces desired results
- selecting only measures that produce statistical significance and ignoring any that do not
- using illegitimate manipulations of research techniques
Still others run statistical analyses until
they find a statistically significant result and publish the one
result. This is called “p-hacking”. Far too many researchers report
their methods unclearly and let the uninformed reader assume they
actually followed a rigorous scientific process.
The most insidious of all scientific
cheating is p-HARKING. That is when a scientist chooses a hypothesis
only after collecting all the data that produces a desired result. A
more obvious word for it is CHEATING. Irreproducible research hypotheses
produced by HARKING sends whole disciplines chasing down rabbit holes.
Publication bias and p-harking collectively
have degraded scientific research as a whole. In addition, for decades
surveys show that researchers are unlikely too publish any negative
results their studies uncover.
A false research claim can become the
foundation for an entire body of literature that is uniformly false and
yet becomes an established truth. We cannot tell exactly which pieces of
research have been affected by these errors until scientists replicate
every piece of published research. Yet we do possess sophisticated
statistical strategies that does allow us to diagnose specific claims
that support government regulation. One such method- an acid test for
statistical skullduggery is p-value plotting described in detail in the
the National Association of Scholars handbook, SHIFTING SANDS. A brief
paper back I can not recommend too strongly
Note: Portions of this
essay were excerpted from the NAS book SHIFTING Sands with permission of
the National Association of Scholars and its authors Peter Wood,
Stanley Young, Warren Kindzierski, and David Randall.
The National Association of Scholars (NAS) recognition of a scientific duplication crisis – Part 3
- The EPA should rely for regulation exclusively on meta-analyses that use tests to take account of endemic questionable research procedures, p-hacking and HARKing.
- The EPA should redo its assessment of base studies more broadly to take account of endemic questionable research procedures, p-hacking and HARKing.
- The EPA should require preregistration and registered reports of all research that informs regulation.
- The EPA should also require public access to all research data used to justify regulation.
- The EPA should consider the more radical reform of funding data set building, and data set analysis separately.
- The EPA should place greater weight on reproduced research.
- The EPA should constrain the use of “weight of evidence” to take account of the irreproducibility crisis.
-
NAS has used the phrase “irreproducible crisis” throughout this
essay, and they note that distinguished meta-researchers prefer to
regard the current state of affairs as a challenge rather than a crisis.
You do not need to believe it to be a crisis. These current
scientific practices are simply not the best available science. We
should use the best scientific practices simply because they are the
best scientific practices. Mediocrity ought not be acceptable.
If this is the first article you have read in this series please go
back to the past two weeks at cfact.org to read the even more extensive
parts 1 and 2 or click on my name at the beginning of this article and
all my previous article titles will pop up on a list. Click on any title
and the full article will appear.
There is no doubt that all CFACT readers question many EPA
regulations. After you read this series of articles extracted from the
National Association of Scholars booklet, SHIFTING SANDS, you will question even more.
Note: Portions of this essay were excerpted from the book Shifting Sands
with permission of the National Association of Scholars (NAS) and its
authors Peter Wood, Stanley Young, Warren Kindzierski, and David
Randall.
