There is a fundamental reason for this that is interesting, or at least can be depending on what one finds interesting.
It's a really unfortunate thing, as scientists use probability (for example p values, which have to do with how random variation relates with the findings) on a constant basis. But any number of scientists and certainly doctors and other professionals have a completely wrong "understanding" of what the statistics mean.
If let's say something is found "statistically significant" to p=0.01, for example, most think, "Oooh! That is pretty good! Only a one percent probability that that was only luck, we can ignore that, this is good stuff!"
In fact the standard for publication usually is p of 0.05 (five percent) or better.
But that is NOT what it means, at all.
What it means is that WHEN chance alone is the only factor, that percentage of the time -- whether 1% or 5% or whatever was calculated by the statistics -- these results will occur and it will falsely APPEAR as if there is a result.
This has got little to do (think about it!) with what the probability is that THESE results are from chance. Generally speaking the probability is far higher!
It is a natural consequence of statistics that something like the above-reported 1/3 of apparent results are in fact not real at all, but were only the product of chance -- the treatment group happening to be a group of people that mostly got better and the placebo group happening to be a group that mostly didn't.
This will happen by chance much more often than the percentage indicated by the p value, for a couple of reasons.
But yet absolutely one can go all the way through a doctoral science program and never be taught this about the MEANING of the statistics.
Very briefly and oversimplified due to my not remembering the details exactly: if the best estimate of the likelihood of a real effect occurring -- for example an untried drug, and we know from previous trials of related drugs that say 99% of the time they don't work, so entering into the new study the probability of actually being effective is at this time best estimated at 1% -- then the needed p value to have a mere 95% confidence of actual effect is not p <= 0.05, but p <= 0.0005!!
Which needless to say, scarcely any medical study meets.
It gets worse.
That is only when considering one study in isolation!
When considering published studies, there is on top of this the selection bias. The detected failures don't get reported. Only the success stories, or apparent success stories, are published, for the most part.
So let's say that there was a class of drugs where in fact NONE of them work, but many studies are done on very many of them.
By chance alone, if many studies are done, in some of them the treatment group will do "significantly" better.
So with the sort of p values that are accepted as being "significant" and with selection bias on top of that, it's no wonder that a high percentage of evidence "showing" that various things supposedly work actually is the product of chance alone.
It's not only no wonder: it's essentially inevitable.
