Dear Shalom,
I am afraid that the 21st. Century statistical emperor has no clothes.
I agree with some of the scholarly observations of Matt Williams, and I would equally apply an extension of his criticism of the research studies in question to much of what is currently called "research".
With the very rare exceptions of some very expensive and extensive longitudinal studies with large enough sample sizes to draw somewhat definitive conclusions, many reputable research studies that are peer reviewed, published in journals and meet otherwise normative qualitative and/or quantitative research standards are often somewhat lacking in validity and could often be categorized as pseudo-scientific. They are just slightly less suspect larger houses of cards upon which to wager big bets of communal investments of philanthropic dollars.
In addition, according to Dr. Len Shlesinger of Harvard University, some results of even the best modern research studies are often obsolete by the time they are concluded in our modern rapidly accelerating world. He has coined a new term "creactivity" to explain how we can make decisions when most research has such a short shelf life. He maintains we can do a type of action research where we take low risk investments for short term pilot projects and see if there are any promising results in order to continue or stop a particular venture. This is more of an entrepreneurial model that presents a low risk high reward strategy that can be practically used to help make investment decisions.
In essence this means that almost all research is useless as a practical way to determine how to spend large sums of money. While Trencher and Grumet produce some trends and results using small sample based data that may be questionable from a scientific validity perspective, at least they acknowledge the shortcomings of their work. It would be minimally comforting to say we should throw out their invalid conclusions in favor of relying on what could be essentially only slightly less invalid conclusions that are "research based".
An even graver concern is some companies fund research studies that look legitimate in order to back up predetermined results as the saying goes "statistics don't lie, but liars use statistics". Since some well funded studies are often funded by a biased entity such as tobacco companies or energy companies, one could as easily cast valid aspersions on the motives and biases of those involved in some very well funded studies in existence.
The experience, ethical standing, and scholarship of the researcher is important to consider. The argument could be made that at least Grumet and Trencher are not getting paid corporate money to produce a study and are impeccable professionals of sterling character. Their research might better be viewed as scholarly opinions with anecdotal evidence to back them up. Just because they aren't what one would categorize as classical research studies doesn't mean they are irrelevant or unimportant. The fact that the analysis of the data collected was done by wise experienced practitioners gives the ideas some merit and standing regardless of their statistical precision.
It seems that nowadays the only way it is possible to gain some real insight using research is to perform meta-analysis of large groups of studies to examine trends that emerge. The magnum opus "Visible Learning" which is a ten-year project that Dr. John Hattie produced using this method is amazing, yet this entire approach is vulnerable to even more criticism from a statistical validity point of view.
One emerging approach to judge the efficacy of a research based predictability model is to perform historical analysis of current theories using computer analysis that analyzes data in hindsight to see if predicted results would have been correct in the past. Ray Dalia makes a compelling case for this approach in his recent brilliant work "Principles" that has produced billions of dollars of return for investors in his company, Bridgewater Associates.
Shalom,
Elisha Paul
Edited 1 time(s). Last edit at 02/24/2018 06:05PM by mlb.
