“TEST, LEARN, AND LISTEN”: THE INSUFFICIENCY OF RANDOMIZED CONTROLLED TRIALS IN PUBLIC POLICYMAKING

Taufiqurrahman Taufiqurrahman; Arga Pribadi Imawan; Agus Wahyudi

doi:10.22146/jf.103353

“TEST, LEARN, AND LISTEN”: THE INSUFFICIENCY OF RANDOMIZED CONTROLLED TRIALS IN PUBLIC POLICYMAKING

https://doi.org/10.22146/jf.103353

Taufiqurrahman Taufiqurrahman^(1*), Arga Pribadi Imawan⁽²⁾, Agus Wahyudi⁽³⁾

(1) Universitas Gadjah Mada
(2) Universitas Gadjah Mada
(3) Universitas Gadjah Mada
(*) Corresponding Author

Abstract

Evidence-based policy (EBP), which advocates for grounding policy decisions in the best available evidence, traditionally adopts a hierarchy that prioritizes evidence from randomized controlled trials (RCTs) as the most robust, while placing expert testimony at the lower end. This paper argues that while randomized controlled trials (RCTs) are a valuable source of empirical evidence for policy interventions, they are not sufficient on their own to support evidence-based policy formulation. Through the lens of the INUS framework of causation, we demonstrate that the effectiveness of a policy is influenced by a complex interplay of contextual factors, which RCTs alone cannot capture. We advocate for the integration of contextual and qualitative knowledge, including testimonies from experts and community members, to supplement RCT findings. This additional knowledge provides insights into the social, cultural, and subjective dimensions of the target population, addressing motivations, preferences, and other factors that can significantly impact policy success. By comparing reductionist and non-reductionist perspectives on the use of testimony in evidence-based policy, we argue for a balanced approach that values credible testimonies as essential to understanding context. Ultimately, this paper underscores the importance of a multifaceted evidence approach in crafting effective, context-sensitive public policies.

Keywords

evidence-based policy, randomized controlled trials, testimony

References

Adams, J., & Hillman, M. (2001). The risk compensation theory and bicycle helmets. Injury Prevention, 7(2), 89–91. https://doi.org/10.1136/ip.7.2.89

Baron, J. (2018). A Brief History of Evidence-Based Policy. The ANNALS of the American Academy of Political and Social Science, 678(1), 40–50. https://doi.org/10.1177/0002716218763128

Birkland, T. A. (2020). An Introduction to the Policy Process Theories, Concepts, and Models of Public Policy Making (5th ed.). Routledge.

Brooks, G., Burton, M., Cole, P., Miles, J., Torgerson, C., & Torgerson, D. (2008). Randomised controlled trial of incentives to improve attendance at adult literacy classes. Oxford Review of Education, 34(5), 493–504. https://doi.org/10.1080/03054980701768741

Buck, S., & McGee, J. (2015). Why Government Needs More Randomized Controlled Trials: Refuting the Myths.

Canadian Task Force on the Periodic Health Examination. (1979). The Periodic Health Examination: Task Force Report. Canadian Medical Association Journal, 121(9), 1193–1254.

Cartwright, N. (2012). Presidential Address: Will This Policy Work for You? Predicting Effectiveness Better: How Philosophy Helps. Philosophy of Science, 79(5), 973–989. https://doi.org/10.1086/668041

Cartwright, N., & Hardie, J. (2012). Evidence-Based Policy: A Practical Guide to Doing It Better. Oxford University Press.

Cartwright, N., & Stegenga, J. (2011). A Theory of Evidence for Evidence-Based Policy (P. Dawid, W. Twining, & M. Vasilaki, Eds.). British Academy. https://doi.org/10.5871/bacad/9780197264843.003.0011

Cowen, N., Virk, B., Mascarenhas-Keyes, S., & Cartwright, N. (2017). Randomized Controlled Trials: How Can We Know “What Works”? Critical Review, 29(3), 265–292. https://doi.org/10.1080/08913811.2017.1395223

Deaton, A., & Cartwright, N. (2018). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine, 210, 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005

Dorsch, M. M., Woodward, A. J., & Somers, R. L. (1987). Do bicycle safety helmets reduce severity of head injury in real crashes? Accident Analysis & Prevention, 19(3), 183–190. https://doi.org/10.1016/0001-4575(87)90002-9

Dye, T. R. (2013). Understanding Public Policy (14th ed.). Pearson.

Edwards, P., Arango, M., Balica, L., Cottingham, R., El-Sayed, H., Farrell, B., Fernandes, J., Gogichaisvili, T., Golden, N., Hartzenberg, B., Husain, M., Ulloa, M. I., Jerbi, Z., Khamis, H., Komolafe, E., Laloë, V., Lomas, G., Ludwig, S., Mazairac, G., … Yutthakasemsunt, S. (2005). Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury—outcomes at 6 months. The Lancet, 365(9475), 1957–1959. https://doi.org/10.1016/S0140-6736(05)66552-X

Gamoran, A. (2018). Evidence-Based Policy in the Real World: A Cautionary View. The ANNALS of the American Academy of Political and Social Science, 678(1), 180–191. https://doi.org/10.1177/0002716218770138

Gelman, A. (2018). Benefits and limitations of randomized controlled trials: A commentary on Deaton and Cartwright. Social Science & Medicine, 210, 48–49. https://doi.org/10.1016/j.socscimed.2018.04.034

Goldacre, B. (2011). How can you tell if a policy is working? Run a trial. The Guardian. https://www.theguardian.com/commentisfree/2011/may/14/bad-science-ben-goldacre-randomised-trials

Haynes, L., Service, O., Goldacre, B., & Torgerson, D. (2012). Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2131581

Hayward, G. C., & Kirst, M. W. (2002). What have we learned about class size reduction in California? https://edpolicyinca.org/publications/what-have-we-learned-about-class-size-reduction-california

Howlett, M., & Giest, S. (2015). Policy Cycle. In International Encyclopedia of the Social & Behavioral Sciences (pp. 288–292). Elsevier. https://doi.org/10.1016/B978-0-08-097086-8.75031-8

Hume, D. (2007). An Enquiry concerning Human Understanding. Oxford University Press.

Ichikawa, J. J., & Steup, M. (2024). The Analysis of Knowledge. In E. N. Zalta & U. Nodelman (Eds.), The Stanford Encyclopedia of Philosophy (Fall). https://plato.stanford.edu/archives/fall2024/entries/knowledge-analysis/

Kemm, J. (2006). The limitations of ‘evidence‐based’ public health. Journal of Evaluation in Clinical Practice, 12(3), 319–324. https://doi.org/10.1111/j.1365-2753.2006.00600.x

Khosrowi, D., & Reiss, J. (2019). Evidence-Based Policy: The Tension Between the Epistemic and the Normative. Critical Review, 31(2), 179–197. https://doi.org/10.1080/08913811.2019.1688520

Knox, V., Hill, C. J., & Berlin, G. (2018). Can Evidence-Based Policy Ameliorate the Nation’s Social Problems? The ANNALS of the American Academy of Political and Social Science, 678(1), 166–179. https://doi.org/10.1177/0002716218769844

La Caze, A., & Colyvan, M. (2017). A Challenge for Evidence-Based Policy. Axiomathes, 27(1), 1–13. https://doi.org/10.1007/s10516-016-9291-5

Le Grand, J. (2003). Motivation, Agency, and Public Policy. Oxford University PressOxford. https://doi.org/10.1093/0199266999.001.0001

Leonard, N. (2023). Epistemological Problems of Testimony. In E. N. Zalta & U. Nodelman (Eds.), The Stanford Encyclopedia of Philosophy (Spring).

Lewis, D. (1973). Causation. The Journal of Philosophy, 70(17), 556–567.

Lisciandra, C. (2020). Reflections on the 2019 Nobel Memorial Prize Awarded to Banerjee, Duflo, and Kremer. Erasmus Journal for Philosophy and Economics, 13(1). https://doi.org/10.23941/ejpe.v13i1.487

Littell, J. H., & White, H. (2018). The Campbell Collaboration. Research on Social Work Practice, 28(1), 6–12. https://doi.org/10.1177/1049731517703748

Mackie, J. L. (1965). Causes and Conditions. American Philosophical Quarterly, 2(4), 245–264.

Menzies, P., & Beebee, H. (2024). Counterfactual Theories of Causation. In E. N. Zalta & U. Nodelman (Eds.), The Stanford Encyclopedia of Philosophy (Summer).

Montuschi, E. (2009). Questions of Evidence in Evidence-Based Policy. Axiomathes, 19(4), 425–439. https://doi.org/10.1007/s10516-009-9085-0

Moore, M. S. (2009). The Counterfactual Theory of Causation. In Causation and Responsibility: An Essay in Law, Morals, and Metaphysics (pp. 391–425). Oxford University Press.

Mosteller, F. (1995). The Tennessee study of class size in the early school grades. The Future of Children, 5(2), 113–127.

O’Brien, D. (2024). Hume and the Epistemology of Testimony. In S. Stapleford & V. Wagner (Eds.), Hume and Contemporary Epistemology (pp. 315–330). Routledge. https://doi.org/10.4324/9781003394129-26

Parsons, W. (2002). From Muddling Through to Muddling Up - Evidence Based Policy Making and the Modernisation of British Government. Public Policy and Administration, 17(3), 43–60. https://doi.org/10.1177/095207670201700304

Pearce, W., & Raman, S. (2014). The new randomised controlled trials (RCT) movement in public policy: challenges of epistemic governance. Policy Sciences, 47(4), 387–402. https://doi.org/10.1007/s11077-014-9208-3

Pérez-González, S. (2024). Evidence of mechanisms in evidence-based policy. Studies in History and Philosophy of Science, 103, 95–104. https://doi.org/10.1016/j.shpsa.2023.11.006

Peters, B. G. (2010). American Public Policy: Promise and Performance (8th ed.). CQ Press.

Petrosino, A., Turpin-Petrosino, C., Hollis-Peel, M. E., & Lavenberg, J. G. (2013). “Scared Straight” and other juvenile awareness programs for preventing juvenile delinquency. Cochrane Database of Systematic Reviews. https://doi.org/10.1002/14651858.CD002796.pub2

Robinson, D. L. (2006). No clear evidence from countries that have enforced the wearing of helmets. BMJ, 332(7543), 722.2-725. https://doi.org/10.1136/bmj.332.7543.722-a

Sackett, D. L. (1989). Rules of Evidence and Clinical Recommendations on the Use of Antithrombotic Agents. Chest, 95(2), 2S-4S. https://doi.org/10.1378/chest.95.2_Supplement.2S

SIGN. (2011). SIGN 50: A guideline developer’s handbook.

Smith, A. F. M. (1996). Mad Cows and Ecstasy: Chance and Choice in an Evidence-Based Society. Journal of the Royal Statistical Society. Series A (Statistics in Society), 159(3), 367. https://doi.org/10.2307/2983324

Thompson, R. S., Rivara, F. P., & Thompson, D. C. (1989). A Case-Control Study of the Effectiveness of Bicycle Safety Helmets. New England Journal of Medicine, 320(21), 1361–1367. https://doi.org/10.1056/NEJM198905253202101

Wasserman, R. C., Waller, J. A., Monty, M. J., Emery, A. B., & Robinson, D. R. (1988). Bicyclists, helmets and head injuries: a rider-based study of helmet use and effectiveness. American Journal of Public Health, 78(9), 1220–1221. https://doi.org/10.2105/AJPH.78.9.1220

Weightman, A., Ellis, S., Cullum, A., Sander, L., & Turley, R. (2005). Grading evidence and recommendations for public health interventions: developing and piloting a framework.

DOI: https://doi.org/10.22146/jf.103353

Article Metrics

Abstract views : 58

Refbacks

There are currently no refbacks.

Jurnal Filsafat Indexed by:

Jurnal Filsafat ISSN 0853-1870 (print), ISSN 2528-6811 (online)

Username
Password
Remember me