Download PDF: Blevins and Toler, Something to Prove?
Introduction
While there are numerous opportunities to develop new paradigms and shift existing ones in interdisciplinary research in religion and health, there are also challenges involved in such work. One key challenge presents itself immediately in this interdisciplinary (in fact, in any interdisciplinary) work: the disjunction among various scholarly fields as to what exactly “scholarly inquiry” into religion and health is. Is an economic analysis of the distinctions between faith-based and governmental health systems conducted by a health economist an example of this scholarly inquiry? How about an exegesis of the differences between Jesus’s healing miracles as recorded in the synoptic Gospels and the Gospel of John? What about an analysis of the responses to gender-based violence understood from the perspective of Shari’a law and from the perspective of post-colonial Asia? How about population-based epidemiological profile of American religious communities? Is a cultural history of American evangelical Protestantism and its influence on the American public health movement such an example? What about an analysis of religion’s role in adolescent sexual health from poststructural philosophy? Would an ethnography of the religious practices of the residents of a South African township who practice a kind of hybridity between African Traditional Religion and Christianity and the ways in which those practices impact HIV risk count as an example? How about a double-blind randomized control trial of the influence of prayer on recovery periods after surgery? In fact, all of these examples count as examples of research into religion and health, but few scholars across the humanities, social sciences, and health sciences can navigate across the theoretical disciplines employed in these fields. The challenges of such navigation complicate efforts at interdisciplinary research into religion and health. And yet, many of us interested in such interdisciplinary research are attempting such navigations, grateful for the critical and technical capacities of our own academic disciplines and also painfully aware of the limitations of those disciplines as we try to find our way.
This question of epistemology—what counts as knowledge in interdisciplinary research—is not merely a theoretical question. It also impacts practice. This paper explores this kind of question in the context of our practice as pastoral theologians who find themselves in clinical and public health contexts in which the demand of evidence-based practice creates tensions with some of the foundational assumptions of our field. We are trying to navigate between the world of quantitative outcomes and qualitative experiences. And we are trying not to lose our way.
Beyond the Need for Outcomes
As pastoral counselors and pastoral theologians, we come from an academic field and a professional discipline not often required to prove outcomes. And yet, this is the world we currently occupy in both clinical and academic settings. In the clinical context, one of us has worked as a pastoral counselor and chaplain in a private, non-profit psychiatric recovery center; funding for that position was secured through a private foundation which required evidence of the efficacy of this pastoral work. In the academic context, one of us works in the field of public health and has to develop procedures for monitoring and evaluating the effectiveness of community-level initiatives that seek to mobilize religious communities to address issues of concern to public health researchers and practitioners. In short, we have to “prove it or lose it.”
The worlds of grant writing, clinical medicine, and public health research were not ones that either of us initially understood, expected, or knew. But now, the terms and dynamics of projected outcomes, Likert Scales, measurements, samples, goals, and objectives have become central in our work. There was no course in our theological and pastoral counseling training and education that had prepared us in the slightest to negotiate the terrain we were encountering. But a lack of preparation was only the beginning; we have found that the theological and epistemological foundations that we employ in our clinical and academic work are in tension with this emphasis on measurement and outcomes. The complexity of pastoral theological reflection and practice—which includes creating pastoral therapeutic space; utilizing systematic, historical, and practical theology; building interdisciplinary connections between theological perspectives and other fields of knowledge; and empowering women and men in their individual and communal lives—has become crowded with the demands of formulating, administering, maintaining, and compiling measurement tools in order to maintain funding and satisfy grant requirements.
We want to be clear: we understand the necessity of providing such reports in order to assure that funds are being spent in responsible ways, but we nonetheless question whether a reliance on evidence-based practice and quantifiable outcome measures is sufficient for fully evaluating the impact of our pastoral work. In short, we believe this kind of assessment tells us something, but it does not tell us everything. Further, we believe that the dimensions of our work left unexamined in this approach are critically important and that an over-emphasis on evidence-based practices alone renders such dimensions invisible or irrelevant. This compromises the level of care offered in our pastoral practice or the capacity of practical theologians to be equal intellectual partners in the interdisciplinary research. This article explores the questions that have nagged us in regard to this reliance on evidence-based practice even as we have become part of clinical and academic systems where it is central and necessary. We raise these questions in three contexts as practitioners: pastoral counseling, public health programs, and practical theology.
Pastoral Counseling
Because pastoral counseling combines theological reflection with clinical practice, it is a discipline that has been directly affected by current trends in the American healthcare system—trends that revolve around questions of finances and funding. Healthcare programs rely heavily on insurance reimbursement for their income, which means that in many cases insurance companies dictate the length and kind of treatment an individual receives. In this broader context, pastoral counseling as a specialized clinical discipline faces pressure in regard to reimbursement rates from insurance, the necessity of licensure as a psychological profession (with no attendant assessment of theological competency), and the need to define the relevance and nuances of the pastoral counseling approach in contrast to other proven “secular” counseling disciplines. In such circumstances, the place of pastoral counseling and other pastoral services is exceedingly vulnerable. In order to secure their place, pastoral services have had to turn to outside grantors to obtain funding. This turn to alternative funding sources has led pastoral counseling as a discipline into the world of “evidence-supported therapies” and “outcome-driven evaluations.” Foundations want positive results for their money, and they want to know how a particular pastoral counseling program is going to prove successful.
Pastoral counseling practitioners, then, are put in the precarious position of creating categories for measurement. Pastoral counselors have to project an outcome for an individual who engages in pastoral counseling and create a scale that will measure that projected outcome. An example of this can be found in the requirements of a private foundation that was a potential funding source for a pastoral counseling program in a mental health facility where one of us worked. The grant application for this foundation specifically asked: “What impact will your program have—what will change about the situation as a result of your project? What are the goals and outcomes identified for the program?” These goals and outcomes include short-term, intermediate, and long-term projections of discernable knowledge, skills, and behaviors of individuals.
Such circumstances raise important issues. Although evidence-based practices purport to encourage or even demand practices that make a measurable, positive difference in the life of an individual, how does prioritizing evidence-based practice offer any safeguard from the biases of the practitioner in characterizing “health” or “dysfunction”? Though masked in the objectivity of numbers and scales, “health” and “dysfunction” are in fact understood in subjective contexts that are shaped in large part by the demands of the funder and the opinion and viewpoint of the pastoral counselor creating the measurement.
One of us wrote a grant to develop and manage the pastoral counseling and chaplaincy program referenced above. In that process we had to develop criteria for measuring the priorities of the foundation that provided the funding for the program. This process gave the foundation tremendous authority to influence a pastoral counseling program to align its priorities with those of the foundation. It also gave one of us, the grant writer, tremendous authority as a potential grantee to define terms central to the project such as health, healing, hope, spirituality, and dysfunction; to name the kinds of perspectives and behaviors that clients would adopt when successfully “treated”; to stipulate the number of participants who would successfully adopt these new behaviors; and to determine the activities that would constitute a “successful” pastoral program. The grant proposal listed the following objectives that were later implemented in the clinical program:
Short-term outcomes (knowledge and skills)
- Targets: 75% of participating clients will indicate increased understanding of and activities related to the search for meaning, purpose, and truth in life and beliefs and values.
- 50% will indicate increased knowledge and skills related to expressing feelings of hope, love, connection, inner peace, comfort, and support.
- 50% will exhibit increased ability to understand and articulate one’s spirituality, explore the roots and nature of their spirituality, and identify those aspects of their spirituality that hinder their recovery.
Intermediate outcomes (behaviors)
- 75% of participating clients will exhibit consistent attendance at pastoral counseling activities.
- 75% of participating clients will exhibit an established prayer/meditation practice and participation in a spiritually minded community.
- 50% of clients will exhibit increased interaction/communication with staff and peers.
Long-term outcomes (values, conditions, and status)
- 50% of clients will exhibit improvements in functionality, mental health, and quality of life related in part to participation in pastoral counseling as determined by results of long-term data.
In short, the grantor required measurable outcome measures to fund a program of spiritual support. These outcome measures attempted to quantify characteristics such as the search for meaning, purpose, and truth in life and beliefs and values; the expression of hope, love, connection, inner peace, comfort, and support; and evidence of prayer/meditation practice and participation in a spiritually minded community. We understand the need for monitoring and evaluation of funded programs, but we wonder if placing these kinds of spiritual practices on an evaluation instrument is the best way to understand the function of spirituality in a psychiatric care facility.
Public Health
Although public health research and practice are not influenced as deeply or directly by questions of insurance reimbursement and healthcare policy, research in the field is still driven in large part by demands for quantifiable outcomes and evidence-supported practices, in large part because these demands are derived from funders of that research. Although interdisciplinary in nature—public health research combines medicine, nursing, biology, chemistry, economics, political science, applied mathematics, computer science, sociology, anthropology, psychology, and other disciplines in various combinations—the ongoing debates in the field between qualitative and quantitative research methods are often eclipsed by the funding that drives public health research; that funding requires outcome measures, evidence-based practices, and criteria for monitoring and evaluation. In some areas of public health research and practice, these demands persist even in contexts in which they are impractical.
For example, one of us coordinates a community-level HIV prevention initiative in a series of informal settlements in Nairobi, Kenya. This initiative begins with an assumption that religious organizations in these settlements provide the bulk of the healthcare and social support services for the community because the national government provides little infrastructure to the settlements since the people who live there (as many as 600,000 people in this specific initiative) are not legal residents. In such a context, Christian and Muslim communities fill that vacuum with programs and services that are vital to the people in the settlements. Measuring the effectiveness of community-level prevention campaigns is notoriously difficult because so many societal and cultural factors are at play and those multiple, inter-related, complex factors cannot be controlled by the prevention campaign. Nonetheless, the funding for such campaigns often demands quantifiable measures such as randomized control trials and measurable outcomes such as a change in the incidence or prevalence of disease across a population.
In the specific project in Kenya, the primary outcome measure is a 50% reduction in HIV incidence among young people over the course of the next five years. While this is certainly a laudable goal, efforts to implement programs on the ground are hampered by the demands of this measure because there is no baseline measure for a population that the government officially refuses to acknowledge and there is no large-scale infrastructure put into place by the government or by international non-governmental organizations to compile epidemiological data. In short, public health efforts to address HIV prevention and to build collaborative partnerships with organizations on the ground (the vast majority of which are faith-based) are stymied by a singular emphasis on evidence-driven practice and on measurable outcomes.
In both of these contexts—pastoral counseling and public health—the demand for measurable outcomes has consequences for practice, either altering the provision of services or delaying implementation of services. Our concerns about the current state of affairs have not arisen simply because we are also theologians; indeed, colleagues in the fields of public health and counseling who are not theologically trained have also raised a host of concerns about this over-emphasis. We raise just a few of the most central and cogent to demonstrate the multiple problems with this kind of singular perspective.
For example, counseling psychologists have criticized a growing demand for evidence-supported psychotherapy for numerous reasons. They argue that a singular focus on treatment efficacy leaves the field unable to determine whether a positive outcome is due to the “evidence-supported treatment” or to another factor that is not ever evaluated. They question the definition of “treatment efficacy” employed by proponents of evidence-supported psychotherapy, a definition predicated on statistically significant data in regard to efficacy from two previous studies; this criterion ignores the preponderance of data from all valid studies (either demonstrating a positive effect, a negative effect, or no effect), focusing instead only on the requirement of two valid studies with positive outcomes. On this issue, critics call for a meta-analysis of all available studies. Critics of a singular emphasis on evidence-supported psychotherapy also claim that it leads to a focus on treating symptoms, giving little credence to growth-oriented therapies. Finally, these critics claim that standardized counseling interventions that are proven to be effective cannot account for the multiple variables that always influence psychotherapy such as client and therapist personalities, cultural contexts, and subjective emotions that influence the course of therapy and either facilitate or hinder progress.
In the area of public health, researchers and practitioners interested in community-level prevention initiatives argue that quantifiable outcome measures are insufficient methods for measuring program efficacy. The complex, inter-related cultural factors that are the very fabric of communities themselves cannot be measured by a gold standard of a randomized control trial. Further, efforts to develop community-level prevention programs by their very nature focus on the social drivers of risk or infection. These various social forces are bound together in a causal chain with some forces more immediately effecting behavior and others creating the broader social environments that support or discourage that behavior while having less immediate impact on the behavior itself. Successful prevention initiatives would need to address the factors up and down that chain; as such they would, by their very nature, be ill-suited for an outcome measure such as a randomized control trial because the initiatives would be working to address multiple factors at the same time.
These kinds of critiques raised by our colleagues in the fields of counseling psychology and public health are important to the extent that they offer insight into the problems with the current over-reliance on evidence-based practice and measurable outcomes. And yet, we believe that theological reflection also raises important critical questions about this current practice and that theological reflection is uniquely valuable in this context because the kind of questions it raises are qualitatively different from those raised by our colleagues.
Theology as Scholarship and Practice
Specifically, we believe there are three central theological questions to be raised in regard to the current enchantment with evidence-based practice in pastoral counseling and public health:
- Who has the power to determine the criteria for evidence?
- What are the costs of defining spiritual health as the quantifiable cessation of negative symptoms?
- Are the practices of caring about one another to be measured only in terms of improvement of one person through the skilled actions of another?
We close by briefly exploring these three questions.
Who has the power to determine the criteria for evidence?
We alluded above to the misgivings one of us felt in writing a proposal and administering a grant for a pastoral counseling and chaplaincy program when such a process allowed for such power to determine the criteria for measuring concepts such as health, hope, dysfunction, despair, meaning, love, and peace. We recognize that earnest efforts to improve health, cultivate hope, address dysfunction, lessen despair, provide meaning, risk love, and encourage peace will not all be equally effective and that responsible use of resources will entail some way of determining what kinds of practices work better in these regards. But we are struck by the lack of attention paid to the question of who gets to set the criteria for such determination. In fact, we think that a singular reliance on quantifiable measures leaves funders, researchers, and practitioners less able to reflect on the power dynamics involved in setting such criteria.
We also think that practical theologians have two particular capacities in this area that enable them to provide a corrective perspective. First, a training in contemporary theological perspectives makes us keenly aware of the potential for any effort to offer care for another human being to turn into an effort to control them and compel them to do what we want them to “for their own good.” We recognize this, at least in part, because we acknowledge just how efficient religion is in such efforts, and our theological education has taught us the importance of such self-critique. Second, and in relation to the first, our commitment to interdisciplinary scholarship and practice leaves us able to appreciate other disciplines and to learn from the scholarship and practices of colleagues. It also make us, we hope, equal partners in interdisciplinary research and practice, enabling us to raise a critical voice honed by the kind of theological self-critique we described above.
What are the costs of defining spiritual health as the quantifiable cessation of negative symptoms?
We express some strong misgivings about efforts to define efficacy solely in terms of those negative components of individual lives or of communities that are alleviated. This question of alleviating symptoms is important, but we believe that by itself it creates a broader perception that the only thing that matters is to avoid difficulties in life. In such a perspective anyone who is sad is suffering from depression and their sadness must be alleviated. But sadness may be the result of a health impulse or of a faithful response to God’s call. It may arise because we love someone who is suffering or because we recognize our own inevitable complicity in systems that lead to others’ suffering. In such instances, the response is not to do away with the sadness per se but to work to address the underlying cause of the suffering or to commit ourselves to work toward a different social order. Such efforts are not quantifiable; they’re not even counted as legitimate criteria for telling us whether our efforts are effective. But they are important, and they are commitments we are taught to make and strive to honor based on theological perspectives and practices. We believe practical theologians can offer reminders to our colleagues in other disciplines about the importance of these kinds of commitments that will have a direct impact on the way people live their lives not merely as well-adjusted individuals but as people committed to a vision of social justice.
Are the practices of caring about one another to be measured only in terms of improvement of one person through the skilled actions of another?
In relation to the second point above, we believe that certain kinds of commitments lead us to recognize our mutual dependence on one another and to commit ourselves to one another both as symptoms are alleviated or communities get healthier and as symptoms are exacerbated or communities struggle. Such a commitment is not the same as complacence. On the contrary, such a commitment works hard for positive outcomes in both individual and communal lives, but it also recognizes that these outcomes are not primary but are, rather, derivative of a deeper commitment to one another. Such commitments require us to recognize the gift of life in all of its complexity and to say yes to all of life in its wonderful, awful mixture of grace and suffering. From such a viewpoint, the criterion for effective programs of pastoral counseling or public health prevention is measured by our willingness to enter into the mix of life in all its complexity.
Conclusion
Seminary education, doctoral studies, clinical practice, and graduate-level teaching did not prepare us for the current reliance on evidence-based practices and outcome measures as the sole criteria by which to measure our work and our scholarship. Even as we have struggled to come to terms with these demands, we have come to believe that they are not the only criteria to which we must commit. In that regard, we believe that theology offers a unique perspective from which to see and appreciate other, deeper commitments. Even if we don’t have any evidence for believing it.
