Running Head: Application of EST Criteria to AT
The Application of Empirically-Supported Treatment Criteria to Adventure-Based Therapy Research: Where Do We Stand and Why Should We Care?
Sandra L. Newes, MA
Department of Psychology
Word Count: 4036 words
Correspondence regarding this
article should be addressed to: Sandra
L. Newes. Department of Psychology, The
The Application of Empirically-Supported Treatment Criteria to Adventure-Based Therapy Research: Where Do We Stand and Why Should We Care?
Adventure-based therapy (AT) is often referred to an effective treatment. However, while numerous researchers have made laudable attempts to conduct solid outcome studies of AT, the reality is that the literature shows a wide range of discrepant findings. Along with this, the inadequate methodological control that plagues this literature unfortunately leaves us with an inability to draw clear conclusions. This article is a discussion of the necessity for solid research in the AT area, within the context of the current movement toward empirically validation in the psychotherapy community. I will provide a brief synopsis of the recent historical background of this movement and address the issue of why empirical validation is relevant and perhaps alarming to the field of AT. I will also discuss the criteria necessary for a treatment to receive the “empirically supported treatment” designation, and apply this criteria to the existing AT literature.
(Keywords: adventure therapy, wilderness therapy, adventure-based therapy, research, empirically supported treatment)
The Application of Empirically-Supported Treatment Criteria to Adventure-Based Therapy Research: Where Do We Stand and Why Should We Care?
Adventure-based therapy (AT) has been used to treat a variety of presenting problems with a variety of populations, and is often referred to in the AT literature as an effective treatment. However, while numerous researchers have made laudable attempts to conduct solid outcome studies of AT, the reality is that the literature shows a wide range of discrepant findings. Along with this, the inadequate methodological control that plagues this literature unfortunately leaves us with an inability to draw clear conclusions. Further well-informed and scientifically rigorous research in AT is vital in order to meet the growing need for empirical support, a need fueled by the movement in the “mainstream” psychotherapy field towards empirically validated treatments.
This article is a discussion of the necessity for solid research in the AT area, within the context of this press toward empirically validation. Specifically, I will first provide a brief synopsis of the recent historical background of this movement. I will then address the issue of why empirical validation is relevant to the field of AT, and how the current movement towards such treatments may be cause for alarm given the present state of the AT literature as a whole. Finally, I will discuss the criteria necessary for a treatment to receive the “empirically supported treatment” designation, and apply this criteria to the existing AT literature.
As an aside, I’d like to preface this article by saying that as I’ve written this article and talked with other AT professionals, I’ve become increasingly aware that what I have to say might not be very popular. In fact, I have been advised by concerned others to re-write it, to “soften it up”. As my current position in my professional life finds me with one foot immersed in the AT field and the other foot immersed in a very “mainstream” and empirically focused clinical psychology program, this provides me with a unique conflict.
some background, my department is a bastion of psychotherapy outcome
On the other hand, I completed a dissertation looking at predictors of success in an AT program. As a person actively involved in AT research, I am intimately aware of the difficulties inherent in conducting a study in a field as diverse and varied as AT. As such, I truly commend the efforts of those who have contributed to the present body of research, and must note that I intend this article to in no way be an attack on the existing literature.
Perhaps the best context in which I can hope this article is to be received is that of the raising of an alarm, or a concerned plea. The concerns I bring forth are grounded only in the recognition and belief on my part that there is only the smallest possibility the adventure therapy field will remain immune from the pressure brought to bear on the psychotherapy community by the movement towards empirically validated treatments. While this pressure may not come immediately, I am certain that it will and that we must be prepared for this eventuality. Therefore, given this belief, I feel a discussion of the empirical basis for claims of treatment efficacy in AT is both important and necessary.
The Empirically-Supported Treatment Movement: What is it?
The movement towards empirically validated, or empirically supported, treatments has its present day roots in the completion of the American Psychological Association (APA) Division 12 Task Force on Promotion and Dissemination of Psychological Procedures (1995) and the APA Task Force on Psychological Intervention Guidelines (1995). In general terms, these reports were a call for empirical validation of specific treatments in psychotherapy, a listing of those treatments that could be considered as having achieved such status, and a putting-forth of the research-based criteria necessary for the recognition of a treatment as being “empirically-validated”. Since their publication, the list of empirically validated treatments has been continuing to grow in length (Dobson & Craig, 1998).
Building on the foundations put forth by the Task Force, Chambless and Hollon (1998) modified this criteria, creating a new designation of “empirically-supported treatment”, or EST (Chambless & Hollon, 1998). As an aside, it is this designation that this article will be focused on, as it is provides categories indicating the establishment of empirical support at differing levels of specificity (i.e., possibly efficacious, efficacious, efficacious and specific). Such categories indicate progressively more stringent levels of established empirical evidence, and can provide categorical markers by which to evaluate future AT research.
Within the psychological community, the publication of these documents has led to an increased recognition of the necessity for specific treatments to establish proven efficacy. This has also led to increased awareness of the conditions required to conduct such research in a scientifically-based fashion. In addition, proponents of those treatments currently lacking the necessary research to claim empirical validation are now facing heightened pressure to become more involved in such efforts.
Simply put, therapists are facing increased demands to use treatments that have been empirically proven to work. In this context, the underlying question becomes “if X treatment has been proven to work and Y treatment hasn’t, how can the continued use of Y treatment be justified?” While this climate may not as yet have pervaded the adventure-based therapy community, it seems unlikely that AT will not eventually become subject to the same pressures. The following section is an elaboration of the issues associated with the movement towards empirical validation, and how this may impact the AT field as a whole.
Empirical Validation: Why Should We Care ?
The heart of this argument is that I believe the time is liable to come when AT professionals may need to prove the efficacy of their treatment in order to defend its continued use. Unfortunately, this pressure is likely to come from such unexpected yet powerful sources as insurance companies, government funding agencies, insurers, liability evaluations, and future clients themselves. Thus, as opposed to this being strictly a philosophical argument, there is the potential that this issue could carry with it tangible consequences for the AT field on a much more pragmatic level.
Realistic concerns about this issue are raised continually among “mainstream” psychotherapists. Will treatments that do not show empirical validation or support continue to receive third party payment? Will government agencies continue to refer clients to such treatment, and will they continue to pay for it? Will there be legal and ethical ramifications for those who provide such non-validated treatments? Should the providers of such treatments be given malpractice insurance at the same rates as those who provide treatments that are empirically validated? What if someone provides treatment that is not empirically validated or supported in place of one that is, and there is a negative consequence or outcome for the client? Is the provider responsible? In what way?
Questions such as these remain as yet unanswered, and it is likely that many of the impacts of the movement towards empirically validated treatment are unforeseen as of now. As distasteful an idea as it may appear, it is a real possibility that at some point in the future treatment providers may become limited in their ability to provide treatments that are not recognized as empirically validated. This is an ongoing and hotly debated issue, one that has permeated the larger psychotherapy community in recent years.
Interestingly, many in the AT field seem to perceive AT as a separate treatment modality, one that operates as independent from the larger psychotherapy culture. With such a belief comes the idea that AT will not be subjected to the same type of scrutiny as are other forms of psychotherapies. In discussions of such concerns with AT practitioners, statements such as these seem to underlie much of the dialogue: “This treatment works. Kids get better. Why do I have to prove it? Who do I have to prove it to? Don’t we already have research that shows that this works?”
While in fact it may be true that AT works, the reality is that currently there is limited empirical evidence that can lend support to such statements. I will expand on this contention in the latter portion of this article. Simply put, although adventure therapy may well be an exceptional treatment modality, without solid empirical evidence we have no way to back up such a declaration. Should the attention of the psychological community become focused on adventure therapy, we have limited ability to defend our treatment of choice or ourselves as practitioners. This is likely to become particularly salient in any of the rare instances of client harm or death, as this brings not only the attention of the psychological community but also the focused attention of the general public as well.
Overall, while it may be true that AT is fairly distinct, assuming that AT will remain free from such pressures, particularly given the reliance of many programs on government referral and government funds, could be dangerous and carry with it very real costs. While I recognize that this may be perceived as overly catastrophic, it is important to consider this realistically as such costs could include the loss of funding sources for our programs, the ceasing of third party payment, the loss of referrals, the withdrawal of liability and malpractice insurance policies, and perhaps other as of now unforeseen ethical and legal consequences as well.
Ideally, none of this will come to pass. Critics of the empirical validation movement are espousing the dangers of forced and exclusive adherence to such treatments (Garfield, 1996; Silverman, 1996), and it is likely that some sort of balance will be achieved. However, this movement is not going to go away. Even with the recognition of the value of non-validated treatments, the possibility remains that there could be a significant impact on the AT field.
Given that this field is often viewed as being “on the fringes” by the majority of mainstream psychotherapists, we are particularly vulnerable and thus AT practitioners must increase their awareness of these concerns. In addition, the AT field must face the reality that, in order to continue indefinitely as a treatment modality, particularly one that receives funding from anything other than private payment, there must be more solid and scientifically-based research that shows that it works. The same concerns can be applied to individual programs as well, as they may also become subject to the same pressures.
Ironically, while an increase AT research (or any other treatment for that matter) may of necessity be motivated by external pressures, there is a very real silver lining. Though it may take some effort to conduct such studies, scientific evaluation of AT can only help us to improve our treatment as a whole. By examining what works and what doesn’t, determining and evaluating the mechanisms of change, and testing elements of theory in adventure therapy, we will ultimately be able to improve on our treatment. Regarding this issue, Borkovec and Castonguay (1998) note:
“The identification of increasingly specific cause-and-effect relationships leads to better theoretical understanding of the nature of the psychological problems being treated and the nature of the mechanisms of change underlying any demonstrated causative roles for a therapy, its elements, its parameters, or elements added to it. From this knowledge, hypotheses about modifications or additions to a therapy emerge and can be tested.” (p. 139).
Thus, through an ongoing process of evaluation, we will eventually be able to provide the best treatment possible. Such is hopefully our goal.
In the next section of this article, I will briefly delineate the criteria necessary for the designation of “empirically-supported treatment
Empirically Supported Treatment: What is it?
Specific guidelines have been established for determining whether or not a treatment has received the empirical validation necessary to support claims of efficacy. This section will review these criteria, and the following section will be focused on a discussion of the existing AT research within this context.
Chambless and Hollon (1998) provide a structure designed to guide these evaluations, defining empirically supported treatments (ESTs) as “clearly specified psychological treatments shown to be efficacious in controlled research with a delineated population.” (p. 7). They also ask researchers to “consider the following broad issues about ESTs in their area:
a) Has the treatment been shown to be beneficial in controlled
b) Is the treatment useful in applied setting and, if so, with what patients and under what circumstances?
c) Is the treatment efficient in the sense of being cost-effective relative to other alternative interventions?” (p.7).
Researchers are able to answer such questions by performing evaluations of treatment efficacy (which includes clinical significance), effectiveness (i.e., clinical utility), and efficiency (i.e., cost effectiveness) (Chambless & Hollon, 1998).
Chambless and Hollon (1998) also provide more narrow and specific criteria that must be met in order for a treatment to receive the designation of EST. While such criteria have been published elsewhere (Crits-Cristoph, 1998), they will be listed here to provide background context for the discussion that follows. These EST criteria are:
1) “Comparisons with a no-treatment control-group, alternative treatment group of placebo (a) in a randomized clinical trial, controlled single case experiment, or equivalent time samples design and (b) in which the EST is statistically significantly superior to no treatment, placebo, or alternative treatments in which the EST is equivalent to a treatment already established in efficacy and power is sufficient to detect moderate differences.
2) These studies must have been conducted with (a) a treatment manual or its logical equivalent; (b) a population treated for specified problems, for whom inclusion criteria have been delineated in a reliable, valid manner; (c) reliable and valid outcome assessment measures which tap at minimum problems targeted for change; and (d) appropriate data analysis.
3) For a designation of efficacious, the superiority of the EST must have been shown in at least 2 independent research settings (sample size of three or more in the case of single-case experiments). If there is conflicting evidence, the preponderance of the well-controlled data must support the ESTs efficacy.
4) For a designation of possibly efficacious, one study with similar characteristics as above suffices in the absence of conflicting evidence.
5) For a designation of efficacious and specific, the EST must have been shown to be statistically superior in at least 2 research settings. If there is conflicting evidence, the preponderance of the well-controlled data must support the ESTs efficacy and specificity.” (Chambless & Hollon, 1998).
While solid research design is assumed in these criteria, Borkovec (1994) discusses in more detail additional important aspects of design which are relevant to an evaluation of any treatment, specifically client considerations, therapist considerations, and dependent variable considerations. As the scope of this article precludes a complete discussion of these methodological considerations, these are to be addressed in another article.
In summarizing this evaluation process, Chambless & Hollon (1998) note that “we use as our starting point [in the discussion of ESTs] the position that treatment efficacy must be demonstrated in controlled research in which it is reasonable to conclude that benefits observed are due to the effects of the treatment and not to chance or confounding factors such as the passage of time, the effects of psychological assessment, or the presence of different types of clients in the various treatment conditions.” (p. 7). It is recognized that these criteria are quite difficult to meet, and thus the receipt of the EST designation for AT may appear to be a daunting task. However, it is one that certainly can be accomplished using our cumulative acumen and ability
The following section is a discussion of some of the challenges that will need to be overcome in order for AT to achieve such status. This section will first focus on a discussion of the methodological errors in the existing AT research, and then compare the existing research to the standards for EST designation.
Empirical Support: Do We Have It?
While it is tempting to conclude from the existing AT research that AT is an effective treatment, when the literature is examined closely the reality is that AT does not meet any of the previously stated criteria necessary for EST designation. Unfortunately, a critical view from the perspective of psychotherapy outcome research reveals that the majority of studies suffer from severe methodological difficulties. In fact, much of the AT research contains flaws such that criterion specific to all psychotherapy research are threatened, deficient, or as yet have not been considered (for an outstanding review of these criteria, see Borkovec, 1994).
Basic methodological considerations including random assignment, standardization, appropriate instrumentation, equivalency across groups, client variables, design considerations, suitable comparisons, and adequate analytical strategies are regularly violated in the AT literature (Bandoroff, 1989; Davis-Berman & Berman, 1994 Gillis, 1992; Gillis & Thomsen, 1996; Hattie, Marsh, Neill, & Richards, 1997; Herbert, 1998; Newes, 2000). Therefore, such studies are rendered scientifically useless in the strict evaluation of adventure-based therapy necessary for empirical validation. As aforementioned, this leaves the conclusions that can be reached from such research limited at best. Simply put, without adequate research methodology that involves appropriate levels of scientific control, alternative explanations (rival hypotheses) for results cannot be been ruled out. This is a central goal in treatment outcome studies, and studies that do not meet this goal must be considered as invalid (Borkovec, 1994).
To illustrate this contention, I will discuss the AT research with regards to several of the key points embedded in the EST criteria. The first most basic design consideration is random assignment, and many of the AT studies do not employ this basic design necessity and often they do not employ appropriate comparison groups. The second consideration is replication, but without standardization across programs and more tightly controlled samples, such replication may be impossible to achieve.
Other considerations put forth by Chambless and Hollon (1998) are based on sample characteristics and measurement considerations. They note that it is essential that samples be clearly defined in order to achieve tighter control. Unfortunately, there are no data available on AT with specific diagnostic groups, and often studies do not report such basic information as sample means and standard deviations on fundamental characteristics such as age. It is important that the research in this area begin assessing pre-participation levels of pathology and other relevant characteristics. More specific reporting of sample characteristics is fairly simple to provide in most studies and is necessary in order to being answering the relevant question of what client is this type of treatment effective for.
Chambless and Hollon (1998) also caution that researchers must consider the negative aspects of treatment. I am unaware of any existing AT studies in which there was a suggestion of a possible negative impact of treatment, and this subject is rarely broached in the AT literature. Without future well-controlled research, research also open to examining potential negative effects of AT, it remains a possibility that aspects of this treatment have the potential for psychological harm. While this may be an unsavory proposition, the potential must be explored in order to rule out the possibility. Ideally, if negative impacts are found, the negative elements can then be isolated and removed from the program. This process will ultimately contribute to improving the treatment overall.
Finally, Chambless and Hollon (1998) discuss the idea of cost-effectiveness, noting that if there are no differences in outcome, treatments that cost the least are likely to be preferred. While I believe such a study may have been undertaken, as of this writing there are no published data available on cost-effectiveness for AT as compared to other treatments. Although the provision of this information may on the surface appear to be fairly straightforward, without solid research there are limited conclusions regarding treatment outcome upon which to base such comparisons. Thus, empirically-based discussion of cost-effectiveness in AT becomes difficult.
As can be seen from this discussion, AT is nowhere near reaching an EST designation. Importantly, without concerted and well-controlled efforts in this area, AT is unlikely to ever achieve such status. As I have stated previously, this is not to say that AT does not work. Rather, in the growing context of the movement towards empirical validation of specific treatments for specific disorders (and the related era of managed care), it is imperative that we engage in such efforts in order to provide the required empirical proof. Unfortunately, should external agencies become involved in this issue prior to our compilation of such data, AT as a treatment modality may be pushed into a forced and unwelcome decline. Alternatively, should external agencies (i.e., insurance companies) begin to compile such data independent of our input, we will have lost control of the process. This could potentially result in an inaccurate, and possibly detrimental, representation of AT outcome.
Paralleling this concern is the fact that, according to the American Psychological Association (APA) ethical standards (1992), psychotherapists have an ethical responsibility to use treatments that work. As ongoing empirical validation has currently established that indeed there are treatments that do “work”, this issue may become even more salient for therapists working in the adventure field. Realistically, without further well-designed research on AT outcome, the time could come when an adventure therapist could potentially be brought up on ethical charges for using such an “unproven” treatment, particularly in the instance of a less than positive outcome. Any such instance would set an unavoidably disastrous precedent. Regrettably, any increased ethical responsibility is likely to come with heightened legal responsibility as well, leaving the AT field increasingly vulnerable to legal repercussions.
As we are all aware, there is a seemingly unlimited amount of anecdotal evidence in support of AT. Practitioners of AT often see first hand the power of this type of treatment modality to therapeutically impact our clients, and success stores can be commonly heard. Unfortunately, our very real belief in the power of this treatment, our true stories of amazing growth and change, and our passionate assertions that AT works are meaningless to those outside of the field who require empirical data to back such claims. As such, therapists using such adventure-based treatments could ultimately be forced to present a justification for their continued use of an unproven treatment modality. In fact, Chambless and Hollon (1998), in speaking generally of the need for controlled clinical trials of specific forms of treatment, caution that “the time is rapidly approaching when unsystematic clinical impressions will no longer suffice to document a treatments value, particularly when alternative treatments such as the pharmacotherapies exist that have been subjected to more rigorous empirical scrutiny.” (p. 16).
At the risk of redundancy, I will take one last opportunity to urge those in the AT field to consider both the reality that such a time may not be far and the potential impact this could have on all who are involved in the provision of AT services. Clearly, in order to combat the external pressures the AT field will unavoidably face, the completion of well-controlled, scientifically-informed research is vital. It is only from such evaluations that we can draw the empirically-based conclusions required to lend support to our continued use of this unique and powerful treatment modality. Ultimately, I hope that it can be seen that any point I’ve tried to make is really as simple as this: If we believe in what we do, we must prove it. If we are to prove it, we must prove it soon.
American Psychological Association (1992). Ethical standards of psychologists and code of conduct. American Psychologist, 47, 4-21.
American Psychological Association Task Force on
Psychological Intervention Guidelines. (1995).
Template for developing guidelines:
Interventions for mental disorders and psychological aspects of
Bandoroff, S. (1989). Wilderness adventure-based therapy for delinquent and pre-delinquent youth: A review of the literature. (ERIC Document Reproduction Service No. ED 377 428).
Borkovec, T.D. (1994). Between-group therapy outcome research: Design and
methodology. In L. S. Onken & J. D. Blaine (Eds.), NIDA Research Monograph
Borkovec, T.D. & Castonguay, L.G. (1998). What is the scientific meaning of empirically supported therapy? Journal of Consulting and Clinical Psychology, 66(1), 136-142.
Crits-Cristoph, P. (1998). Training in empirically validated treatments. In K.S. Dobson, & K.D. Craig, (1998). Empirically supported therapies: Best practice in professional psychology.
Davis-Berman, J. & Berman, D. (1994). Wilderness therapy.
Dobson, K.S. & Craig, K.D. (1998). Empirically supported therapies: Best practice in professional psychology.
Gillis, H.L. (1992). Therapeutic uses of adventure-challenge-outdoor-wilderness: Theory and research, 35-47. Paper presented at the meeting of the Association for Experiential Education.
Gillis, H.L. & Thomsen, D. (1996). A research update (1992-1995) of adventure-based therapy: Challenge activities and ropes courses, wilderness expeditions and residential camping programs. [On-line] Available: http://fdsa.gcsu.edu:6060/lgillis/
Hattie, J., Marsh, H.W., Neill, J.T., & Richards, G.E. (1997). Adventure education and Outward Bound: Out of class experiences that make a lasting difference. Review of Educational Research, 67(1), 48-87.
Herbert, J.T. (1998). Therapeutic effects of participating in an adventure-based therapy program. Rehabilitation Counseling Bulletin, 41(3), 201-216.
Newes (2000). Adventure-Based Therapy: Theory, Characteristics, Ethics, and Research. Unpublished manuscript, The Pennsylvania State University.
Silverman, W.H. (1996). Cookbooks, manuals, and paint-by-numbers: psychotherapy in the 90’s. Psychotherapy, 33, 207-215.