A Measurement Tool for Assessing the Effects of Adventure-based Programs on Outcomes for Youth-at-Risk Participants

Contents

• Abstract
• General description
• Evaluating youth development outcomes
• Youth development objectives
• Scales & Items
• Important decisions to be made
• References

Downloads

• YARPET package (.zip)
  • instrumentation
    • self-report
    • observer
  • administration guidelines
    • briefing staff
    • briefing students
  • data entry
    • instructions
    • excel templates
  • data analysis
    • notes
    • sample SPSS syntax

Abstract

The rationale, development, and items for 17 typical youth development objectives of youth-at-risk adventure-based or experiential interventional programs are presented.  The underlying intention of the instrument design was to provide an administratively efficient and psychologically valid method for assessing typically targeted youth development objectives for adventure-based youth at risk programs.  The measured objectives emerged from consultation with numerous youth-at-risk programs.  Sixty-five self-report measurement items are proposed to to assess the 17 youth development objectives (approximately 3 to 5 items per scale) in Personal, Social and Environmental domains.  In addition, observer items have been developed.  The style of the items and the measurement scale derives from the Life Effectiveness Questionnaire ( Neill, Marsh, & Richards, 1997), an instrument which has been widely used to measure the effectiveness of life skill education programs, particularly in outdoor education.  It is recommended that potential users select from the 17 measured objectives in order to reduce the overall number of items and to customise the instrument so that its relevant to their program goals.  Potential users should also consider that approximately half the scales have not been previously psychometrically tested, although their face validity appears reasonable.  It is also requested that potential users read the conditions of use.

General description of the YAR-PET

The Youth at Risk Program Evaluation Tool (YAR-PET) consists of a set of 17 proposed factors and 65 measurement items designed to assess life skill and personal, social, community, and environmental development constructs typically targeted by intervention programs with youth 'at risk'.  The measured objectives emerged from consulting with youth-at-risk programs about commonly sought outcomes and build upon previous constructs from the Life Effectiveness Questionnaire and the Review of Personal Effectiveness.  Self-report and observer assessments can be gathered at least before an after an intervention program.  Empirical analysis can then be used to estimate the degree to which intervention program participants may have experienced change with regard to the desired outcomes.  Users are encouraged to select from amongst the factors and to consider developing new factors in order to customise the measurement to match specific program objectives.  Limitations of the instrument include a lack of reported psychometric properties, thus reliability and validity is yet to be demonstrated.

Introduction: Evaluating Youth Development Outcomes

Measuring youth development outcomes in outdoor education

Good instrumentation is vital for good quality empirical research (Neill, 2003).  Designing valid measurement of psycho-social constructs, particularly amongst youth at risk, has attracted considerable attention, but relatively little methodology and measurement development has been directly applied to experiential and outdoor education settings.  In other words, there are some useful research tools (e.g., Dahl & Reed, 1999), but none that are ideally suited to immediately measure all the objectives identified by any particular youth-at-risk adventure-based intervention program. 

For example, Russell (2002) used the Youth Outcomes Questionnaire (Y-OQ) in his longitudinal assessment of seven outdoor behavioral health programs (which ran wilderness-based programs of between 21 and 56 days in  length).  Advantages of the Y-OQ include that self-report and parent observer versions exist, the psychometric development has been solid, and that clinically comparable norms are available.  Downsides include the cost (it is a commercial instrument), its length, and that it lacks customizability for adventure-based youth-at-risk interventions.

Of course, no instrument is perfect, particularly when it is applied to a variety of circumstances.  The current instrument is designed to provide a low-cost, adventure-oriented, user-friendly instrument for program evaluation and research into youth-at-risk programs which target typical youth development outcomes.  Some other possible instruments and places to search are listed on the Tools, Instruments, and Questionnaires page.

Holistic range of youth development objectives

Experiential intervention programs for youth at risk typically target a holistic range of objectives.  The proposed items are designed for use in longitudinal research and evaluation designs which aim to determine the degree to which a range of typical youth development objectives are achieved by outdoor and adventure intervention programs for youth at risk.

Guiding principles for this evaluation tool and system

The evaluation tool derives from a program evaluation system that has emerged from the Life Effectiveness Questionnaire project.

This new tool aims to provide for a rigorous and reasonably comprehensive evaluation of the extent to which program objectives are achieved in a cost and time efficient manner. 

Most youth-at-risk programs are conducted by leaders and administrators who do not have a background in research, but who are skilled in report writing and knowledge about their own program.  Thus, as long as program operators are supported in selecting an appropriate evaluation methodology, and with good quality data analysis, they can construct good quality reports to funders or evaluators, or for internal program development.   

This instrumentation is also designed to be user-friendly for participants, program instructors, and administrators.

The self-report tool can be administered pre- (Time 1) and post-program (Time 2) to get a measure of the short-term program impact.  It is recommended, however, that baseline and followup assessments are obtained, as well as observer measures (e.g., by leaders, parents, teachers, and/or peers).

This evaluation tool is also designed to be customizable.  Program evaluators too often use full instruments, when only part of the instrument is relevant to program objectives, for fearing of undermining the overall validity of the questionnaire.  With this instrument, the goal is to get the best of both worlds - psychometric validity and flexibility.  Evaluators can select from the pool of 17 scales, to develop a set of items for evaluating the effects of a particular program.

Developing the list of objectives and the item pool

An initial youth development objectives was developed through consulting with a range of youth-at-risk programs and outdoor education organizations.  In addition, current instruments (e.g., the Life Effectiveness Questionnaire) and lists of youth development objectives (e.g., the Search Institute's 40 Developmental Assets) often referred to by adventure-based youth-at-risk programs were consulted.  Objectives most commonly cited by youth at risk programs were emphasized in selecting the objectives for measurement.

Electronic database searches were then conducted for existing instrumentation which could be adapted.  Where suitable instrumentation was not identified, new items were constructed.  Note that where there was existing instrumentation, three items per scale are recommended.  For new items, five items per scale are used with the intention of refining this to three items per scale when sufficient data is available for psychometric analysis.  Potential users should view the proposed tool as a promising pilot tool which is ready for use, but which requires establishment of reliability and validity through analysis of initial data.

Youth Development Objectives

Personal objectives include self-esteem and self-confidence, managing personal risks, managing  health and well-being, setting and achieving appropriate goals, making decisions effectively, and physical and outdoor skill development. 

Social objectives include being able to work in a team, taking on leadership opportunities, conflict resolution skills, listening skills, and engagement with community. 

Environmental awareness and skills include knowledge and interest in local environment. 

Institutional outcomes include recidivism and completing of intervention programs.  The current measurement tool does not assess institutional outcomes since these are usually derived from existing records.

Table 1. Seventeen youth development objectives which are typical of adventure-based youth-at-risk programs

Searching for Measurement Methods

Generating the items

Initially, searches were conducted for existing instrumentation to measure each of the objectives.  Relevant scales and items were consulted and adopted or adapted as appropriated.  For approximately half the objectives, items from existing measurement tools were used or adapted; for the other objectives, new items were generated.  More detailed academic background behind each of the proposed items will be provided in a separate report.

Improving reliability of the measurement

In order to reliably assess the “fuzzy” constructs, several general strategies were adopted / are suggested:

1. Where appropriate, intact items were used directly or slightly adapted from other questionnaires.  In such cases, the best three items per scale were selected.  Three items per scale (outcome) is generally considered the minimum necessary to create a psychometrically reliable measure.  For scales where major adaptations or item generation was performed, a minimum of five items were selected.
2. Collect multiple sources of data (i.e., items were constructed so that they could be completed as both self-report and observer-report)
3. The psychometrics and practical utility of those who trial evaluations using these scales should be systematically examined to improve the reliability of future evaluation efforts.

The measurement scale

The recommended measurement scale on which each item would be rated is based on that used for the “Life Effectiveness Questionnaire” (LEQ), a well-known and widely used tool for assessing personal development outcomes from outdoor education programs.  An 8-point Likert rating scale is used, as follows:

The measurement items

The proposed measurement items are in draft format.  In a program’s final questionnaire, the items would be intermixed.  For review, however, they are presented within their target categories.

For each outcome, self-report items and one or two observer items are indicated.  Observer items would be completed by the instructors and/or youth workers who know the participants.  Bold items indicate the most prototypical item in each scale – these items are recommended for use in a short version for use with participant groups who may have difficulty completing the longer version.

Important criteria to be considered in review the utility of the items:

1. Do the items target reflect the program’s intended objectives?
2. Will typical clients be able to provide an appropriate response?

Important Decisions to be Made

The total item pool of 65 items may be too onerous for participants.  Completion of the 65 self-report items is predicted to involve approximately 30 minutes, but for slower participants could conceivably take up to 45 minutes.

Possible strategies for reducing the number of items:

1. Reduce the number of objectives being measured:  Perhaps most important is that the selection of scales to include should reflect program objectives e.g., rank the 17 outcomes from most to least important and select those which really matter.  It is generally better to measure the most important constructs and make good use of the outcome data than to simply measure lots of outcomes.
2. Reduce the number of items per factor: The proposed items include the minimum per scale that is recommended for reliable self-assessment of these constructs.  However, the best single item has been indicated for each scale, to allow for a short (1 item per scale) instrument.  Alternatively, a 2-item per scale instrument could be constructed, consisting of 34 items, which would take approximately 15 minutes to complete.  This would provide broad coverage in a less demanding format for participants.  The data would be useful for program evaluation purposes, but as on ongoing tool, the instrument may not be ideal for future reporting which would be scrutinized for its academic rigor.

A further consideration is whether you would like to also include direct measure of participants' mental health and psychological well-being.  If so, there are many possibilities, but consider a short version of the Mental Health Inventory which measures psychological well-being (10 items) and psychological distress (10 items) in adolescents (Heubeck & Neill, 2000).

Table 2. Summary of the youth development objectives and their proposed measures

References

Dahl, & Reed (1999). The challenges of measuring the impact of recreation programs on youth resiliency: Designing culturally sensitive and age-appropriate evaluation instruments. California Parks and Recreation magazine, Summer.

Frydenberg, E., & Lewis, R. (1993). Adolescent Coping Scale: Administrator's manual. The Australian Council for Educational Research, Hawthorn, Australia.

Heubeck, B. G., & Neill, J. T. (2000). Internal validity and reliability of the 30 item Mental Health Inventory for Australian Adolescents.  Psychological Reports, 87, 431-440.

Marsh, H. W. (1990). Self-Description Questionnaire - II: Manual. Sydney: University of Western Sydney.

Neill, J. T. (2004). How to Choose Tools, Instruments, & Questionnaires for Intervention Research & Evaluation.  Outdoor Education Research & Evaluation Center.

Neill, J. T., Marsh, H. W., & Richards, G. E. (1997). Development and psychometrics of the Life Effectiveness Questionnaire. Sydney: University of Western Sydney.

Richards, G. E , Ellis, L. A., Neill, J. T. (2002). The ROPELOC: Review of Personal Effectiveness and Locus of Control: A comprehensive instrument for reviewing life effectiveness. Paper presented at Self-Concept Research: Driving International Research Agendas, 6-8 August, Sydney.

Russell, K. (2002). Longitudinal assessment of treatment outcomes in outdoor behavioral healthcare. Technical Report 28. Wilderness Research Center, University of Idaho, Moscow, ID. [.16 MB, .pdf]