Overview

CARF's long-time commitment to performance improvement via a performance improvement system (PIS) for service delivery is a practical approach that assists organizations with obtaining information useful for managing program operations. Components of the performance improvement system to evaluate services include:

Effectiveness indicators—For opioid treatment programs (OTPs), measures could relate to the reduction of criminal activity, or the reduction of illegal drug use, or the improvement of physical health, and/or the improvement of quality of life.

Efficiency indicators—Related to whether or not resources are being used economically and productively and may include one of the following issues: utilization, appropriateness, personnel turnover, retention rates, direct service hours, and/or cost

Access indicators—Access to services includes such measures as waiting time for first appointments, wait time for actual admission to services, telephone response time, and reducing waiting lists

Client input and stakeholder input—Satisfaction with the delivery of services from a variety of perspectives; a key piece of information for decision-making

Post-discharge follow-up—Collecting clinical information that compares the current status of the clients to their status at discharge and their reported satisfaction with OTP services

Accreditation Requirement(s)

The organization's PIS must include the following to be in conformance with the standards:

• Input from clients and stakeholders
• One effectiveness measure (results)
• One efficiency measure (the relationship between the results achieved and the resources used)
• One indicator of access to services
• Post-discharge information
• Measurement when services began, during treatment (if appropriate), at discharge, and after transition from services
• For each indicator, a description of to whom the indicator will be applied, how the data will be collected, a performance goal, and any extenuating factors that need to be considered when conducting the analysis

Implementation Tips

Some Implementation Tips provided, in part, by Robert Johnson at: www.accreditationnow.com.

• Performance improvement outcomes logic can be described as:
  
  
1. What do you want to know
2. Getting the data right
3. Processing the data
4. Formatting the data for the right audience (moving from data to information)
5. Sharing the information
6. Using the information to make decisions
7. Closing the loop and using the improvement opportunity by changing the indicators and processes
   
   
• The key to the successful implementation of an PIS is keeping the process simple and practical. Utilizing a one-page Performance Improvement Grid (see example) is a compact tool to describing and defining the system. Since CARF only requires one efficiency, one effectiveness, and one service access measure, we suggest you choose only one of each. Oftentimes programs become interested in learning so much about outcomes that they go overboard in terms of choosing multiple indicators, and they do not think through how they will collect, analyze, manage, interpret, and report all the data. Indicators should be chosen based on their usefulness for management decision-making and data that are already collected by the organization from industry benchmarks and/or for learning new information about the organization.
  
  
• Involving clients in the indicator selection process is a sure way to capture useful information. If the organization has a client advocacy or advisory group, it may be a source of clients who would be interested in assisting with this function. It may also be a resource in terms of interpreting the data.
  
  
• Stakeholders are defined by CARF as persons served, family members, personnel, funding sources, regulatory or licensing agencies, and other community sources such as referral sources and collaborating agencies. This evaluative input may be received from written surveys/questionnaires, focus groups, community forums, or grant funding (e.g., the United Way).
  
  
• Effectiveness measurement should occur at admission, during a point of time during services (if it is a long-term program), at discharge, and after discharge from services. At admission, baseline information is collected, and then progress toward identified effectiveness indicators and goals are determined. CARF does not expect this information to be reported at a per client level. Instead, CARF is interested in aggregate information. Also, data may be from a representative sample of clients, not necessarily the entire population served.
  
  
• Service access indicators may be measured as the time taken to obtain a first appointment, time between intake and admission to a program, telephone response times, and waiting times for routine or emergency care.
  
  
• Post-discharge outcomes information includes obtaining program satisfaction and perception information from former clients who may have a different perspective now that they have been discharged from the program for any type of reason, and collecting some clinical information to determine the person's current status compared to their status at discharge. The program should try to find as many persons as possible whom they intended to serve (unplanned discharges) and to whom the program actually did serve. CARF's Transition/Recovery Support Services standards also require follow-up after transition to determine whether the client needs further or other services and to offer needed services, when possible. The organization may want to combine these two efforts into one and ask the necessary questions, either via a telephone survey or through a written questionnaire, to conform to the accreditation standards.
  
  
• Follow-up processes are often the most difficult and bothersome to the organization. While it is true that the substance abusing population can be difficult to track down due to factors like not having telephones or permanent addresses, and programs themselves are overwhelmed with long waiting lists so they are not interested in offering additional services to former clients, there can be valuable information gleaned from former clients that can lead to program improvements.
  
  
• The expected outcome of implementing and using an PIS may be improvement of services to the clients throughout their phases of treatment and recovery. In addition, the process might lead to training for staff that promotes reasonable therapeutic outcomes, and hopefully will provide data-based, effective, customer-oriented program planning and resource utilization at the decision-making level of the organization.
  
  
• For further information and clarification on how to conform to the standards and ideas on indicators, refer to CARF's monographs Performance Indicators for Rehabilitation Programs, Version 1.1, 1998; Outcomes Management in Behavioral Health, 1997; and Managing Outcomes in Employment and Community Services, 1997. These can be ordered at www.carf.org.