Functional Components and Processes of CQI

An effective continuous quality improvement (CQI) system consists of five “functional components” that must be addressed by the agency. Those five functional components are listed below and described in more detail in this section:

  1. Foundational administrative structure describes specific details an agency, whether State-administered, county-administered, or privatized, must take into consideration to ensure that its CQI system is applied consistently and functions effectively, has appropriate oversight, and that the process is being consistently administered as designed.
  2. Quality data collection details the importance of collecting accurate, complete, timely, and meaningful data from a variety of sources for use in assessing and improving practice and systems, and discusses ways to help ensure that data are accurate.
  3. Case record review data and process considers the agency's quality assurance (QA) case review process, including the qualifications of QA case reviewers, how to ensure inter-rater reliability, and how to deal with special challenges that may be encountered in the QA process itself.
  4. Analysis and dissemination of quality data discusses data-based decision making, qualifications of data analysts, differing levels of data analysis, and data dissemination.
  5. Feedback to stakeholders and decision-makers and adjustment of programs and processes explains the "feedback loops," or bi-directional communication, that must exist among everyone involved in the CQI process, including all levels of the agency, external stakeholders, consumers, and decision-makers. It also shows how States use data and information to drive organizational change, at varying levels, and improve child and family outcomes.  

Quality assurance (QA) and CQI activities in States administered by State agencies should take place with consistency and quality statewide. This is also true for reviews in county-administered or privatized States, or any combination of types of administration. The results of QA case reviews, considered in conjunction with other CQI activities, will help ensure and sustain high quality services across the agency.

For more information about the five functional components of an effective CQI system, see Information Memorandum 12-07, published by the Children’s Bureau. It is available online at LINK.

Foundational Administrative Structure

A solid foundational administrative structure is a critical component in the development of a well-functioning continuous quality improvement (CQI) system. Strong administrative oversight and commitment by leadership is an obvious element of this component. To illustrate commitment and to promote staff and stakeholder buy-in, leadership should ensure that:

  • Agency-wide CQI standards, requirements, policies, and procedures are clear and consistent
  • The State possesses or builds the capacity to implement a strong operational CQI system
  • There is strong guidance of the CQI program

To be successful, the new systems must create linkages within the entire agency, both vertically and horizontally. By focusing on creating a strong administrative structure, with adequate resources and solid direction, the agency will help ensure the effective functioning and sustainability of its CQI system.

State agencies should have procedures in place that result in a statewide systematic approach to implementing, overseeing, and exercising oversight of the CQI process, ensuring that it is being applied fairly and consistently. There should be a well-articulated, common approach to implementing, reviewing, and adjusting any CQI process.

Some States facilitate CQI by establishing a centralized CQI unit or division that serves as a bridge to connect all areas of the agency, from upper management to support staff. In this case, agencies may assign joint responsibility for monitoring the progress of CQI and any initiatives to both field and central office CQI staff, based on the initiative. Regardless of centralization, an agency will illustrate its commitment to CQI by having designated CQI staff, adequate resources, clearly written procedures, and by expanding its CQI system to go beyond the case review process.

The agency’s CQI requirements, policies, and processes should be clearly articulated in writing, and should illustrate consistent standards and procedures for the entire State, including other public agencies that operate title IV-E programs for the State. The standards and requirements should be structured to ensure that all jurisdictions across the State are implementing and executing CQI activities as designed and intended. Additionally, there should be an approved, consistent training process for CQI staff, under the guidance of the CQI oversight division, to include any external stakeholders who are involved in conducting CQI activities.

State agencies should have capacity and resources to implement and sustain a statewide CQI program on an ongoing basis, using designated CQI staff. Designated staff who focus on CQI activities will be able to develop a high level of expertise in quality assurance (QA) case review and other CQI activities. Having such staff also helps send a message to frontline caseworkers that they are accountable for the outcomes of cases they carry.

It is important that all jurisdictions of the State be assessed and reviewed at least annually through the QA case review process, in order to ensure consistent evaluations of all areas and that practice and system adjustments are made as needed, achievements are recognized, and new goals are being set. 

Quality Data Collection

Collecting data and ensuring quality are critically important in a State’s efforts to establish a robust continuous quality improvement (CQI) system of data compilation, analysis, and dissemination. The collection of quantitative and qualitative data from varying sources is the foundation of a CQI system; a robust connection between administrative data and other sources of information is key to a plausible vision of change. If solid process and outcome data are used to identify strengths and concerns and establish strategies for improvement, and if progress and trends are tracked by repeated measuring, the results can provide management with a simple, visually compelling thermometer of the organization’s performance and health at every level and can help the agency see where it wants to go in the future.

All agencies want data that are timely, complete, understandable, and relate to the task at hand. Data must be accurate and relevant before data analysis can yield beneficial results; for this to occur, any issues that exist with caseworkers and data entry must be identified and resolved. Furthermore, there must be an efficient process in place for the resolution of data quality issues.

An agency’s interrelated activities and processes are anchored by the quality of its information systems and their ability to produce accurate, reliable, interpretable data that are consistent in definition and usage across the State and nationally.

Collecting Data and Ensuring Quality

States should have the ability to input and collect or extract quality information from a variety of sources, including data from Federal reporting systems, case review data, as well as other administrative, quantitative, and qualitative data sources. States should also be able to ensure that the quality of their data is maintained.

Data collected in relation to continuous quality improvement (CQI) should be related to both practice standards (Did monthly visits with the child occur?) and outcomes (Did the child experience repeat maltreatment?). Agencies are already collecting large amounts of aggregate data, or data compiled from several measurements, much of which feeds into systems such as the Adoption and Foster Care Analysis and Reporting System (AFCARS), National Child Abuse and Neglect Data System (NCANDS), and National Youth in Transition Database (NYTD) systems. States may also collect case review data and data that reflect performance in systemic areas. Many agencies collect data specific to various other areas, such as length of time to complete investigations, occurrence of team meetings with families, worker caseloads, and evidence of racial or ethnic disproportionality. Some States also access data that are available from partner agencies, such as the courts, juvenile justice, and mental health providers.

Data from case record reviews, in a well-functioning CQI system, will help determine whether case review instruments and ratings are completed as per instrument instructions and with consistency across reviewers. Review data should also support practice and outcome summaries. Additionally, processes to extract accurate quantitative and qualitative data from across the State’s jurisdictions should be clear and consistently implemented. These methods and processes should be documented, with a process in place to review and verify that they are being followed.

It is possible to generate so much data that an agency becomes overwhelmed as it begins the process of analysis. According to Reveal and Helfgott (2012), “There is a simple and universal answer to ‘what data do I need?’ and that is, it depends on what question(s) you are trying to answer.” For example, agencies might ask themselves “How can we increase placement stability of children in care?” or “How can we stabilize children emotionally and decrease placements in residential treatment facilities?” In other words, agencies will need to make strategic decisions about data they need based on an understanding of what they want to achieve. Thus, agencies should tie data back to their goals, key strategies, and system change efforts.

Agencies should also prioritize attention to data by focusing on the most critical data first. They should then consider data that have the broadest value, are of the greatest benefit to the majority of users, or are of value to the most diverse of users. As agencies learn from their earlier efforts and increasingly improve and become more skilled at analyzing data, the better they will become with analysis of more and varied data. 

Quantitative and Qualitative Data

States will input and extract both quantitative and qualitative data from their continuous quality improvement (CQI) systems, in order to have a more complete understanding of issues being evaluated and addressed. Quantitative data are those that are expressed by numbers and/or frequencies rather than by meaning and observation/experience. In other words, quantitative data are numerical measurements of an object or event (e.g., how many, how much, or how often), while qualitative data are descriptive of characteristics or attributes, representing what someone observes or otherwise gleans. Qualitative data and research help agencies understand action and experience as a whole and in context.

Some common sources of quantitative data are questionnaires, case record reviews, and extractions from institutional databases, while obtaining qualitative information can be achieved from activities such as focus groups, review of case files, and case-related interviews. Because qualitative data are descriptive, they may be more challenging to analyze than quantitative data. Each type of data has positive attributes, and combining both can result in gaining a more comprehensive picture of agency functioning, as both types enable deeper understanding of various phenomena and provide new knowledge.

The following chart shows the difference in qualitative and quantitative elements for the same group, youth age 17 about to age out of foster care in a region of a State:

Aging Out Youth - Qualitative Data Aging Out Youth - Quantitative Data
Employment readiness 173 youth
Relationship with birth families 83 girls, 90 boys
Support systems with caring adults 62% (107) graduating from high school
Preparedness to live unsupervised 48 youth college-bound

To augment their case review systems or to delve further into specific issues, States may want to administer surveys, conduct interviews, or hold focus groups with staff, external stakeholders, or consumers to obtain more qualitative information. Collecting this additional information will go far to provide a more complete picture of overall agency strengths, needs, and functioning in terms of outcomes for children and families, and may be particularly helpful in evaluating systemic factors, such as adequacy of services in the community and training of staff and resource parents. For example, if foster parent retention is a challenge for the agency, it may choose to interview foster parents who dropped out of the program in the past year, or conduct a focus group with current foster parents, to gain valuable qualitative data about changes needed in the program in order to increase retention.

In general, internal stakeholders who would typically be interviewed or included in focus groups include caseworkers (investigation, foster care, and in-home), supervisors, foster home finders, adoption staff, information technology staff, and the local child welfare director. External stakeholders could include organizations and individuals who are representative of entities who participated in the development of the State’s Child and Family Services Plan. Likely participants would be the courts, guardians and attorneys ad litem, directors and staff of community agencies who serve agency consumers, Tribal representatives, law enforcement personnel, and agency attorneys. Foster and adoptive parents and consumers, such as youth served by the agency, would also be included.

To assist in the process, the State’s CQI oversight division might develop a set of core and follow-up questions for the various groups to be used as guidelines in interviewing and facilitating focus group discussions across the State. However, if specific issues are being targeted, then questions may need to be added to reflect local/regional concerns. The interviews/focus group meetings should be standardized as much as possible to help ensure more consistency in information/data that are obtained throughout the State. It will be important to remind group participants that their responses should reflect current agency information, or within the past year or two, rather than anecdotes or information (whether positive or negative) from several years in the past.

Caseworkers and Data Entry

Caseworkers are commonly the originators of the bulk of an agency’s data in its Statewide Automated Child Welfare Information System (SACWIS) or other statewide system, and often this is where data quality begins. A study (Carrilio, 2008) on caseworkers’ use of computers and data systems found four variables relating to the accuracy of their data entry:

  1. Skills and experience with using computers (worker background and comfort level with use of computers)
  2. Perceived ease of use of the agency’s automated system (worker perception regarding user-friendliness of the system)
  3. Utility of the data (worker belief about usefulness and helpfulness of data being gathered)
  4. Attitude about the data (worker perception regarding importance of inputting and gathering the data)

Agencies already employ tools for checking the accuracy and completeness of data. States that continue to have significant data errors and inconsistencies should address any worker entry issues through training and coaching. A well-functioning help desk and other supports for direct delivery staff will also assist greatly in minimizing errors and ensuring a collective sense of responsibility for accurate data. In addition, States should examine how they define data elements to be captured, the clarity of instructions overall, and if data entry screens and systems are well-designed. Well-designed systems will have:

  • Clear screens
  • Well-spaced and uncluttered fields
  • Easy-to-read font sizes
  • Descriptive captions that are easy to understand
  • Information that flows in a logical order within the screen and from screen to screen
  • Ease of entry

Agency leaders should accept responsibility for the appropriate breadth, quality, and usefulness of an agency’s data, and should continually look for ways to improve data. When data are faulty or otherwise inadequate, management should ensure that effective processes are in place to identify, report, and address data errors, inconsistencies, and omissions at whatever juncture and level they may occur. Corrective mechanisms may involve instituting a vigorous data quality assurance (QA) process, training or re-training staff, re-examining skills of those analyzing the data, and/or creating partnerships with outside entities for training and technical assistance to ensure more effective data collection and analysis.

Resolution of Data Quality Issues

Child welfare agencies must stay current on the demographic data about the children, youth, and families being served at any given time. Additionally, they must be able to ascertain the practices and services being provided by frontline staff, as well as services provided by contractors and community agencies. Finally, they must be able to determine whether outcomes of services and practices are meeting agency expectations. Sound assessments of practice and outcomes depend on correct, consistent, and complete data.

Even though agencies strive for data accuracy, in the best of systems there will occasionally be inconsistencies. For example, if the number of children free for adoption differs significantly from the number of children on whom termination of parental rights (TPR) has occurred, there is an obvious error in at least one of the numbers. The State’s process should be clear about who is responsible for entering data, ensuring data accuracy, and correcting errors that are found. Additionally, the process for correcting such errors should be clear, transparent, and effective.

Three of the most common elements that contribute to poor data quality are:

  1. Duplication of information across files and systems
  2. Incomplete and missing data elements
  3. Inconsistent or untimely data entry

To be successful, data quality improvement activities need widespread support and active involvement from all levels of staff. Data quality management must be a collaborative effort that bridges the gaps between the information technology (IT) department and the program divisions.

One approach to managing data might be a collaborative model in which the program side is accountable for ensuring that there are well-defined data quality rules, elements to be captured, measures, and acceptability levels, while IT is responsible for instituting and maintaining the architectural framework to ensure ease of capturing information, that rules are observed, and measures are accurately reported. As Reveal and Helfgott (2012) explained in their article Putting the Pieces Together, “In a fact-based decision-making culture, operational, policy, and regulatory data are all treated as assets of the agency and system, not the purview of a single office.”

Case Record Review Data and Process

A critical component of any agency’s continuous quality improvement (CQI) system is the ongoing, periodic review of case files taken from a statewide case sampling of children who are or were served under the title IV-B and IV-E programs. These quality assurance (QA) case reviews should be performed by skilled QA case reviewers who collect information to assess practice, services, and outcomes for children and families, and to determine whether specific requirements have been met.

Pivotal to gaining a complete picture of the case is conducting case-related interviews, or interviewing various parties involved in the cases. QA case reviews employing a comprehensive case review instrument as well as case-related interviews will yield meaningful data that can be used to make individual, unit, division, regional, and statewide practice improvements.

Through these thorough case file reviews and interviews, the State can better understand how the agency’s policies, procedures, and practices are impacting children and families. Assessment of this detailed case-level data helps in evaluating the quality of services being delivered, and how the agency can better ensure children's ongoing safety, permanency, and well-being. The State’s policies and manuals should provide clear guidance for carrying out and completing case reviews.

The QA case review process should:

  • Cover the entire State
  • Have clear, consistent written policies and processes
  • Be underpinned by strong infrastructure
  • Include interviews of case participants
  • Ensure reviewer skill
  • Promote inter-rater reliability
  • Identify, measure, and clarify practices that guide safety, permanency, and well-being in terms of daily practice

The QA case review activities in all States, whether they are State-administered, county-administered, or privatized States, should take place with consistency and quality statewide, with ongoing involvement and monitoring by the State’s CQI oversight division. Case review activities are an integral part of an agency’s CQI program as a whole. The meaningful results generated by QA case reviews, considered in conjunction with other CQI activities, will help ensure and sustain high quality services across the agency.

Case Sampling

A continuous quality improvement (CQI) system ensures that the States review cases of children based on a sampling universe of children statewide who are or were recently in foster care and children statewide who are or were served in their own homes. State data systems should be able to clearly identify a relevant sample frame. The universe of cases should include the title IV-B and IV-E child population directly served by the State agency, or served through title IV-E agreements (e.g. with Indian Tribes, juvenile justice, or mental health agencies).

QA Case Reviewers

Ensuring the skills of quality assurance (QA) case reviewers is integral to a continuous quality improvement (CQI) system that yields quality data and functions with integrity. Reviewers are normally drawn from staff within the agency and are centrally or regionally supervised, or they may be contractor/private agency staff. Sometimes reviewers may be consultants from outside the agency who have expertise in child welfare; some States have QA case reviewers who are a combination of types. In any case, having designated staff who focus on the CQI role promotes the development of a high level of expertise in case review. Desired qualifications of quality assurance case reviewers would likely include:

  • Extensive experience in, and knowledge of, the State’s child welfare system
  • Critical thinking skills
  • Well-versed in best practices and the agency’s practice standards
  • Ability to assess and synthesize large amounts of case information that may be complex and/or conflicting
  • Ability to work well as a team member
  • Ability to write well
  • Good interviewing skills 
  • Ability to be diplomatic and collegial with other staff

To help ensure objectivity, it is recommended that QA case reviewers and those who provide QA review for the instruments have no responsibility, directly or indirectly, for the cases being reviewed. The further removed reviewers are from the cases they are reviewing, the better. A process should also be in place to screen potential reviewers to ensure that there is no conflict or personal interest or bias in a case that could impact the reviewer’s objectivity. If a conflict of interest is identified with a particular reviewer or team, a process should be in place for dealing with those conflicts (e.g., selection of cases as back-ups that can substitute for the case(s) in question, switching of cases among different reviewers or review teams, or back-up reviewers who can review the case or cases in question, etc.).

Caseworkers and supervisors whose cases are being reviewed are more willing to be full participants in the process if they perceive the QA case reviewers to be open, flexible, and receptive. Administrators should continue to ensure that the reviewers have the ability to put themselves in the shoes of the caseworkers and supervisors whose case(s) they are reviewing, and that reviewers are promoting, on an ongoing basis, supportive and collegial relationships with casework and other staff.

CQI is an approach that is built upon a partnership between the child welfare practitioner and the case reviewer. Each party is mutually seeking to learn, to elevate both the practice of the organization and the skills of the individual practitioner, and to achieve ongoing improvements toward more effective outcomes.

Case Review Instrument

It is critically important that agencies have a well-developed case review instrument that is clear, concise, and definitive, with instructions that systematically guide reviewers in how to assess items and complete the instrument. For agencies that are currently in the process of designing or revising their case review instrument, it is recommended that reviewers, the instrument's end-users, be involved in the process. As noted data experts Hiruma and Kaiho explained, “Research demonstrates that what is considered a quality tool from a designer’s perspective may differ from the user perspective. Engaging the end-user in the tool development process can improve alignment between the purpose of the tool and the needs of the end-user” (Wandersman, Chien, and Katz, 2011). Further, any new or revised instrument or tool should be piloted prior to its full release.

In any case, a review instrument should always include the State’s practice standards, with a focus on the outcomes that the agency wants to achieve. As an example, if an agency has a practice initiative focused on engaging non-custodial parents, it might want to ensure that the review instrument provides a focus on what the worker has done to engage a non-custodial parent. The worker's activities would be compared with the requirements and measures of the strategy or initiative. For instance, it may be that in every case it is expected that the worker complete a formal search and other specific activities to identify and locate non-custodial parents, as well as engage in specific activities to promote engagement. By expanding its case review instrument to cover those areas, the agency could use its case review process to determine whether the initiative has been properly implemented and whether the enhanced practice is having the intended effect or outcome of more non-custodial parent/child involvement and, possibly, reunifications.

Note: The Onsite Review Instrument (OSRI) used during Round 2 of the Child and Family Services Reviews (CFSRs) is a good example of a case review instrument. It is available for download on the Resources Page of the CFSR Information Portal (https://www.cfsrportal.org/node/1159).

Case-Related Interviews

It is essential that agencies, in their quality assurance (QA) case review programs, include interviews of case participants as a part of their review process. There may be others, but parties interviewed typically include:

  • The child/youth (if old enough to participate)
  • Birth parents (as applicable)
  • Caregivers
  • Caseworker and/or supervisor
  • The child’s attorney or representative (as appropriate)
  • Service providers involved in the case

The interviews add valuable case information that may be missing from the file, or that may differ from case file information. Interview information may also corroborate information found in the case record. Reviewers and interviewers should prepare for each interview by noting areas in the case file where information is missing, unclear, or needs validation by the person being interviewed.

Interviews help complete case information, and they can provide unique perspectives that would not otherwise be known. It is particularly important that parents, foster parents, and children/youth be interviewed, because they are direct consumers of agency services and can provide an otherwise unavailable qualitative assessment of practices and services such as case planning, appropriateness of a specific service, and accessibility issues. 

Inter-Rater Reliability

Inter-rater reliability in the continuous quality improvement (CQI) process refers to the degree to which the agency's quality assurance (QA) case reviewers agree with each other in their assessments and ratings of the review instrument items. It is critical to ensure inter-rater reliability to the greatest degree possible, so that practice and outcomes for the State or area can be accurately assessed and review data will be of good quality.

Some elements to help ensure inter-rater/reviewer reliability are:

  • case review instrument that is clear, flows in a logical way, and has instructions that guide the reviewers in assessment/completion of instrument items.
  • Comprehensive, uniform training with refresher training sessions as needed for all case reviewers. There should be as much consistency among trainers as possible, and a component of the training should be the completion of a sample case using the case review instrument.
  • New reviewers who shadow and are mentored by experienced reviewers on a case review, case review exercise, or training session outside an actual review.
  • Reviewers who read and perform QA on each other’s instruments.

Other processes which help ensure instrument quality are:

  • Another level of QA performed on all reviewers’ instruments by someone with reviewer or QA expertise from outside the chain of command or even outside the agency, which helps to ensure consistency, quality, and objectivity.
  • Reviewers’ inclusion of verbal summary presentations, or debriefings, of their case ratings to the review team at the end of a case review to help determine and promote consistency in ratings.

One of the most effective ways to ensure inter-rater reliability and consistency is to have dedicated staff who are completely focused on QA and CQI activities. Dedicated staff persons who repeatedly perform case reviews will likely be able to develop and maintain a high level of expertise and accuracy in their case reviews. Another way to promote inter-rater reliability is to ensure that reviewers are re-trained as needed to assess instrument items and instrument completion. Additionally, CQI oversight staff should provide observation and feedback of reviewer processes, such as interviews of case participants, to help ensure quality and consistency. Monitoring of reviewer processes and results, in terms of reliability and consistency, should be ongoing.

Analysis and Dissemination of Quality Data

States have long collected child welfare data from a variety of sources. Of course, the ability to regularly track, categorize, and analyze data varies from agency to agency, particularly as it relates to obtaining information about safety, permanency, and well-being for children. In recent years, though, the advantages of using data have become more apparent to those who use data extensively in managing their organizations. Both quantitative and qualitative data provide evidence to help take the emotion and guesswork out of decisions that can be difficult. Data from multiple sources can help an agency define its current status versus its desired status; identify its strengths, needs, and trends; and set strategic priorities for reaching desired goals and improving outcomes.

The process of turning data into meaningful information that can be used to make decisions is data analysis, sometimes called analytics. Analytics has become a critical component of managing performance, which normally involves setting goals, monitoring progress toward meeting the goals through use of specific measures, and making necessary adjustments along the way to improve performance. Developing an “analytics mindset” is a process that evolves over time as staff become more accustomed to managing by data. Increasing staff and stakeholder access to data is a crucial element of this mindset.

Data Analysis

Generally, data analysis is defined as the compilation, evaluation, and presentation of data to highlight useful information or suggest answers or conclusions to questions or issues that have been raised. It is essential that only those who have the requisite skills are initially interpreting and compiling data. For example, agencies may want to ensure that those analyzing data have been trained in data analysis, have a thorough understanding of the structure of the State’s child welfare system, and have knowledge of case practice. It is also helpful if analysts have knowledge of and experience with the Children’s Bureau national databases. Additionally, analysts should have the ability to interpret and transfer data results into user-friendly formats that can be understood and used by other staff as well as consumers and external stakeholders.

Some States do not have sufficient qualified staff to analyze data and have created partnerships or alliances with other entities for this purpose. For example, some agencies have developed creative partnerships with one or a consortium of universities to analyze agency data, identify key needs, and develop outcomes and performance measures. When considering these alliances, it is important for agencies to ensure that partners who lend their expertise in data analysis have a working knowledge of agency functioning, including services and goals.

A systems approach in data analysis is important in providing an understanding of how different elements in a system interact with each other. Many things are interrelated, so the whole always has to be considered. If something changes in one area, it is necessary to see what is affected in other areas that may be related. Changes in one outcome can affect other outcomes. For example, as an agency’s time to reunification decreases, the rate of re-entry into care may increase. Close examination and analysis of possible relatedness between the two activities should take place, which may lead to adjustments in practice.

Staff and Stakeholder Access to Data

In order to encourage community support and involvement of key partners like courts and law enforcement, as well as to promote understanding of agency decisions by all community partners, it is crucial that external stakeholders be “at the table” in providing feedback on data collection and data analysis as well as in receiving data, both local and statewide, about agency performance. Fully involving stakeholders in understanding where the agency is, where it wants to go and how it hopes to get there, and eliciting stakeholder input into agency planning, will:

  1. Promote ownership of the agency and its services
  2. Foster support for the work and its difficulties
  3. Ensure multiple perspectives that will enrich the continuous quality improvement (CQI) process overall

Aggregated data are data compiled from different measures so that figures relate to broad classes, groups, or categories. An agency should provide its local and statewide aggregated data to all levels of staff for their consideration in determining and assessing strengths, trends, and needs. Along with aggregated data, many agencies provide more specific non-aggregated data for staff, including at the caseworker and unit levels, which promotes ownership, illustrates transparency, and fosters the concept of “managing by data.” Additionally, having the data helps frontline staff to assume responsibility for the outcomes on their caseloads by providing tangible evidence that their actions do matter. 

Feedback to Stakeholders and Decision-Makers and Adjustment of Programs and Processes

Essential to a well-functioning continuous quality improvement (CQI) system is building productive CQI teams and ensuring that information generated through the system will be effectively used to make needed improvements. A productive CQI system requires a mechanism that promotes circular feedback and communication among staff, stakeholders, and teams. These feedback loops permit an ongoing, bi-directional information exchange across all levels of the agency, which in turn facilitates the change process. Equally important is sharing data with agency staff and sharing data with consumers and external stakeholders.

By sharing data and information and then using staff, stakeholder, and consumer feedback as a starting point, the agency can create a dialogue about improvements it should make in policies, practices, systems, planning, services, and in its CQI program as a whole. Frontline staff, particularly workers and supervisors, will show an increased understanding of how their day-to-day actions, as revealed by data, impact short- and long-term outcomes for children and families, and how their practices can be enhanced as a result. Thus, staff, youth, families, and external stakeholders should receive information and actively participate in analyzing and interpreting data, connecting data to practice, and identifying trends and key findings.

Through this process of data-based decision-making, the CQI process as a whole is subject to continued examination and evaluation and can be adjusted as needed to better meet agency needs. This ongoing adjustment is one of the key factors in an agency maintaining the momentum of effective systemic change.

CQI Teams

A key component of setting up an agency-wide continuous quality improvement (CQI) process that enables the systematic gathering of information for analysis is building a statewide CQI implementation team composed of experienced evaluators, external stakeholders, and differing levels of committed staff with varying skills. This team should be tasked with leading the overall CQI initiative. The agency may also choose to form regional teams to augment the work of the statewide team and to closely monitor regional initiatives. If so, these regional teams should be composed of all levels of staff, as well as external stakeholders and agency consumers.

The external stakeholders who serve on these implementation teams may be selected based on general knowledge, skills, interest, and involvement in the agency. They may also have special areas of expertise that would be particularly helpful in implementing systems change. For instance, given a State’s array of service and practice needs, agency leadership may want to include workers from mental health agencies, both public and private, as they may be particularly helpful in bringing knowledge and resources to strengthen the State’s counseling and treatment services. Or leadership may want to appoint representatives of public health and the school systems so they can assist with ways to improve medical, dental, and educational services for agency consumers. Depending on the particular needs of the State, stakeholders with other special areas of expertise may help round out the team. Special areas of expertise may include:

  • Grant proposal writing
  • Data analysis
  • Media
  • Planning and administration
  • Criminal justice
  • Evaluation
  • Research

It is essential that external stakeholders be committed to systems change and able to give of their time and efforts to be deeply engaged in the process. Rather than being seen as “window dressing” for the systems change process, they should be viewed as true partners, willing to contribute their time, ideas, and energy to ensuring better outcomes for families and children. Likewise, they should be valued for the skills and experience they bring to the team and for their ongoing contributions in developing the change vision.

In order to foster a cooperative spirit, staff members’ contributions should be based on their knowledge, skills, and experience rather than their positions in the agency. Staff must feel comfortable speaking truthfully, particularly about barriers they see regarding CQI implementation, and have confidence that their contributions will be taken seriously. CQI requires that all employees, units, and departments share a unified purpose, direction, and commitment. De-emphasizing hierarchy on teams and encouraging cross-sector alliances will enhance productive team functioning.

Employing team structure is at the core of CQI, and open, transparent communication between team members, among various teams, and between teams, leadership, and agency staff is critical to effective functioning of the CQI program. There should be constant, bi-directional, formalized “feedback loops” for communication among staff, stakeholders, and teamsIn these loops, data/information is being imparted, assessment is taking place, and information is being fed back in planning, implementing, and overseeing the State’s CQI program.

Agency leaders are responsible for ensuring that CQI teams meet regularly and that they consistently push forward to complete planning and activities. If group enthusiasm and commitment begin to flag, one option to get teams back on track is to bring in trained facilitators to lead team meetings for a time. 

Communication Among Staff, Stakeholders, and Teams

Because continuous quality improvement (CQI) teams are populated by staff from all levels and divisions of the agency, administration should ensure that there are strong, formalized communication loops (“feedback loops”) among the various groups. There should be ongoing bi-directional communication between the administration and policy division, the training unit, the data unit, program staff, information technology staff, and legal staff. All agency divisions should routinely interact to ensure that both field and support staff are actively engaged in assessing data, brainstorming solutions, exchanging ideas, recommending changes, monitoring implementation, and reviewing results of practices and CQI processes. Ongoing interaction and communication among these agency units and levels are essential so that everyone understands the change initiative(s) and joins in a unified effort toward the same end.

It is especially critical that the policy division and training unit be strongly connected. For example, if the agency implements more thorough measures to screen infants for risk in investigations but the training unit fails to incorporate the policy into its new worker curriculum, poor implementation could significantly compromise the effectiveness of the change.

States should have firm processes for sharing, analyzing, and using data and information with external stakeholders (staff, courts, Tribes, service providers, and others) and consumers, to guide collaborations and inform the Child and Family Services Plan and other planning efforts. Leaders, staff, and external stakeholders should consider trends, comparisons, and other data findings, to drive both incremental and larger-scale change when improvements are needed in specific practices or in the CQI program as a whole. 

Sharing Data with Agency Staff

Management may be unsure about how to determine which data are most relevant to the field. Ideally, the primary data that should be going to workers and supervisors are in the key areas that they are trying to improve. For example, information on timely initial contacts and time to finish investigations should be prioritized for investigative staff. If more foster homes are needed, data and information on recruitment and retention of foster parents should be prioritized for placement and foster home finder staff. The data should be focused and updated regularly.

Data importance will shift to coincide with changing priorities and change initiatives. For example, as goals for the frequency of visits between worker and child are met, that data should be accorded a lower priority level. On the other hand, if frequency of visits is currently met but there are fears that it may fall again, that data may continue to bear close watching.

Many agencies have developed creative ways to share data. Several agencies use data “dashboards,” presented in Web applications or portals on staff computers, which provide various types of data to staff at all levels across the State. The dashboards display the most relevant data in a clear, user-friendly format, many times with charts and graphs. In some instances, data access is interactive so that high-level data can be broken down into more specific units. Some agencies even break down the information to the unit, worker, and client level. The design of the dashboards helps bring the data to life for staff and highlights the importance of accurate data entry. Also, it emphasizes the direct connection between their casework and consumer outcomes.

Dashboard screenshot

Some States display key data on their State agency Web sites, so it is accessible to both employees and those outside the agency. These data are updated on at least a quarterly basis. Other agencies provide information to staff through “tips” documents intended to help in their day-to-day work; topics may be based on quality assurance (QA) case record review results or other data. If, for example, a State is having a particular problem engaging absent parents, the tip sheet would contain useful guidance on how to promote engagement.

Note: To view an example of a “Tips” document developed by the State of Minnesota to help improve visits with siblings in foster care, visit: http://www.dhs.state.mn.us/main/groups/county_access/documents/pub/dhs_id_028202.pdf

Sharing Data with Consumers and External Stakeholders

Agency leaders should institute strong bi-directional communication among not only all levels of staff, but stakeholders and consumers as well. Youth, families, and stakeholders should be an integral part of continuous quality improvement (CQI) activities. Leaders should regularly and directly share with these groups the results of data analyses as well as successes, concerns, and contemplated changes. They should also actively solicit feedback from stakeholders and families about their experiences with the agency, what is working and what is not, and how they perceive and recommend that practice be changed.

This ongoing, bi-directional communication is known as a feedback loop. To cite an example of this loop, consider a large-scale, significant issue such as insufficient services statewide for emancipating youth. As a starting point, agency leadership might solicit feedback from youth groups/alliances and youth workers, with those groups framing the issues and preparing data showing concerns and trends. Then, management might charge regional leaders with having regional/area forums with staff, external stakeholders, families, and youth to assess and plan. At these forums, mixed-population groups would be encouraged to share experiences, further identify sub-issues, brainstorm, exchange ideas, and provide group recommendations.

The recommendations from each region would be collected by agency leadership, and, based on those recommendations, a tentative plan, approved by a group composed of staff, youth, and stakeholder representatives from each region, would be formulated for moving forward. The plan would be sent to all participants as quickly as possible, with a request for any further feedback or recommendations. Additional benefits of the forums might include the forming of new, productive alliances of action-oriented individuals who would help ensure that the forums’ recommendations become reality.

In any case, staff, children, youth, families, and external stakeholders should actively and continuously provide advice and feedback throughout the agency. Those involved in working teams, regardless of their roles, should take ownership of issues and be deeply engaged in the change efforts. Their observations, analysis, and recommendations should help inform the State’s strategic priorities and initiatives. There should be constant cross-fertilization of ideas and communication among various individuals and groups, with vision shaped by the varying areas of expertise and perspective each brings, and all should have complete access to processes and information.

Data-Based Decision-Making

The concept of data-based decision making underpins the entire process of a continuous quality improvement (CQI) system. Statistical data, if accurate, are a reliable means of information on which to base important practice and staffing decisions. If decisions and changes are based on solid data rather than on observations, anecdotal evidence, or conventional wisdom and if practice changes are sound, the chances are greatly increased that anticipated outcomes will be achieved.

The use of analytics, or the process of examining raw data from multiple sources to determine patterns, trends, and other useful information, has become an essential component of good management and sound decision-making in today’s organizations. Most agencies now recognize the need to manage by data so they will know how they should improve, but some are unclear on how to begin. The first step is to assemble a CQI team that includes staff with analytical skills, people familiar with the work being done, and subject matter experts. This team should be tasked with identifying the questions that need to be asked, such as:

  • “What are we trying to achieve, how do we measure the results, and are agency goals and objectives clearly tied to the desired results?”
  • “How do our processes and activities relate to our goals and objectives?”
  • “What is working well and what is not?”
  • “What will success look like, and how will we know we’ve achieved it?”

With these questions as a starting point, the team should brainstorm to determine what an improved result or outcome might be, then identify roles and responsibilities, barriers to be addressed, and resources that can be leveraged. An action plan should be developed with small, incremental steps identified to test the process and further refine requirements for data analysis.

The value of data’s use in decision-making and achieving improved outcomes should be constantly shared with staff and external stakeholders in an open, transparent way. Management should lead by example, setting the vision and clearly communicating to those involved how data have informed decisions. Soliciting staff and stakeholder input into development of measures, particularly, will boost their enthusiasm for, and understanding of, the process.

Data should be analyzed on an ongoing basis to determine what is not working, and that analysis should then be used to inform further data-based decision making to drive changes in policy, practice, support services, training, consumer services, and the CQI program as a whole. As a practice example, if data reveal that worker/child visits are meeting the agency's quantitative frequency element but review information shows that the visits are brief and superficial, then the agency understands that it has a significant missing qualitative element. For example, the data may reflect that workers are not assessing foster parent needs during the visits. As a result, the agency will work to improve the practice, using input from foster parents, staff, and others.

Data-based decision-making is a repetitive process that, depending on the outcome, may involve re-examining decisions that have already been made and making ongoing adjustments to the course(s) of action. This may require back-tracking to an earlier activity. In an environment of continual learning and striving for improvement, data should be used at every level of both staff and agency decision-making—from workers, supervisors, and regional managers to central office managers—to assess policy and case practice, determine how the CQI system is functioning, identify shortfalls and trends, pinpoint what is working well, decide on corrective actions, and anticipate and plan for various issues that may impact staff, consumers, and the community. A well-functioning CQI system enables the implementation of changes at any level, from the worker to the entire agency, in a continuous cycle of learning, adjusting, and improving. 

Workers, Supervisors, and Regional Managers

Staff at all levels should be encouraged to use data to constantly examine their own practice and the practice of those whom they supervise. For example, investigative workers, in assessing safety of children, can and do use basic data in a substantive way to inform their case decisions. First, they look at data/information on past referrals to ascertain the family’s history with the agency, then use that information to help determine threat of harm to the child and parental protective capacities so that better decisions can be made about the next course of action. Workers also use basic data to monitor and improve their performance, such as their timeliness in initiating and completing investigations, the frequency with which they see children and parents, the number of placement changes of children on their caseload, etc.

Supervisors should examine the same types of data as their workers to ensure that practice standards and outcomes are being met, and to identify areas, such as placement instability and lack of timeliness in initiating investigations, where the unit needs improvement. They will also use the data to identify individual worker strengths and challenges in different areas, so that challenges can be addressed and strengths can be recognized, analyzed, and used to help others.

A county or regional manager might look at the same practice issues as the supervisors, but would examine data from several units to determine which were functioning best regarding specific issues, how the county or regional outcomes compare with other regions or State or national standards, and how the well-functioning units could be used to help improve practice of the others. Also, the manager might consider issues in light of an area’s context, like the number of substantiated investigations or number of children coming into care, an unexpected increase in community unemployment, a rise in local drug use, or the sudden closure of a community resource. With this type of analysis, the manager would be better able to predict and plan for what may be on the horizon in terms of investigations, support needed by families, and numbers of children who might be entering care.

Central Office Managers

At the State or central office level, managers should look at the same data examined by county or regional managers in terms of practice and areas needing improvement, and should compare areas and the State as a whole with the national standards. They should also make comparisons between counties and regions regarding practice and outcomes. If, for example, one or more areas use family team meetings and others do not, management might look at ways the meetings impact caseloads, such as the time it takes to hold the meetings or the effect they have on outcomes such as children’s length of care, and use that data to determine whether and how to roll out the practice to other areas.

In this example, if management determines that family team meetings are having positive impacts but need strengthening before expansion, then training for family team facilitators might need to be more comprehensive, and stronger policies might be needed regarding the importance of relative involvement. If family engagement as a whole is an issue system-wide, then strategies to improve case planning, parental involvement, worker/parent visits, extended family connections, and so forth would need to be in place before family team meetings are expanded further.

Managers should constantly use data to analyze outcomes and then set goals for what those outcomes should be. If, for instance, managers discover that an unacceptably high percentage of children in care are having no visits with their fathers, they may decide that a closer look is needed at worker/father visits, father assessments, and services to fathers. They may also look harder at how the agency involves fathers with their children outside of visitation so that realistic goals can be set and strategies to reach those goals can be put into place.

Finally, State and central office managers should also consider how activities in one program might affect another. If, for example, there were high numbers of children coming into care, that data could be used to help examine staffing issues and how cases might be effectively handled later on by permanency and adoption workers.

Maintaining the Momentum

If management senses that the continuous quality improvement (CQI) process is weakening, it should try to pinpoint the areas that are waning. Administrators should ask themselves probing questions:

  • In which areas is the agency strong?
  • What are the assets that are supporting and reinforcing quality improvement efforts?
  • In which areas is the agency failing to sustain the quality improvement philosophy and practice?
  • Are measurement and feedback systems still clear and effective?
  • Have we truly applied lessons learned from CQI in particular areas to the rest of the agency?

Sustaining change momentum may require adjusting the CQI vision, plan, and strategies in response to the new ideas and answers offered by staff, stakeholders, and families. Leaders should adjust the change vision and strategy to reflect new learning and insights. Management should continually challenge employees to be open in their communication and, whenever feasible, integrate dissident perspectives into the vision. Leaders should recognize and celebrate accomplishments, identify consequences of not implementing a change, and provide a solid reason and motivation to act.

Staff, stakeholder, and consumer understanding of and commitment to CQI will deepen as they observe that the agency's leadership has truly embraced CQI as the process by which it makes decisions and evaluates progress. Enormous effort is involved in initiating and maintaining a CQI program, but the ongoing rewards of a successful program are even more immense.

CQI in County-Administered or Privatized States

In county-administered States where the various counties operate with some degree of autonomy, as well as in States that have largely privatized their child welfare services, States should continue to have consistent expectations statewide regarding continuous quality improvement (CQI). The State agency should establish basic requirements that are uniform for the entire State, while considering negotiation with counties or private agencies on issues that may allow some flexibility (such as less frequent reviews for areas that consistently perform well).

The CQI focus will be on ensuring that the CQI process is comprehensive, contains a robust case review process, that collected data are uniform statewide, and that there is consistency throughout the State in CQI processes and casework standards and practices. In the case review process, CQI staff, whether centrally supervised or supervised from their counties/regions, will assess practice more objectively and consistently in their review of cases if they are dedicated to CQI activities. Having dedicated CQI staff who work closely with the State’s CQI administrative oversight division will also help ensure that case record reviews are conducted with necessary frequency, meet quality standards, and adhere to other policy requirements.

If the counties in county-administered States have their own CQI staff, then the State’s CQI oversight staff could either pair with the county staff for case record reviews or have close involvement in planning and overseeing the reviews in a county or region to help ensure a quality process. If that is not possible, then county reviewers could cross-review with other counties and the State’s CQI oversight division could take steps to ensure consistency and quality, such as the provision of QA for case review instruments. Likewise, in the case of private agency contractors who have their own CQI staff, the State’s CQI oversight staff might pair with the reviewers, at least on initial reviews, and/or take steps to ensure accuracy, quality, and objectivity such as providing QA review for case review instruments completed by contractors.

Oversight of county and private agency QA case reviews by the State’s oversight division should be strong enough to ensure that review-related procedures are being followed, including that:

  • County, regional, or contractor reviewers are qualified and appropriately trained
  • The necessary degree of objectivity exists
  • Inter-rater reliability is assured
  • Reviews are being completed on schedule
  • A back-up plan is in place when conflicts of interest are identified with a reviewer, private agency, or county
  • The case sample is randomly identified and the size is correct
  • Required review results are drawn up and dispersed in a timely way

If CQI staff are county-based and -supervised, they should receive uniform training (and re-training as needed) coordinated by the CQI oversight division, as should private agency staff. 

It is critical to ensure that private agency contractors have a thorough understanding of the State’s CQI efforts and requirements and that they be held to the same standards as agency staff regarding:

CQI expectations, processes, and outcomes should be clearly articulated in private agency contracts, and should be closely overseen. Contractor performance, on an ongoing basis, should be monitored by the State agency’s program staff or contract managers or both, as should action plans for contractors to correct any deficiencies. Diligent monitoring and oversight of contractor activities will help ensure quality and integrity in their CQI programs. Enthusiasm for and commitment to the CQI process should be just as high in the private agencies as it is in the State agency.