Collecting Data and Ensuring Quality

States should have the ability to input and collect or extract quality information from a variety of sources, including data from Federal reporting systems, case review data, as well as other administrative, quantitative, and qualitative data sources. States should also be able to ensure that the quality of their data is maintained.

Data collected in relation to continuous quality improvement (CQI) should be related to both practice standards (Did monthly visits with the child occur?) and outcomes (Did the child experience repeat maltreatment?). Agencies are already collecting large amounts of aggregate data, or data compiled from several measurements, much of which feeds into systems such as the Adoption and Foster Care Analysis and Reporting System (AFCARS), National Child Abuse and Neglect Data System (NCANDS), and National Youth in Transition Database (NYTD) systems. States may also collect case review data and data that reflect performance in systemic areas. Many agencies collect data specific to various other areas, such as length of time to complete investigations, occurrence of team meetings with families, worker caseloads, and evidence of racial or ethnic disproportionality. Some States also access data that are available from partner agencies, such as the courts, juvenile justice, and mental health providers.

Data from case record reviews, in a well-functioning CQI system, will help determine whether case review instruments and ratings are completed as per instrument instructions and with consistency across reviewers. Review data should also support practice and outcome summaries. Additionally, processes to extract accurate quantitative and qualitative data from across the State’s jurisdictions should be clear and consistently implemented. These methods and processes should be documented, with a process in place to review and verify that they are being followed.

It is possible to generate so much data that an agency becomes overwhelmed as it begins the process of analysis. According to Reveal and Helfgott (2012), “There is a simple and universal answer to ‘what data do I need?’ and that is, it depends on what question(s) you are trying to answer.” For example, agencies might ask themselves “How can we increase placement stability of children in care?” or “How can we stabilize children emotionally and decrease placements in residential treatment facilities?” In other words, agencies will need to make strategic decisions about data they need based on an understanding of what they want to achieve. Thus, agencies should tie data back to their goals, key strategies, and system change efforts.

Agencies should also prioritize attention to data by focusing on the most critical data first. They should then consider data that have the broadest value, are of the greatest benefit to the majority of users, or are of value to the most diverse of users. As agencies learn from their earlier efforts and increasingly improve and become more skilled at analyzing data, the better they will become with analysis of more and varied data.