The CFSRs use multiple information sources to assess State performance, including data indicators, the Statewide Assessment, case record reviews, and a variety of interviews. These interviews might be with children, parents, foster parents, social workers, or other professionals working with a child; they may also involve State or other community stakeholders. All of these sources are necessary to gather both quantitative and qualitative data and to gain a comprehensive picture of the child welfare system.
Reviewers and Local Site Leaders serve as the principle data collectors during the onsite portion of the CFSRs. The amount of data they collect on site, as well as the sources of those data, are very important. No amount of subsequent data analysis can make up for a lack of original data quantity or quality. In other words, the data analysis required to inform both the Final Reports and the Program Improvement Plan can only be as good as the original data allow. This is why the Quality Assurance (QA) process, which ensures the validation and integrity of data collected onsite, is so important.
The QA process as a whole is divided into seven steps.
- Preliminary QA is conducted by reviewers prior to transferring completed cases to a site leader for First-Level QA. It is also conducted by site leaders before they begin First-Level QA.
- First-Level QA, also called Onsite QA, is the first QA review of a case by a Local Site Leader. Cases are reviewed to ensure that data collected are clean, complete, and consistent.
- Second-Level QA, also called State QA, is the final local case-level QA conducted on an individual case. It is conducted by site leaders with a great deal of CFSR and child welfare-related experience who are approved to do Second-Level QA by the Children's Bureau. They may be located onsite or offsite. They conduct a final review of cases to ensure that data collected are clean, complete, and consistent.
- Local Site Finalization is a final confirmation that all case and stakeholder data have been finalized and uploaded to the central server.
- Data Validation is a final review of all State data to ensure that data are clean, complete, and consistent across the State.
- Third-Level QA is a post-review check of case data.
- Data Change Management is a process to validate and update OSRI data as applicable and appropriate.
There are a number of tools designed to support this process, called collectively the "Data Integrity Materials."