Data Integrity and Quality Assurance

The CFSRs use multiple information sources to assess State performance, including data indicators, the Statewide Assessment, case record reviews, and a variety of interviews. These interviews might be with children, parents, foster parents, social workers, or other professionals working with a child; they may also involve State or other community stakeholders. All of these sources are necessary to gather both quantitative and qualitative data and to gain a comprehensive picture of the child welfare system.

Reviewers and Local Site Leaders serve as the principle data collectors during the onsite portion of the CFSRs. The amount of data they collect on site, as well as the sources of those data, are very important. No amount of subsequent data analysis can make up for a lack of original data quantity or quality. In other words, the data analysis required to inform both the Final Reports and the Program Improvement Plan can only be as good as the original data allow. This is why the Quality Assurance (QA) process, which ensures the validation and integrity of data collected onsite, is so important.

The QA process as a whole is divided into seven steps.

  1. Preliminary QA is conducted by reviewers prior to transferring completed cases to a site leader for First-Level QA. It is also conducted by site leaders before they begin First-Level QA.
  2. First-Level QA, also called Onsite QA, is the first QA review of a case by a Local Site Leader. Cases are reviewed to ensure that data collected are clean, complete, and consistent.
  3. Second-Level QA, also called State QA, is the final local case-level QA conducted on an individual case. It is conducted by site leaders with a great deal of CFSR and child welfare-related experience who are approved to do Second-Level QA by the Children's Bureau. They may be located onsite or offsite. They conduct a final review of cases to ensure that data collected are clean, complete, and consistent.
  4. Local Site Finalization is a final confirmation that all case and stakeholder data have been finalized and uploaded to the central server.
  5. Data Validation is a final review of all State data to ensure that data are clean, complete, and consistent across the State.
  6. Third-Level QA is a post-review check of case data.
  7. Data Change Management is a process to validate and update OSRI data as applicable and appropriate.

There are a number of tools designed to support this process, called collectively the "Data Integrity Materials." 

Preliminary QA

Before transferring an OSRI to a designated Local Site Leader for First-Level QA, reviewers should conduct Preliminary QA to make sure the case is complete and accurate. Preliminary QA consists of the following processes:

Review the OSRI. Review instructions and definitions and carefully re-read the Main Reason statements and follow-up questions to ensure that all documentation requested is addressed appropriately. Use the OSRI Quality Assurance Guide and the Combined QA Tip Sheet to ensure that you have responded to item questions appropriately and have correctly documented item ratings.

Review the Completeness Column. Check the Completeness column on the Overview Screen of the automated application. The Face Sheet, OSRI, Documentation, and Items Rated should indicate that all items are 100 percent complete.

Check the Unanswered Questions Navigator: The Unanswered Questions Navigator on the bottom left corner will display all the unanswered item and rating documentation questions. If the case is complete, no items should appear.

Examine the tablet reports: The automated application features various reports that allow you to review the entire instrument for clean and complete data as one continuous document, rather than going screen by screen. These include the Completed Case Report, the Preliminary Case Summary Report, and the Case QA Rating Summary Report.

Once you’ve verified that the instrument is complete and using the methods above, you must complete the Case Finalization Checklist and turn it in to the onsite JBS representative. At this point, Preliminary QA is complete and you are ready to proceed to First-Level QA.

First-Level QA

First-Level QA is the first quality assurance review of a case by a Local Site Leader. The specific steps involved in First-Level QA vary depending on your role.

First-Level QA for Reviewers

Reviewers who are going through the First-Level QA process will use the following steps:

  1. Change the status of the case: Before you can transfer your case to a Local Site Leader to begin First-Level QA, you must change its QA Status to "Ready for QA Review."
  2. Locate a site leader to transfer case: Once the QA Status has been changed, locate the appropriate site leader or his or her designee. The site leader will transfer the case to his or her tablet to conduct First-Level QA.
  3. Case becomes locked: Once the site leader transfers your case to his or her tablet, it will become inactive on your tablet and grayed-out in the Record Summary Grid. You can still navigate the case to access information or to prepare the Nightly Debriefing Report, but you will not be able to edit any information until the site leader returns the case to you.
  4. Receive case from site leader after review: When the site leader has finished conducting QA on the case, he or she will notify you and return the active case to your tablet.
  5. Refresh the Overview Screen: The first thing you’ll want to do when the case is transferred back to your tablet is refresh the Overview Screen. When the case refreshes, you’ll see that it no longer appears as dark gray. This means it’s now unlocked and can be edited normally again. In addition, the case’s QA Status will have been changed to “QA Review Complete.”
  6. Locate stickies: Now you will need to find the comments, or "stickies," left by the site leader. These are the application’s version of the actual sticky notes that were added to the instrument during the first round of reviews, when everything was done in hard copy. The Unresolved Comment Navigator makes this easy by displaying all comments and allowing you to jump immediately to any sticky note.
  7. Address each sticky: As you work through the stickies, be sure to address each problem indicated in the note, making any corrections or additions necessary. If you don't fully understand or agree with the comments in a sticky, be sure to discuss the issue with the site leader.
  8. Respond to each sticky: After you take all necessary steps to address a sticky, be sure to write a response. Your response might be as brief as "done" or "fixed," or, if necessary, it might briefly summarize the actions you took to address the issues raised by the sticky.
  9. Return case to site leader: When you've addressed the issues in every sticky and left a response for each one, you will return the case to the site leader so that he or she can review your changes and comments.
  10. Resolving stickies: Once the site leader is satisfied that the issue has been adequately addressed, he or she will mark the comment as resolved. When a comment is resolved, it disappears from the Unresolved Comment Navigator. If, however, the site leader determines that not all of the comments were appropriately addressed, he or she will return the case to you again. This process may continue, or he or she may discuss the issues with you face to face, until all comments are successfully resolved.

When all stickies are resolved, First-Level QA is complete and the case is ready for Second-Level QA.

First-Level QA for Site Leaders

Specifically designated Local Site Leaders will bear responsibility for performing First-Level QA on OSRIs as they are completed by reviewers. In order to accurately perform First-Level QA, the site leader should complete both an initial review and an in-depth review.

Initial Review

The initial review is an opportunity for the site leader to become familiar with the case and quickly review the instrument. Follow these steps to complete your initial review:

  1. Confirm Completeness and Preliminary QA. Before you transfer a case to begin First-Level QA, make sure the case is complete by checking the Completeness column. Also verify with the review pair that they have completed Preliminary QA on the case.
  2. Transfer the case. You will need to transfer the case to your tablet to conduct First-Level QA. Before transferring a case, ensure that the reviewers have changed the case's QA Status to "Ready for QA Review." Also remember that after you transfer a case, reviewers will be able to access the case but not edit it until you have returned the case to them. JBS technical support staff on site will assist you in your first transfer, which you will find to be a very simple process.
  3. Check sample appropriateness. Use the Case Elimination Guidelines located on the ACF website, to make sure that the case belongs in the sample. If, after review, it appears that the case does not belong, you may need to consult with the NRT Local Site Leader about eliminating it from the sample.
  4. Add stickies where necessary. As you begin to review the case, add stickies to any questions that need correction, clarification, or additional information.
  5. Gain a basic understanding of the case. You’ll want to start to gain a basic understanding of the case as quickly as possible. Different site leaders employ different strategies for learning about the case details. One particularly effective strategy is to scan the Face Sheet, items 1–4 of the instrument, item 7, and item 17. Site leaders will go back over each of these items in more detail later in the QA process; however, if reviewers have appropriately completed these items, they should provide a basic overview of the case.
  6. Take basic notes. Site Leaders have also found it useful to begin taking notes from the Face Sheet at this time. Noting some basic information, such as the age of the target child or the ages of the children in the home and the date that the case was opened for services, can help the site leader get an overview of the case and prepares them for reviewing missing information.
  7. Identify missing information. Next, you’ll quickly review the case for missing information. The Combined QA Tip Sheet, particularly the sections that provide information on general errors in the body of the instrument and the Face Sheet, should be your guide.

In-Depth Review

Once you’ve completed the initial review of the case, you should begin reading through the case more carefully, paying special attention to the Rating Documentation for each item and using the item-specific errors and correlations section of the Combined QA Tip Sheet.Use the following steps to complete your in-depth review of the case:

  1. Examine Main Reason statements. Each Main Reason statement should be clear, concise, focused. Each statement should address only the issue being assessed in the item. It must also be consistent with the item’s rating and should contain enough information to fully explain or justify the rating.
  2. Check exploratory questions. Some of the information needed to fully explain the rating is covered in the follow-up questions, and you need to make sure that they have all been addressed fully and adequately in their Main Reason statement.
  3. Check for sources. You should also ensure that the Main Reason statement indicates the source or sources of the information—for example, the case record or an interview with parents. Again, the QA Tips should be used to help you here.
  4. Check for consistency throughout. Also note that there should be no contradictory information between items throughout an instrument. The QA Tips should be used to ensure that you have reviewed each case for discrepancies in the information provided from one item to another. For example, the Face Sheet might list three siblings of a targeted foster child, but the later items, such as items 12 and 13, might reference only two siblings. Also, item 17 might reference two fathers, but item 19 might reference only one. Inconsistency across items can be the most difficult and time consuming to identify.
  5. Transfer back to Reviewers. Once you have completely reviewed the case and added the necessary stickies throughout, you should change the case's QA status to "QA Review Complete" and then transfer the case back to the reviewers so that they can begin addressing your comments.
  6. Resolve Stickies. Once the review pair has finished addressing your comments, they will inform you that it is ready, and you will again transfer it to your tablet. At this point, you will review the stickies you added. If further changes are necessary, you can add to the sticky and send it back for further correction or meet with the review pair in person to discuss the problem. If the sticky was adequately addressed, you should resolve it to remove it from the list.

Once all of its stickies have been resolved, the case is ready to move on to Second-Level QA.

Second-Level QA

Second-Level QA, sometimes referred to as State-Level QA, is the final local case-level QA conducted on an individual case. It is conducted by site leaders who have a great deal of CFSR and child welfare-related experience and are approved to conduct Second-Level QA by the Children's Bureau. These site leaders may be located on or off site. If the site leader is located on site, then the review pair may work directly with him or her during Second-Level QA in a process identical to that used during First-Level QA. If the site leader is off site, the overall process remains the same, but the case must first be transfered to an onsite site leader's tablet before it can be moved to the offsite leader.

To conduct Second-Level QA, site leaders conduct a final review of cases (following the steps outlined in Preliminary QA and First-Level QA) to ensure that data collected are clean, complete, and consistent. In addition, they place special emphasis on consistency across cases in the site and State-specific issues that may arise. If they discover issues that may impact all sites, they will communicate with the NRT Local Site Leader so that he or she can discuss such issues with other NRT Site Leaders in order to ensure consistency throughout the State.

During Second-Level QA, the site leader may add additional stickies to the case, which will require that reviewers go through the same process as in First-Level QA to find and address the stickies. Once all stickies from First and Second-Level QA are resolved, the case is considered to be finalized and ready for Local Site Finalization.

Local Site Finalization

Local Site Finalization is completed by a JBS representative on site on the Thursday immediately prior to the NRT Local Site Leader’s transfer of cases to the central server. This is a final confirmation that all case and stakeholder data have been finalized and uploaded to the central server.

Data Validation

Data Validation takes place on Thursday evening and Friday morning. Federal staff and consultants work together to conduct a final review of State data to ensure that data are clean, complete, and consistent across the State.

Third-Level QA

Third-Level QA is a post-review check of the data by the Final Report writer assigned to the case following the review week.

Data Change Management

Data Change Management involves Final Report writers, the CB Central and Regional Office, and JBS staff collaborating to validate and update OSRI data as applicable and appropriate. This process includes tracking and getting CB approval for any corrections that need to be made as a result of Third-Level QA.