Emerging evidence indicates that collaborative improvement is a cost-effective way to improve health care quality in diverse cultures. Such improvement generally relies on data from quality improvement (QI) teams’ own assessment of their facility’s performance and results. The validity of self-assessment data is important to both the teams themselves and to the collaborative as a whole: These data provide QI teams with the information they need to identify quality problems and to learn whether their actions actually improved quality. 

This report presents the results of an investigation of sequential validity of self-assessment by service providers in an improvement collaborative in the Mtwara Region of Tanzania. Study objectives were to determine the validity of self-assessments by the QI teams and whether validity improved during the first year of the collaborative.

The study was carried out in nine health care facilities participating in an improvement collaborative in Mtwara Region of Tanzania, during its first 10 months of activity. The collaborative is addressing HIV/AIDS care, particularly as it relates to antiretroviral therapy (ART) and the prevention of mother-to-child transmission of HIV (PMTCT). 

The research team defined eight activities in the self-assessment process that can influence the validity of the information that results: 1) writing the records, 2) storing and retrieving records, 3) selecting records from which to abstract data, 4) abstracting data from the selected records, 5) summarizing the abstractions, 6) the agreement of computer and written records, 7) the quality and use of computer records, and 8) communicating the summary data (results related to improving the quality of care) to other members of the QI team and the clinical staff. The team then developed and tested forms and procedures for measuring the validity of the information each activity produced.   

The study found significant upward trends in measurement scores occurred for the tasks of writing the record, selecting the sample, the use of computer results, and communicating results. No significant change in validity occurred in storing and retrieving records, abstracting or summarizing selected records, or agreement of written records with computer records. However, some of these activities started high and remained high throughout the study: For retrieving records, validity was close to 100% in the first and last two measurements for most cases; for abstracting records, a small increase occurred in validity during the study for all three indicators but was significant for only one of them; and for summarizing abstracts, errors were zero or close to it throughout the study. Changes in validity were roughly the same for all three indicators.

Over the course of the study, validity either improved or started and remained high for most self-assessment activities; none decreased. The communication activity, which differs from the others in that it does not contribute directly to the validity of the performance scores reported by the QI teams, had a very low end-of-study score across all sites, suggesting limited use of data for QI activities. With few exceptions, this study shows that self-assessment as part of Mtwara improvement collaborative provided valid data and improved as the collaborative matured. This finding—coupled with the result that some steps in the self-assessment process, such as storing and retrieving records and communicating results, are not always done well—suggests the need to address these activities early in a collaborative. The finding that the validity of abstracted data between QI teams and the gold standard set by the expert reviewers was not statistically different is especially encouraging.