Delivery by a skilled birth attendant (SBA) serves as an indicator of progress towards reducing maternal mortality worldwide – the fifth Millennium Development Goal. Though WHO tracks the proportion of women delivered by SBAs, we know little about their competence to manage common life-threatening obstetric complications. We assessed SBA competence in five high maternal mortality settings as a basis for initiating quality improvement. The WHO Integrated Management of Pregnancy and Childbirth (IMPAC) guidelines served as our competency standard. Evaluation included a written knowledge test, partograph (used to record all observations of a woman in labour) case studies and assessment of procedures demonstrated on anatomical models at five skills stations. We tested a purposive sample of 166 SBAs in Benin, Ecuador, Jamaica and Rwanda (Phase I). These initial results were used to refine the instruments, which were then used to evaluate 1358 SBAs throughout Nicaragua (Phase II).