You be the judge: How we compiled the hospital ratings

^(iStock)

CBC News' "Rate My Hospital" project is getting a great deal of national attention, and for good reason: it's the first comprehensive compilation of data about Canadian hospitals, organized in a way that allows patients and families to see how well their hospitals are performing on a wide range of metrics.

Canadians are giving it very positive reviews, telling us that they're pleased to have, for the first time, a way of assessing the care they can expect.

In an earlier blog post we talked about the reasons why the fifth estate and our online staff launched the project.

There have been some criticisms, most notably from the Ontario Hospital Association. They didn't like Rate My Hospital very much, including our A+ to D ratings system, suggesting that our data wasn't gathered properly and couldn't be trusted.

You be the judge. Here's how it was done.   

First, the grade each hospital received in our report card is based entirely on empirical data that is reported by the hospitals themselves.

To calculate the hospital grade, we used data that each hospital collected and reported to the Canadian Institute for Health Information (CIHI). This information reflects how well or poorly patients do after having surgery or medical treatment at a particular hospital.  

CIHI analyzed the data and made adjustments to account for differences in hospital size and the types of cases they handle. The adjustments allow for fair comparisons between hospitals.

Our CBC News report card uses the same model. It rates hospitals only against other hospitals of the same size, as defined by CIHI.  (This addresses one of the major criticisms of the Ontario Hospital Association)

Our fifth estate/online team rated hospitals on performance indicators that measure patient results after common medical and surgical procedures. We limited our ratings report to five basic indicators in order to fairly compare the largest number of acute care hospitals possible.

We recruited a front-line expert international panel to assist us in this work.

The panel met in Toronto in November 2012 to develop an outline for the report card. Their input was supplemented by Jason Sutherland and Peter Austin, Canadian biostatisticians noted for their work in evaluating health system performance, who developed the statistical methodology used to rate hospitals.

Hospital ratings are based on data collected and reported by over 600 acute care hospitals in Canada and analyzed and published by the Canadian Institute for Health Information (CIHI). The data represents hospital performance in the fiscal years 2010-11 or 2011-12.

We sought supplementary data from provincial health ministries, which proved a struggle. the fifth estate requested the information, but 12 of the 13 provinces and territories refused to provide it. Prince Edward Island provided the data after a request was made to the Privacy and Information Access Coordinator at Health PEI, the agency that oversees health services in the province.

As we learned through Access to Information requests, a co-ordinated strategy was formed and followed to block our efforts. Despite the the exceptional measures the provinces took in this respect, we persisted using public records and, of course, the CIHI trove of information.

In addition to the hard data it amassed, our team also conducted three large surveys, one of patients, one of nurses--those results were reported Monday--and another involving hospital CEOs. The results of these surveys were used for news stories and were kept separate from empirical data.

We asked the chief executive officers of about 680 acute care hospitals to complete the survey this January; 107 agreed to do so.

The survey included questions about 18 measures that are associated with safety, patient experience and quality of care in a hospital.

The measures we looked at include nurse staffing levels, hand hygiene compliance, the use of electronic medical records, use of safe surgical practices, the extent of palliative care services and a range of patient services designed to address psychosocial needs by accommodating family and friends.

As we compiled and collated our date and prepared it for broadcast and publication, we sought and accepted advice and input from our expert panel at every turn.

The result speaks for itself. It is a major step in providing accessible, accurate information so Canadians can ask the right questions and make informed decisions about their health care. It should not replace the info your doctor can provide and the care he or she is trained to provide. On that last point we agree with our critics.