Technical Notes

1. How did we decide to color-code performance red or green on a numeric indicator?
2. How does risk-adjustment work?
3. Where did these indicators come from?
4. Where are the data sources for these numbers?
5. What are some of the known limitations of our report on these quality and safety indicators?

1. How did we decide to color-code performance red, yellow, or green on a numeric indicator?

We use an objective statistical test. We apply the red and green coloring only if the difference from the benchmark, or average, is more than 95% - and is not just random variation. If the national average is within 95%, we consider our results "near the national average." Otherwise, we color-code our performance better (green) or worse (red) than the stated average.

Because patient satisfaction surveys can be based on very large samples, standard statistical tests can be too sensitive, color-coding average performance as red or green. To avoid this problem, we have not allowed the sample size for significance testing to be larger than that required for 80% power. This refinement rarely changes the color coding, but it keeps the statistical test from labeling a 0.4% difference red or green.

Top

2. How does risk adjustment work?

Risk adjustment is a mathematical calculation that takes into account differences in patients and procedures. We use either the analysis provided by the national organization that supplies the comparative data, or - if that is unavailable - we use standardization by the indirect method, which is the usual approach.

Example: Imagine results like those shown in the table. The "All patients" row shows that the U.S. complication rate is 8%, while the hospital's complication rate is 16% - twice the national average.

	# with complication	# of cases	Hospital's complication %	U.S. complication %
Low-risk patients	7	200	3.5%	4%
High-risk patients	73	300	24.3%	25%
All patients	80	500	16.0%	8%

Now imagine classifying patients as high or low risk. For example, low-risk patients might be patients under age 75 who have no serious medical conditions, while high-risk patients would include everyone else. Looking at the table above, suppose that the hospital's low-risk patients do slightly better than the national average (3.5% vs. 4%), and so do its high-risk patients (24.3% vs. 25%). The hospital's overall complication rate is high because it sees a larger proportion of high-risk patients than the average U.S. hospital does. Instead of showing the hospital's complication rate as twice as high as the national average, it should be shown as slightly better than the national average.

Calculation. Indirect standardization predicts what each hospital's rate would be if it had the same complication rates as the nation has in each risk group. So, looking at the national percentages, we predict that 4% of the hospital's low-risk patients and 25% of its high-risk patients will have a complication. That means that the hospital's predicted number of complications for all patients is 4% of 200 + 25% of 300 = 8 + 75 = 83 complications. A standardized ratio may be calculated by dividing the hospital's actual number of complications by the predicted number, which is 80/83 = 0.964. The hospital does not actually have 100% of the complications it is predicted to have using national averages; it has only 96.4% of the national average. Multiplying the 0.964 ratio times the national rate of 8%, gives a risk-adjusted rate of 7.7%.

So, if we don't risk-adjust the hospital's rate, it has a complication rate equal to twice the national average. If we do risk-adjust the hospital's rate, we give it credit for its tougher cases. We say the hospital's risk-adjusted complication rate is 7.7%, compared to the U.S. rate of 8% -- which gives a much more accurate representation of the hospital's performance.

Top

3. Where did these indicators come from?

Methodist Medical Center is responding to lists of indicators and safe practices endorsed by national healthcare organizations. Click on an indicator to find the national organization endorsing the particular indicator. This comprehensiveness is part of our assurance to the public that we give a complete picture of our quality.

Top

4. Where are the data sources for these numbers?

We use Premier and the most recent software programs available from The Agency for Healthcare Research and Quality (AHRQ) to calculate risk-adjusted (but not "smoothed") rates for the hospital PSIs.

Premier is a nationally known comparative, risk-adjusted hospital database. By using Premier's risk-adjusted methodology, Methodist can compare their performance to other hospitals.  The data available to hospitals participating in Premier include risk adjusted comparisons for like conditions (heart failure, heart attack and pneumonia). To ensure a direct comparison, Premier requires Methodist to go through a series of standard quality controls to ensure data integrity before data is submitted.

One of the comparisons that are available in Premier is the AHRQ patient safety indicators (PSI).  The AHRQ PSI's are used in Premier to calculate the complication index.  Other available comparisons used by Premier are the DRG or APR-DRG diagnosis groupings developed by 3M.

Joint Commission/CMS data, including the Surgical Infection Prevention (SIP) program. These include several of the NQF Hospital Care indicators.

These data come from nurses reviewing paper and electronic medical records. After various audits and reliability checks, we enter the data into Premier and send the results to national databases. We display data from Joint Commission feedback reports, and we use Joint Commission or CMS public sites to calculate the Illinois median performance.

Infection control data.

Infection control nurses review medical records according to National Healthcare Safety Network (NHSN) guidelines from the CDC (Centers for Disease Control and Prevention).

Patient satisfaction.

Methodist Medical Center of Illinois uses an external company, Press Ganey Associates, Inc., to conduct and analyze mail surveys of a statistically valid random sample of our patients. Because the questions and methods differ from one survey to the next, it is not valid to compare the results shown here to results from patient satisfaction surveys other than those conducted by Press Ganey Associates, Inc.

Top

5. What are some of the known limitations of our report on these indicators and safe practices?

Perhaps the most important limitation is that the nationally endorsed lists cover so little of what prospective patients might want to know about a hospital's performance. Much more extensive information is needed to evaluate hospital care at the level of specific procedures and conditions - rather than trying to capture hospital-wide complication rates, for example. There are almost no indicators that address outpatient care or events that occur after the patient's hospital stay. The current lists of indicators are essentially silent about the patient's long-term survival and condition.

Current medical records codes do not capture important factors that should be used - but can't be used - to adjust the statistics. For example, the data used by these indicators do a poor job of distinguishing an infection the patient already had, from an infection the patient developed in the hospital. The data do not distinguish an emergency case from one where more time was available to react. The data do not indicate if the patient had "do not resuscitate" orders, which would indicate that the patient's death was expected and not a result of the care provided. Hospitals also differ in their documentation and coding practices.

Premier attempts to adjust for those differences through risk-adjustment. We are probably not risk-adjusting the PSIs as much as they should be. This limitation may be trivial for some of the indicators, but may lead to greater inaccuracy for high-risk patients and procedures.

The number of procedures performed is at best a proxy for other quality indicators. Some authorities suggest not using these volume-based indicators at all; others suggest using them only in conjunction with other indicators of quality of care.

We cannot be certain about the comparability of the U.S. and Illinois averages. The U.S. average may be based on a biased sample of states or hospitals. For example, the average on a particular indicator may be too high, because it is based only upon hospitals proud or interested enough to submit their data to a national group. Presumably, the comparative average could also be too low, if - for example - states with high-risk or older populations are over-represented in the data.

Patient satisfaction. The only U.S. comparison available for our patient satisfaction results is the average of the relevant Press Ganey database. Although more than 1,300 hospitals participate in the Press Ganey surveys, those hospitals may not be representative of all hospitals in the U.S. In other words, we may be comparing ourselves to an average that is easier or tougher than it would be if it included every U.S. hospital. Also, patient satisfaction results are not risk-adjusted, so they do not take into account the different services that different hospitals provide. Patients who are in the hospital to deliver a baby may tend to rate their hospital experience differently from patients who are in the hospital because they had a heart attack. Patient satisfaction averages do not take these expected differences into account.

Data from one-day prevalence studies and limited staff questionnaires are subject to time-of-year and low-volume variability and may not accurately represent what more complete data would show.

Although we follow national definitions, we still have countless judgment calls to make about how to display data, how to classify data for some indicators, etc. We hope that the Methodist Quality Report helps contribute to the growing national and state interest in quantifying hospital quality performance and helps hasten the day when hospitals will have agreed-upon standard approaches to these decisions.

Top