What is it?

ProPublica, an investigative journalism group, created the Surgeon Scorecard to give people the opportunity to learn more about the complication rates and death rates of patients associated with particular surgeons prior to their surgery.  Ideally, this searchable scorecard is supposed to guide patients’ decisions in determining which doctor should perform their operation based on the surgeon’s track record and protect patients from high risk doctors.

What procedures does it cover?

Released in July 2015, this software program focuses on eight different elective procedures:  1) knee replacement, 2) hip replacement, 3) laparoscopic gallbladder removal, 4) posterior column lumbar spinal fusion, 5) anterior column lumbar spinal fusion, 6) prostate resection, 7) prostate removal, and 8) cervical spinal fusion.  They selected these eight categories because they are common procedures done every day across the country.

How was data collected?

ProPublica collected their data by looking at Medicare billing records for in-patient hospital stays between 2009-2013.  They analyzed 2.3 million procedures, a combination of surgical operations from all eight elective procedures mentioned above.  ProPublica only focused on surgery-related patient readmissions to hospitals up to 30 days after the date of patient’s initial operation.  This included readmissions within the first 30 days of operation for blood clotting, uncontrollable bleeding, infections, or misaligned orthopedic devices.

How does it work?

When you go to the Surgeon Scorecard’s home page, you are given the option to select a location, a surgeon, or a hospital.  You are then shown a scale of adjusted complication rates and given a percentage.  The lower the percentage, the fewer documented complications.  Ideally, you would like to select a surgeon who has a percentage rate the low-medium ranges on the graph.  The graph also shows where the surgeon falls within a 95% confidence interval.  Medicare only allows ProPublica to report complications of 11 accounts or higher.

Thoughts from the American College of Surgeons:

  1. The American College of Surgeons (ACS) is not convinced, as evidenced by the lack of validity with this Surgeon Scorecard. They argue that if the severity of the patients’ condition would have been accounted for, the data would be more accurate.  For example, ProPublica does not seem concerned if the patient was an 80-year-old diabetic male with heart disease or if the patient was a healthy 40-year-old male with no known health concerns.  By not factoring in the surgeon’s success rate on more challenging patients, the potential for wrongly directing patients away from these surgeons undoubtedly increases.
  2. ACS also mentions that it is important for ProPublica to consider collecting data from private payers, government records, professional societies, and public interest groups to collect the most accurate clinical data.
  3. Another point that ACS argues is the fact that surgery is a team experience. Surgical nurses & anesthesiologists are very involved in the surgical process, indicating multiple factors and individuals contribute to the surgical outcome – not all the responsibility lies solely with the surgeon.

Limitations of Surgeon Scorecard: 

  1. ProPublica’s data collection is very restricted. It only looks at 5 years’ worth of data from one source – Medicare.  This means no data is collected from private insurance companies, so the data that was initially collected by ProPublica could be significantly skewed.
  2. The Surgeon Scorecard could give surgeons a poor and inaccurate reputation due to the lack of validity of their results. This could have a negative impact on surgeons because some insurance companies may begin restricting patient access to them in the future due to results shown on the scorecard.
  3. ProPublica only includes elective cases in their research because elective procedures tend to involve healthier patients with the greatest chance of a smooth recovery. This is also not an accurate representation of the patient population because not all of those eight specific procedures are majorly elective.  It is probably that hip replacements or gallbladder removals, for example, may need to be performed as an emergency surgery.
  4. The Scorecard does not factor in any complications that may arise 30 days or more from the date of initial operation.
  5. Also, emergency room cases are not included in raw data, nor are out-patient Medicare cases. The data comes solely from in-patient hospital stays.
  6. No case-specific analysis was done. Surgeons were graded solely on billing information sent to Medicare.  No chart review or further individual investigation was conducted.

Conclusion:

Although their intentions may be good, ProPublica is a non-medical, non-scientific organization that is analyzing complex surgical data.  They have a long way to go to increase the validity of their project.  Their methodology needs improvement, and there are multiple other factors that need to be accounted for in order for their findings to be accurate.  According to physician Jeffery Parks, who analyzed himself using the Scorecard, the consensus of among practicing physicians is that they are all bewildered by it and question the accuracy of its findings.  Dr. Charles Mick, of the North American Spine Society, takes a more positive approach and hopes this is a step towards a culture of transparency where mistakes and complications can be discussed openly and not be hidden.  Overall, there is a benefit for patients in that that they are given the opportunity to research surgeon’s complication rates prior to their surgeries.  Another positive aspect of the Scorecard is that it could serve as an accountability measure for surgeons and other staff in the operating room.  However, the only way this Surgeon Scorecard will work as intended is if more precise, all-encompassing data is collected and used to present truthful and accurate statistics to the patients and surgeons, alike.