Calculating System Availability: Lessons from Kaiser

This note is a joint effort by myself and my colleague and Director of Pathology Informatics, Department of Pathology, University of Michigan Medical School, Dr. Ulysses J. Balis.

In a recent blog note (see: Kaiser & Epic Respond to Justen Deal's E-Criticism), an Epic spokesperson is quoted as saying that the EMR had an availability to Kaiser employees of 99.5%. In another recent note in Lab Soft News, Dr. Howard Landa, CMIO of KP Hawaii, says that they are striving to achieve system availability of 99.8%. In a Modern Healthcare article, Justen Deal is quoted as saying that the Kaiser EMR was typically "down for thousands of user hours a month."

Whom are we to believe here? One response would be to treat all estimates of system availability from Kaiser, or anywhere else, with extreme caution without an explanation of exactly how the data were gathered and calculated. Another perspective would be be assume that all of these estimates are correct from the perspective of the individual quoted. The reason for this conundrum is that there are two vastly different ways to define computer system availability -- the vendor/vendor-ally view and the end-user view, which can be defined in the following ways:

• Vendor/vendor-ally view: If any one individual service from myriad possible services in the total application suite is available to any single user in any hospital, the system is operating (system up).
• End-user view: If any individual user in any hospital can't log on to any service in the total application suite and complete a desired task, the system is not operating (system down).

We believe that the fairest way to approach this problem of describing system availability is to use the Services Oriented Architecture (SOA) approach and think in terms of the total set of services offered within an EMR suite rather than in terms of access by individual users to a service. In order to summarize system availability, one would then express system availability as the percentage of services available for a particular percentage of the time (e.g., 98% of services available for 99% of a day). Because some EMR services are more critical than others (e.g., writing drug orders by a physician), emphasis would need to be placed in such a report on those that are mission critical. Lack of service availability due to network, PC, or wetware failure, of course, are excluded.

We are purposefully ignoring here the Epic "solution" being used at Kaiser involving multiple instances of the EMR because, as we understand it, the single application would not scale up. Some individuals have been quoted as saying (link here) that as many as 15 instances of the Epic application are now running simultaneously at Kaiser. Presumably, each of these instances would have its own database which can get out of sync with the others. Moreover, the system availability report described above would need to be calculated for each instance. Otherwise, one would need to aggregate the uptime of each respective service for all the instances into one cumulative record and then report service metrics based upon concurrent availability of each service across all instances.  Either of these fascinating forays into set and graph combinatorics theory would boggle the mind.

Comments

[A.B.]

Pointing out how Care Everywhere is supposed to work, and then demanding that "the deriding of Epic's ability to scale should stop" are two separate things.

The one thing Kaiser, Epic, and Justen Deal seem to agree on is that Epic is not available 99.999% of the time. From inside KP, the points made in the Computerworld are obvious: the system is not scaling, and it's having a big impact on workflow and care.

It isn't just databases (Cache, in our case). It isn't just Citrix. It isn't just the network. It isn't just this or that. It's all the things, working together, or not working together in the case of HealthConnect.

As they say, if you can't take the heat, get out of the kitchen. Computerworld has a 722-page report outlining thousands of hours of outages, including dozens of situations in which patient card was dangerously impacted. Simply saying the system can and does scale doesn't cut it anymore. It's not true, and most people see that now.

[gadfly]

It seems to me the "presuming" is coming from the people who are actually experiencing the failure of the system. There seems to be a huge gap between the theory of database distribution and the reality of downtime. And all the PR-pushing isn't just a matter of business as usual: quality of patient care is at stake.

[Anon]

re: Epic scalability. I wish folks wouldn't "presume" unless they know more about the underlying technology. Epic utilizes a distributed database structure for their largest customers - such as Kaiser - so that the system acts as one large database, but not all the data resides on one physical server. This architecture is how most computer databases are structured these days, so it shouldn't be seen as a disadvantage. Epic has technology (termed "care everywhere") that allows the distributed databases to sync up - i.e. I'm a Kaiser patient who lives in southern california and I'm on vacation in Colorado. I go to the Kaiser clinic in colorado, they pull my record up in Epic (it is "transferred" from the So. Cal. instance), enter new data on my chart, and then the new data is synched back up with my home instance in So. Cal. So to the patients and providers it's invisible - the data is complete in both instances, and the charges go where they should as well. Tricky stuff, but it's the only wa that a nation-wide EMR can really work effectively.

So, the point is that the deriding of Epic's ability to scale should stop. It's not true - their database goes bigger than most others, and with their other tech innovations it allows the database to be distributed & go even bigger.