Below is an excerpt from an email written by my colleague at the University of Michigan, Dr. Ulysses J. Balis. He currently serves as the Director of Informatics in the Department of Pathology. This is his response to a hospital physician who asked him why major systems fail.
1. Application Not Designed to Scale: Because not all database programmers are created equal, neither are the mountains of code that they produce. An inappropriately recycled segment of code designed to perform multiple joins for a complex report retrieval can quickly generate highly non-linear query cross-products. If such a returned table has, for example, 40 rows, the execution time will be .020 seconds. However if it contains 4500 rows, the execution time can reach 30 minutes. This difference is due to the intermediate query cross-products that take up to several billion (not million) rows to fully populate. With the use of some simple intermediate tables, this cross-product problem can be easily avoided and the system performance will scale linearly with set size. Because it is quite common in deployment timetables to ignore system load-testing prior to go-live, it is also quite common to launch a new system and to discover only then that the application cannot respond to the live transaction volume. This is the class of deployment error plaguing Kaiser at present (link here).
2) Inadequate Resources to Properly Deploy/Enhance the Application: No commercial product comes ready to deploy in a hospital "out of the box". Some local additional effort will always be required at the time of deployment including the need for co-development of the application to allow it to perform at the expected level of functionality. If neither the vendor or the client can provide the resources to make these necessary changes, the final build will be inadequate in scope and the deployment will fail. This is the prototypical area of IT software deployment where more time or more development resources can prevent a go-live failure. However, throwing additional resources at a project immediately prior to go-live is not a sure thing because 10X additional programmers does not necessarily equate to 1/10th the completion time for a code change.
3) Inadequate Training of Users: Regardless of the quality of the software deployment, if end-users are not ready for the change, the deployment will either greatly suffer or fail. A common pitfall is to train users and then to push back the go-live date because of the need for more development/fixes. In the interim, users will invariably forget most of what they learned during training and the ensuing go-live will be a disaster. A simple operational rule mitigates this reality: If you move the go-live date by more than two to three weeks, you must retrain the entire staff. The new training regimen can be an abbreviated but it must be offered shortly before the new go-live date. People generally do not retain skill sets that they are not actively using.
Comments