Notes from Using the Beam to fight COVID-19, presented by Bryan Hunter at Code BEAM V 2020.

  • HCA Healthcare currently covers about 186 hospitals, 2000 surgery centers, emergency rooms, clinics.
  • Project Waterpark started in 2018 as part of the Enterprise integration group.
  • HCA Healthcare needed to work with HL7 data so they built and open-sourced a library for this, elixir-hl7.
  • Nurses often work 12-hour shifts to provide continuity-of-care for patients and to reduce the frequency of dangerous handoffs. 1
  • In Waterpark, they run long-lived genservers (sometimes weeks!), one per patient, holding all the state for that patient, continuously updated as HL7 messages come in or go out.
  • They can’t ever go down, or take systems down for maintenance, because of this design.
  • They run read-replica processes in different locations (availability zones)
  • Bryan talks a bit about how their long-lived processes and replicas model nurses running long shifts to reduce handoffs.
  • Server Hashrings are used to map Processes to locations. This lets them avoid needing a global registry.
  • Every Process has a key (made up of patient ID and facility, IE hospital), projections and an event store in its local state.
  • COVID-19 Long term care facility alerts
    • COVID-19 propelled Waterpark from a neat tech experiment to a crucial project
    • When a new patient is admitted, they check whether they are a long term care patient.
    • Hours later, a message comes in signaling the patient tested positive for COVID-19.
    • At that point, if they were a long term care patient (check 1), a notification is sent to the case manager.
    • This allows the facility to take extra measures, like preemptively testing everyone there.
  • Initial R&D was Bryan Hunter on his own for the first year.
  • Today (late 2020) there are 5 developers on the team for Waterpark.

  1. From https://www.jointcommission.org/assets/1/6/tst_hoc_persp_08_12.pdf: “Ineffective hand-off communication is recognized as a critical patient safety problem in health care; in fact, an estimated 80% of serious medical errors involve miscommunication between caregivers during the transfer of patients. The hand-off process involves “senders,” those caregivers transmitting patient information and transitioning the care of a patient to the next clinician, and “receivers,” those care-givers who accept the patient information and care of that patient. In addition to causing patient harm, defective hand-offs can lead to delays in treatment, inappropriate treatment, and increased length of stay in the hospital.” ↩︎