February 29th, 2008 andrew
We know we want good semantics in messages and the best way to get that is atomic data in a standards based format. However the roadblock we run up against is that its not sexy enough!
The availability of computers rather than typewriters for Medical Notes means that the visual presentation of a clinic report or letter has become important to many and when you suggest that readers are really interested in the content rather than the presentation you often get that sinking feeling that you have just lost them.
So it seems to keep everyone happy we need both. However that sexy display needs to be portable and ideally standards based. It is also desirable to have some semantics in the display so a consumer of the report can eg. Click on a drug or diagnosis to drill down for more information. Ideally combining the atomic data in the HL7, which for example include images, with a standards based display format using xHTML and CSS, would fit the bill. Using Microformats it would also be possible to mark up diagnoses and drugs with there SNOMED-CT representation.
eg. Say you wanted to say the patient Had “Large B-Cell Lymphoma” – as Text this is not something you can reliably parse out, however if you used Microformats in xHTML you could represent it as:
<span title=”64572001|Disease| : 116676008 = 46732000|Malignant Lymphoma, large B-cell diffuse|”>Large B-Cell Lymphoma</span>
Or for usual more compact usage you could just use the SNOMED-CT codes alone:
<span title=”64572001 : 116676008 = 46732000″>Large B-Cell Lymphoma</span>
Now this embedded in the display text has real meaning and could be used as a link for disease information for example. The Australian HL7 V2 standard defines a html display segment, as the last OBX in a message and by transmitting the atomic data in the preceding OBX segments for machine usage and including a xHTML display segment you get the best of both worlds, atomic data and sexy display. If you add microformats to the xHTML then the display still has semantics that are safely extracted. The SNOMED-CT grammar can also go further and define such information as “This is a past history or Family History of a disease” which extends the semantics further.
Currently HL7 Freetext is used for display, and the only real formatting available is bold. This is causing some user pushback and a move to xhtml and CSS (probably with no javascript and inline CSS) would allow the embedding of good display into messages containing atomic data. HL7 CDA – Clinical Document Architecture allows to a html display segment as well so the concept is usable here as well. The alternatives would be rtf display, but xhtml is far more semantic and standard and is likely to be more inter-operable. There are free xhtml validators on the web and some form of view control is installed in virtually every OS available, which increases the attractiveness of xhtml.
It is an idea we are exploring at this time.
Posted in EHR, HL7, MESSAGING, SNOMED, STANDARDS, XHTML | No Comments »
February 26th, 2008 andrew
Analysis of SMTP
for Secure and Reliable Document Delivery.
By Peter R. Tattam
It has been proposed that secure email in the form of S/MIME over SMTP (Simple Mail Transfer Protocol) be used for the transmission of documents such as Electronic Medical Records (EMR) with high reliability. This article discusses the limitations of using SMTP for such delivery based on its poor suitability to meeting the requirements for high reliability document delivery, in particular in the context of that required for EMR transmission.
To begin discussion, one needs to define what the requirements for high reliability of document delivery might look like. There principal factors are as follows.
- Timely Delivery. One should expect that a document be delivered within an acceptable timeframe. This can vary depending on the needs of the document, however for many types of EMR, delivery times are expected to be in the order of seconds or minutes, especially when the nature of the data they convey is likely to age quickly.
- In-Order. One should expect that documents originating from the same source be delivered in the same order in which they originate from the source.
- No Duplicates. One should expect that an original document be received once and only once at the destination.
- Detection of delivery failure. One should be able to notify the sender that delivery failed for whatever reason.
- Secure Delivery. One should expect that a document be delivered intact and free of alteration and also be subject to typical expectations of a secure transmission. There are several aspects of this but the principal components are encryption and authentication.
- Safe from Unsolicited input. One should expect that documents be originated only by originators who are authorized to transmit such documents and that the system is free from unsolicited documents being injected into it.
- Safe from Hijacking. One should expect that the delivery system at both ends be predictable and free from interference.
This list is not necessarily exhaustive, but covers most of the needs required for reliable delivery.
Now let us see how SMTP deals with these factors. I base my analysis on the experience I have gathered in more than 15 years in the development of email clients and servers and from managing an ISP.
Timely Delivery.
The SMTP system can transmit an email through its system very quickly under ideal conditions. Typically, times are of the order of < 1 minute and comprise the sum of the following components.
1. Transmission by the MUA (Mail User Agent) to the first MTA (Mail Transfer Agent).
2. Transmission by Intermediary MTAs to other Intermediary MTAs or the Destination MTA. This step may occur zero or more times.
3. Receiving the document from the destination MTA to the destination MUA.
In step 1 and 3, there may be hidden processes that the user is unaware of, such as virus scanner software or corporate email conversion gateways when the internal mail protocols is not SMTP. Some of the steps involve complex and time variable operations such as looking up the DNS system which further compounds any delays.
There are many factors that can impede the throughput of the MTAs. These include server failures due to hardware failure, server overload due to SPAM (unsolicited email), email worms, and other hostile Denial-of-Service (DoS) attacks directed specifically at a given server. In the short term, email may be delayed anything from 30 minutes up to 4 hours (depending on site configuration) if an intermediary email server is not available for reasons such as network deterioration of any sort, or general unavailability due to maintenance etc. Generally users of email accept that delivery is not necessarily immediate, their experience being that of the real world “snail mail”, the physical postal system. A minor temporary delay caused by a communication failure between MTAs can often cost up to several hours even when the temporary delay lasts only for a brief time like a few seconds. There is no way for a MTA to determine when to resend an email based on network conditions although some servers do have some degree of intelligence if there are a large number of emails waiting to be sent to a particular MTA. Generally if an email can’t be sent, it will be queued for resending with a retry of between 30 mins to 4 hours. There are no rules for how often email should be resent – it is entirely arbitrary and at the whim of the administrators of the MTAs.
However, when a mail server is in a non-optimal state, such as when an internet worm is at the peak of its activity or a rather active group of spammers are busy, mail can be delayed for more than 24 hours, sometimes several days, especially if the source of unsolicited email into the system cannot be controlled. Clearly such a level of delay for important documents such as EMRs would be totally unacceptable, and generally when such events occur, they are totally outside the control of the sender or receiver of these documents.
Another contributory factor towards server delays is that very often in a large organization such as a Telco or large ISP, the mail server may be providing service for 100,000 users or more. Having that many user active, although not necessarily concurrently using the server, does place a high demand on general throughput, and often throughput is sacrificed for scalability.
In Order
There is no concept of ordering delivery of emails in SMTP. Mail will get delivered whenever it can. Mail can and will arrive out of order, especially if delays occur for the many reasons outlined above.
No Duplicates
Generally, the mail system will not send duplicates as usually the MTAs will take ownership of an email as it transitions through the system. There are however occasions when these systems fail in various ways, and it is possible to receive duplicate emails, for example if a MTA has suffered a catastrophic failure and needed to be restored from a backup, or a poorly configured server having been fixed. Failures and mistakes do occur in the administration of sometimes unwieldy MTAs. One other source of duplicates is that due to significant delays in transferring an email, a sender may resend the message.
Detection of Delivery Failure
The SMTP system by nature is a push system. There is no inbuilt mechanism to notify that the MUA of the recipient has received the document. Although there are modest extensions to SMTP for the acknowledgement of delivery, they generally rely on end-end acknowledgement of some kind in the MUA software.
Secure Delivery
This is a wide topic and encompasses the issues surrounding S/MIME. As with all security related issues, absolute best practice in the industry should be followed at all times. For truly secure transmissions to take place, both encryption and authentication with fully verifiable PKI must be followed. Any weak link will reduce the ultimate security. Proof of identity through authentication is absolutely necessary.
Safe from Unsolicited Input
SPAM, internet worms and viruses, and mail bombing can have a very adverse affect on both MTAs and MUAs. The essentially anonymous authentication system which the SMTP system is based around allows for unsolicited email to enter the system unchecked. While good configuration of servers is important to prevent SPAM, spammers are evolving more advanced methods to circumvent these measures. Also the proliferation of consumer based computing systems with less than average immunity to Viruses, Trojans and various other Malware has seen the rise of internet worms which can penetrate even the best engineered consumer ISP networks.
Safe from Hijacking
While technically a security related issue, there is a range of attacks at the network management layer which could interfere with the safety of email transmissions. When a MUA sends and receives emails, they generally do so via an internet account of some kind. For transmission of email, the MUAs use SMTP, the same protocol used for MTAs to communicate, however at the final receiving end, MUAs typically use POP3 or IMAP to receive the emails from the MTAs. This invariably requires an authenticated login of some kind. While there are measures to prevent eavesdropping of usernames and passwords in such a service, generally one is restricted by what the ISP supplies. Often there may be no choice of authentication method (plaintext passwords are common), and the ISP will often not allow the customer to operate their own SMTP server at the customer’s premises, nor will they allow direct access to SMTP servers other than those operated by the ISP. Since passwords are sent in plaintext over the network, it is entirely feasible that the account login may be hijacked, possibly without knowledge of the recipient. For example an eavesdropper may read mail without leaving a trace, and could also possibly remove emails without the genuine recipients knowledge. Even if the emails are secured with S/MIME, they could still use the knowledge to harvest email addresses for other purposes such as denial of service. Emails when sent and received carry a good deal of additional header information, and this information could be used by a hostile attacker to find weaknesses in the SMTP systems which comprise the virtual EHR system.
Other studies
There have been a number of studies made on the reliability of email for general use. A Google search on the phrase “email reliability” brings up a number of useful links. Here are some references to get started.
http://en.wikipedia.org/wiki/Email discusses the basics of email.
http://itmanagement.earthweb.com/columns/executive_tech/article.php/3500506 cites some studies.
http://www.broadbandreports.com/shownews/36176
http://www.emailwatchdog.org/
http://www.terryscomputertips.com/computers/email-reliability-in-this-internet-world.php
http://www.cbsnews.com/stories/2005/02/28/tech/main676962.shtml
Conclusion
This discussion only scratches the surface of the issues. Given the large number of points of failure for the SMTP system, it could hardly be relied upon for timely delivery of messages containing important documents where the age of the information is important. As for security, as long as best practices are followed, S/MIME should provide at least the same level of security as SSL or PGP encrypted HTTP since it uses tried and true Public Key Encryption. That however needs to be tempered with the reality that the login accounts of MUAs are still prone to hijacking in various forms. Since there are other more suitable protocols available that provide a more direct connection between endpoints (e.g. HTTP), one should consider that S/MIME be used only as a last resort when all else is unavailable. One would have a very poor internet experience indeed if the only services available were SMTP and POP3. That the SMTP system is the target of choice for the delivery of SPAM and other Malware should be sufficient warning to look at the alternatives.
My considered opinion is that using SMTP for the secure delivery of EMR is at best a poor second to better protocols like HTTP or SSH, and at worst a recipe for disaster - lawsuits from both the civil and medical fronts just waiting to happen.
Posted in EMAIL, HL7, MESSAGING, STANDARDS, Uncategorized | No Comments »
February 17th, 2008 andrew
It seems that after spending over $300M trying to kick start eHeath we are going to have an eHeath plan! Seems to me like that is a very good idea and the question is why don’t we have one already and what should be in “it”.
It’s an eHeath plan drafted in 3 months and will last 5-10 years! Still doesn’t sound quite right to me but any plan is better than no plan. Unless something major is changed its likely to be formulated by people with more political clout than technical expertise so maybe some ideas from the trenches would help.
Firstly, we don’t want a grand plan as they usually fail. We want to put in place measures that get us to a point that a grand plan is a possibility. A grand centralised EHR will not work unless everyone is using standards for their every day eHeath dabblings. That’s where the plan should focus. Australia has widespread usage of HL7 V2 and thats what we should aim for, but at a better level of quality. HL7 V3 has some nice modelling but is is not ready for widespread usage yet and the NHS in the UK is pouring buckets of money into that pond so we should swim in the functioning pool. CDA is not really any advance on good V2 and while the xml might be easier to parse it has no support in the current primary care applications and should not be an immediate target because of this.
What we don’t have is good quality HL7. The labs produce it but the receivers are fragile and the lowest common denominator is limiting what we can do. We need to have certification of the HL7 produced and certification of the applications ability to consume it. If this was done correctly then the reliability and quality of our clinical messaging would soar. We already have AHML able to certify HL7 produced and test message sets to test on applications so its easily organised! The State Heath Departments should be forced to only spit out 100% pure AHML accredited HL7, its not safe to do anything else and I am sure it could be done give enough push.
We do need provider directories, but almost all the lab messaging uses Medicare provider numbers for this and it works well. The grand all singing and dancing provider directory is a way off so why not use something that just works. Medicare Australia need to loosen their grip on the monthly lists of provider numbers for this to work but I am sure this could happen with only a modicum of counselling.
We need standards that allow you to send an entire patient history in HL7 V2 including SNOMED-CT encoded concepts. The archetypes in V2 project has the ability to allow this but this project has been running on a shoestring. Standards Australia needs to have its eHealth budget lifted by about 10 times. Currently there is one 1/2 time person for all the eHeath standards! Projects like this should have a full time person on them. It can take 2 months to get a document edited. This is not a Standards Australia problem, but a lack of funding support issue and it needs to be fixed ASAP. Someone, who can read and write HL7 in their sleep needs to be employed full time doing examples and testing concepts. Currently this is done by volunteers at midnight the night before the meeting.
We do also need messaging of course, but we need good quality messages before this is actually useful. The most useful option here would be to produce a standard and mandate its use. Even producing a standard would help as at the moment the “Use WS-Security” is nowhere near enough detail! How could this be done, this one would take some bravery but throwing together 10 technical (Not political) experts from the companies involved for a week with the expectation of a draft standard at the end of a week is probably the best option. Once that is done you could legislate that the receivers paid for the service if you were really brave. Next issue a directive to HESA (Now inside Medicare) to issue a site certificate and Individual certificate to everyone with a provider number and do it every time a new number is issued. You would also insist on a certificate signing service so certificates could be generated onsite, maybe using a valid individual key to upload it for signing.
Now drugs, we need SNOMED-CT codes and they need to be in the PBS by the end of the year. They also need to be integrated into the SNOMED-CT hierarchy so we can use them for decision support. A ministerial to advise that this must be done by the end of 2008 is what’s required here. SNOMED-CT may not be perfect but expecting clinicians to take responsibility for patients with no viable decision support is just not fair. When we have this we can look at scripts as they are just HL7 orders.
Next we need some money for all this, Standards Australia just need a one line budget, but perhaps we could add a”e” in front of every consultation/script that was messaged using AHML accredited HL7 and Medicare could pay the doctor/lab/pharmacy a premium and link all future rebate increases to this. That would certainly make it happen and you would only pay for performance. Its pointless paying vendors big $ to implement “X” – it has not worked. Reward some rubber on the road.
Ok, there it is, a 30 minute eHeath plan. All it does is put some onus on people to do what they should have already done and throw away the idea of expensive central systems and replace that with some good solid tested code in the trenches. That’s what we need.
Posted in DECISION SUPPORT, EHR, HL7, MESSAGING, SNOMED, STANDARDS, Uncategorized | No Comments »
February 17th, 2008 andrew
Our initial cross platform porting efforts were directed at OSX, but given its heritage we knew that linux would be close. It was quite easy in fact in the end and was running within a few hours. The yahoo user interface toolkit has been chosen for the UI and its certainly impressive. As capricorn is mainly a client for secure messaging it doesn’t not need much of a UI, but much could be done in a browser when combined with JSON and rest methods. Pictured is a form to send simple secure clinical messages directly from Firefox. Its useful to send quick notes but usually capricorn would integrate directly with other software.
Posted in HL7, LINUX, MESSAGING, SOAP, Uncategorized | No Comments »
February 2nd, 2008 andrew
This is the challenge of health IT at the primary care level. Most of the medical-objects destinations are Australian general practices which while computerised, are only just computerised and the stability and management of their systems is well… poor in general.
On top of this we run real time messaging using full PKI security for privacy and authentication. It is a challenge at times and we are continuing to learn, sometimes the hard way. The average life of a client install is < 12 months and often no one in the practice knows which computer is on, and frequently that computer disappears as part of an upgrade or reshuffle.
Managing this over thousands of sites is something we are trying to perfect. The first lesson we learnt was to get the content of the messages standards compliant. This has been vital as when we know that the payload meets Australian standards we can safely tweak the envelope to suite to recipients software, which often can’t handle compliant HL7. Compliant messages have allowed us to develop a large number of message modifiers or decorators which can be chained together in a huge number of ways to achieve a message that can be consumed. Even with a huge array of standard modifiers we still run into problems. Corporate systems are virtually impossible to tweak and the fact that they read to report date from the wrong place is something that you sometimes just have to accept. So the next element that makes it work is scriptable modifiers. We have a pascal script interface into our hl7 and xml parser that allow virtually any modification to be made. This is less routine and scalable that the canned modifiers, but is something you just need to have.
The other area that has proved vital is ongoing monitoring of server uptime. This allows us to detect clients with problems before someone notices that there have been no results for a while. Installation of new anti-virus software or changes to authentication parameters commonly cause this. We have also found sever instances of ISP misconfiguration of routers that block traffic between 2 clients.
Forwarding of errors in sending to support is another vital health monitor. Analysis of this is still not to the stage we would like it as many clients are disconnected after hours and weekends and the monitoring has to try and take this into account. Automating analysis of the problem is also an area we can improve on. We have started to identify patterns in the errors that mean something but the analysis is quite complex. We are considering using GELLO (Clinical decision support Language) to try and recognise these patterns in an automated way.
Double checking of new user and upgrade configuration is now much enhanced also. Users are good at finding shortcuts and at one stage we found support found a way to copy installation setups, followed by a few modifications. This made their setup much quicker but they did not change the GUID that is used for the URL on the proxy server…. The pleasing thing was that the PKI infrastructure prevented breaches of privacy as a wrong recipient could not decrypt the data but it had us scratching our heads. The sender and recipient both had each others keys and they were signed and correct but still we were getting errors in both authentication and decryption. In the end several clients were randomly getting each others messages so it would fail commonly, but if the correct client eventually received the message it would work. It was only when we noticed no hits in the client log despite a failed transmission that we tweaked to this one. The servers now avidly check all setups to prevent this happening.
The process has also made us aware of the difficulties in testing distributed systems. We have complex test setups but trying to simulate the real world of a distrubuted messaging system is also impossible. You have to try but the ongoing monitoring of connectivity and errors is something you have to have and the real world will always throw up surprises.
However we are excited about how well it works and how fast distributed systems are. The real world works quicker than any test setup and the message processing is done on different machines. In a test setup on a few machines you can almost follow it by looking at logging windows. In the real world it moves so fast that you have no hope of following it. Now to build something useful on top of it, after all that’s why we built it!
Posted in GELLO, HL7, MESSAGING, STANDARDS, Uncategorized | No Comments »