Open by design: Why the way the new Healthcare.gov was built matters [UPDATED]

UPDATE: The refresh of Healthcare.gov in June went well. On October 1st, when the marketplace for health insurance went live at the site.gov, millions of users flocked to the website and clicked “apply now.” For days, however, virtually none of them were able to create accounts, much less complete the rest of the process and enroll for insurance. By the end of the week, however, it was clear that the problems at Healthcare.gov were not just a function of high traffic but the result of the failure of software written by private contractors, with deeper issues that may extend beyond account creation into other areas of the site. On October 9th, as prospective enrollees continued to be frustrated by error-plagued websites around the country, I joined Washington Post TV to give a preliminary post-mortem on why the HealthCare.gov relaunch went so poorly.

The article that follows, which was extended and published at The Atlantic, describes the team and process that collaborated on launch of the new site in June, not the officials or contractors that created the botched enterprise software application that went live on October 1st. In the Atlantic, I cautioned that “…the site is just one component of the insurance exchanges. Others may not be ready by the October deadline.”  The part of the site I lauded continues to work well, although the Github repository for it was taken offline. The rest has …not. I’ve taken some heat in the articles’ comments and elsewhere online for being so positive, in light of recent events, but the reporting holds up: using Jekyll is working. Both versions of the story, however, should have included a clearer caveat that the software behind the website had yet to go live — and that reports that the government was behind on testing Healthcare.gov security suggested other issues might be present at launch. If readers were misled by either article, I apologize. –Alex


Healthcare.gov already occupies an unusual place in history, as the first website to be demonstrated by a sitting President of the United States. In October, it will take on an even more important historic role, guiding millions of Americans through the process of choosing health insurance.

How a website is built or designed may seem mundane to many people in 2013, but when the site in question is focused upon such a function, it matters. Yesterday, the United States Department of Health and Human Services (HHS) relaunched Healthcare.gov with a new look, feel and cutting edge underlying architecture that is beyond rare in federal government. The new site has been built in public for months, iteratively created by a team of designers and engineers using cutting edge open source technologies. This site is the rarest of birds: a next-generation website that happens to be a .gov.

healthcare-gov-homepage

“It’s fast, built in static HTML, completely scalable and secure,” said Bryan Sivak, chief technology officer of HHS, in an interview. “It’s basically setting up a Web server. That’s the beauty of it.”

The people building the new Healthcare.gov are unusual: instead of an obscure sub-contractor in a nameless office park in northern Virginia, a by a multidisciplinary team at HHS worked with Development Seed, a scrappy startup in a garage in the District of Columbia that made its mark in the DC tech scene deploying Drupal, an open source content management system that has become popular in the federal government over the past several years.

“This is our ultimate dogfooding experience,” said Eric Gundersen, the co-founder of Development Seed. “We’re going to build it and then buy insurance through it.”

“The work that they’re doing is amazing,” said Sivak, “like how they organize their sprints and code. It’s incredible what can happen when you give a team of talented developers and managers room to work and let them go.”

What makes this ambitious experiment in social coding unusual is that the larger political and health care policy context that they’re working within is more fraught with tension and scrutiny than any other arena in the federal government. The implementation and outcomes of the Patient Protection and Affordable Care Act — AKA “Obamacare” — will affect millions of people, from the premiums they pay to the incentives for the health care they receive.

“The goal is get people enrolled,” said Sivak. “A step to that goal is to build a health insurance marketplace. It is so much better to build it in a way that’s open, transparent and enables updates. This is better than a big block of proprietary code locked up in CMS.”

healthcare-gov-marketplace-graphic

The new Healthcare.gov will fill a yawning gap in the technology infrastructure deployed to support the mammoth law, providing a federal choice engine for the more than thirty different states that did not develop their own health insurance exchanges. The new website, however modern, is just one component of the healthcare insurance exchanges. Others may not be ready by the October deadline. According to a recent report from the Government Accountability Office, the Department of Health and Human Services’ Centers for Medicare & Medicaid Services (CMS) is behind in implementing key aspects of the law, from training workers to help people navigate the process to certifying plans that will sold on the exchanges to determining the eligibility of consumers for federal subsidies. HHS has expressed confidence to the GAO that exchanges will be open and functioning in every state on October 1.

On that day, Healthcare.gov will be the primary interface for Americans to learn about and shop for health insurance, as Dave Cole, a developer at Development Seed, wrote in a blog post this March. Cole, who served as a senior advisor to the United States chief information officer and deputy director of new media at the White House, was a key part of the team that moved WhiteHouse.gov to Drupal. As he explained, the code will be open in two important ways:

First, Bryan pledged, “everything we do will be published on GitHub,” meaning the entire code-base will be available for reuse. This is incredibly valuable because some states will set up their own state-based health insurance marketplaces. They can easily check out and build upon the work being done at the federal level. GitHub is the new standard for sharing and collaborating on all sorts of projects, from city geographic data and laws to home renovation projects and even wedding planning, as well as traditional software projects.

Moreover, all content will be available through a JSON API, for even simpler reusability. Other government or private sector websites will be able to use the API to embed content from healthcare.gov. As official content gets updated on healthcare.gov, the updates will reflect through the API on all other websites. The White House has taken the lead in defining clear best practices for web APIs.

Thinking differently about a .gov

According to Sivak, his team didn’t get directly involved in the new Healthcare.gov until November 2012. After that “we facilitated the right conversations around what to build and how to build it, emphasizing the consumer-facing aspects of it,” he said. “The other part was to figure out what the right infrastructure was going to be to build this thing.”

That decision is where this story gets interesting, if you’re interested in how government uses technology to deliver information to the people it serves. Government websites have not, historically, been sterling examples of design or usability. Unfortunately, in many cases, they’ve also been built at great expense, given the dependence of government agencies on contractors and systems integrators, and use technologies that are years behind the rest of the Web. Healthcare.gov could have gone in the same direction, but for the influence of its young chief technology officer, an “entrepreneur-in-residence” who had successfully navigated the bureaucracies of the District of Columbia and state of Maryland.

“Our first plan was to leverage Percussion, a commercial CMS that we’d been using for a long time,” said Sivak. “The problem I had with that plan was that it wasn’t going to be easy to update the code. The process was complicated. Simple changes to navigation were going to take a month.”

At that point, Sivak did what most people do in this new millennium when making a technology choice: he reached out to his social networks and went online.

“We started talking to people about a better way, including people who had just come off the Obama campaign,” he said. “I learned about the ground they had broken in the political space, from A/B testing to lightweight infrastructure, and started reading about where all that came from. We started thinking about Jekyll as a platform and using Prose.io.”

After Sivak and his team read about Development Seed’s work with Jekyll online, they contacted the startup directly. After a little convincing, Development Seed agreed to do one more big .gov project.

“A Presidential Innovation Fellow used same tech we’re using for several of their projects,” said Cole. “Bryan heard about it and talked to us. He asked where we would go. We wanted to be on Github. We knew there were performance and reliability benefits from building the stack on HTML.”

Jekyll, for those who are unfamiliar with Web development trends, is a way for developers to build a static website from dynamic components. Instead of running a traditional website with a relational database and server-side code, using Jekyll enables programmers to create content like they create code. The end result of this approach is a site that loads faster for users, a crucial performance issue, particularly on mobile devices.

“Instead of farms of application servers to handle a massive load, you’re basically slimming down to two,” said Sivak. “You’re just using HTML5, CSS, and Javascript, all being done in responsive design. The way it’s being built matters. You could, in theory, do the same with application servers and a CMS, but it would be much more complex. What we’re doing here is giving anyone with basic skills the ability to do basic changes on the fly. You don’t need expensive consultants.”

That adds up to cost savings. Sites that are heavily trafficked — as Healthcare.gov can reasonably be expected to be – normally have to use a caching layer to serve static content and add more server capacity as demand increases.

“When we worked with the World Bank, they chose a plan from Rackspace for 16 servers,” said Gundersen. “That added tens of thousands of dollars, with a huge hosting bill every month.”

HHS had similar strategic plans for the new site, at least at first.

“They were planning 32 servers, between staging, production and disaster recovery, with application servers for different environments,” said Cole. “You’re just talking about content. There just needs to be one server. We’re going to have 2, with one for backup. That’s a deduction of 30 servers.”

While Jekyll eliminates the need for a full-blown content management system on the backend of Healthcare.gov (and with it, related costs), the people managing the site still need to be able to update it. That’s where Prose.io comes in. Prose.io is an open source content editor created by Development Seed that gives non-programmers a clean user interface to update pages.

“If you create content and run Jekyll, it requires content editors to know code,” said Cole. “Prose is the next piece. You can run it on your on own servers or use a hosted version. It gives access to content in a CMS-like interface, basically adding a WYSIWYG skin, giving you a text editor in the browser.”

In addition to that standard “what you see is what you get” interface, familiar from WordPress or Microsoft Word, Prose.io offers a couple of bells and whistles, like mobile editing.

“You can basically preview live,” said Cole. “You usually don’t get a full in-browser preview. The difference is that you have that with no backend CMS. It’s just a directory and text files, with a Web interface that exposes it. There are no servers, no infrastructure, and no monthly costs. All you need is a free Web app and Github. If you don’t want to use that, use Git and Github Enterprise.” Update: Cole wrote more about launching Healthcare.gov on the DevelopmentSeed blog on Tuesday.

Putting open source to work

Performance and content management aside, there’s a deeper importance to how Healthcare.gov is being built that will remain relevant for years to come, perhaps even setting a new standard for federal government as a whole: updates to the code repository on Github can be adopted for every health insurance exchange using the infrastructure. (The only difference between different state sites is a skin with the state logo.)

“We have been working in the .gov space for a while,” said Gundersen. “Government people want to make the right decisions. What’s nice about what Bryan is doing is that he’s trying to make sure that everyone can learn from what HHS is doing, in real-time. From a process standpoint, what Bryan is doing is going to change how tech is built. FCC is watching the repository on Github. When agencies can collaborate around code, what will happen? The amount of money we have the opportunity to save agencies is huge.”

Collaboration and cascading updates aren’t an extra, in this context: they’re mission-critical. Sivak said that he expects the new site to be improved iteratively over time, in response to how people are actually using it. He’s a fan of the agile development methodology that has become core to startup development everywhere, including using analytics tools to track usage and design accordingly.

“We’re going to be collecting all kinds of data,” said Sivak. “We will be using tools like Optimizely to do A/B and multivariate testing, seeing what works on the fly and adapting from there. We’re trying to treat this like a consumer website. The goal of this is to get people enrolled in health care coverage and get insurance. It’s not simple. It’s a relatively complex process. We need to provide a lot of information to help people make decisions. The more this site can act in a consumer-friendly fashion, surfacing information, helping people in simple ways, tracking how people are using it and where they’re getting stuck, the more we can improve.”

Using Jekyll and Prose.io to build the new Healthcare.gov is only the latest chapter in government IT’s quiet open source evolution. Across the federal government, judicious adoption of open source is slowly but surely leading to leaner, more interoperable systems.

“The thing that Git is all about is social coding,” said Sivak, “leveraging the community to help build projects in a better way. It’s the embodiment of the open source movement, in many ways: it allows for truly democratic coding, sharing, modifications and updates in a nice interface that a lot of people use.”

Open by design

Sivak has high aspirations, hoping that publishing the code for Healthcare.gov will lead to a different kind of citizen engagement.

“I have this idea that when we release this code, there may be people out there who will help us to make improvements, maybe fork the repository, and suggest changes we can choose to add,” he said. “Instead of just internal consultants who help build this, we will suddenly have legions of developers.”

Not everything is innovative in the new Healthcare.gov, as Nick Judd reported at TechPresident in March: the procurement process behind the new site is complicated and the policy and administrative processes that undergird it aren’t finished yet, by any account.

The end result, however, is a small startup in a garage rebuilding one of the most important federal websites of the 21st century in a decidedly 21st century way: cheaper, faster and scalable, using open source tools and open standards.

“Open by design, open by default,” said Sivak. “That’s what we’re doing. It just makes a lot of sense. If you think about what should happen after this year, all of the states that didn’t implement their systems, would it make sense for them to have code to use as their own? Or add to it? Think about the amount of money and effort that would save.”

That’s a huge win for the American people. While the vast majority of visitors to Healthcare.gov this fall will never know or perhaps care about how the site was built or served, the delivery of better service at lowered cost to taxpayers is an outcome that matters to all.

About these ads

12 thoughts on “Open by design: Why the way the new Healthcare.gov was built matters [UPDATED]

  1. Pingback: 10 Ingredients Missing in Federal Government Innovation | Reach The Public

  2. Pingback: The openwashing of Healthcare.gov

  3. How’s about an update on what’s causing all the problems on healthcare.gov this first week? To me, a lot of what was stated and promised in the above has not come to fruition – even remotely.

    • Hi Steve. I’ve been covering the rollout over social media — Twitter, Facebook, Google+ — but haven’t written a detailed followup yet, for a variety of reasons, both personal and professional. (I have other research commitments I have to meet.) I went on Washington Post TV today to break down what appears to have gone wrong, why and what’s being done. I hope the conversation is useful, along with this brief recap.

    • It’s not “my database,” Frank. There are a series of databases that the backend of Healthcare.gov has to query to authenticate a user, calculate eligibility of subsidies and filter for relevant insurance plans. The way the front-end of the site is built, the “data” — graphics and text — have all been pushed out into Akamai’s CDN and are pulled in on demand as a user navigates the site. The insurance marketplace was built by different contractors using different methods with a different outcome.

  4. As a long time S/W architect (I registered domain #43 on the planet ;) ), I think it would be helpful to disambiguate the front and back-end systems of this portal.

    The front end system looks fantastic. I *love* the creative switch to static-generation of the site. It follows a principle long used for huge conferences (remember COMDEX?) — precalculate rather than use dynamic technology, to serve a predictably high load.

    The front end fails as an information system however, because it doesn’t deliver the information needed by inquirers. I should be able to see every detail of every insurance plan available in every state to every constituency. And I should be able to see the subsidy levels available as well. IOW, without saying anything about myself, I should be able to click through to all the information needed to understand what options exist. A specific example: to what extent do any of the new standard plans provide coverage for cancer treatment, organ transplants, and/or physical therapy? I still have not found an answer to these basic questions.

    The above is just about the front end: a friendly non-secure system that’s more or less an online catalog and entry portal.

    But it is the BACK end that’s the heart and soul of ACA. Here we find user data validation, verification, eligibility, security, qualification, product offering and acceptance, etc etc. Here is where gov’t/business policy meets the real world. And here, AFAIK, the entire system was badly botched. There are endless stories that indicate a lack of architecture for security, proper data validation/verification, confidentiality, etc etc. These have nothing to do with load management. And then there’s the botched transaction management.

    I can’t say that the front end is perfect and the back end horrible. But the front end appears well-architected and easily fixed, while the backend sounds like a mess. :(

  5. I use to do this for a living too and I can tell you there are far too many JavaScript calls in the front end, so much so that applicants have been advised to empty their cache. The correct way to do all this would have been to use a proper database manager like Adobe ColdFusion and a proper front-end database like IBM’s DB2 to capture the user-entered data. That data could then have been translated to the health providers schemas.

  6. Perhaps we’d have been better if had they gone with tried and true technology platforms, methodologies and architectures rather than attempting to jump on the bloody edge of things… And while they were at it, wouldn’t it have reduced risk if they had used a major IT services vendor, someone with decades of experience and tens of thousands of professional developers?

    • Wayne, healthcare.gov worked perfectly all summer. The front end of the relaunched site, described in this article, continues to work fine, based as it is on open standards, robust open source technology and a modern approach to Web development and deployment at scale. The backend, which went live on October 1, was built with “tried and true technology platforms, methodologies and architectures,” by a “major IT services vendor” — CGI Global and QSSI, amongst others — and had many developers working on it.

      The notion that “tens of thousands of professional developers” is the way to build software suggests some unfamiliarity with development. Wayne, the troubles behind the version of Healthcare.gov that went live in October 1 appear to be a result of poor management decisions, political fears, insularity and yes, bad coding.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s