Today, Jeff Zients offered an update on our efforts to improve HealthCare.gov. In Jeff’s own words –
“This was a week where we made progress, and also ran into some roadblocks that slowed us down. The focus is on working through our punch list, one by one. Our bottom line continues to be that we are on a path to improve HealthCare.gov each week and to meet our target by the end of November.”
Here is an update on where we are at the end of this week:
- QSSI, the general contractor named last week, is now firmly in place. They have brought in their top talent — a deep team with expertise across a full range of technology and program management.
- There is a rigorous management structure that is focused on prioritizing the punch list and real time decision making, 24/7.
- With QSSI, we have established dedicated teams to fix and monitor both software and infrastructure issues. There are four core teams:
- Application and software – this team addresses glitches so the site is faster and smoother for users;
- Infrastructure and hardware – this team is focused on adding capacity and redundancy to minimize disruptions;
- Security – this team is continuously working to ensure rigorous protections of the system and its data;
- Monitoring and troubleshooting – this team is focused on analyzing system performance and spotting problems early.
With these teams in place, we have the right management structure and accountability to make the necessary progress. Unfortunately, we hit a frustrating roadblock this week with hardware problems at the Verizon data center that hosts the site. Last Sunday, we experienced a failure in a networking component in the data center and a related incident happened mid-week. We’ve fixed the failed hardware, and we will be making further hardware upgrades to the system to improve stability and add redundancy this weekend. We will provide an update once the upgrade is complete, but make no mistake, the hardware failure was a setback and was very frustrating.
That said, we have made progress this week in crossing items off the punch list, fixing bugs and improving site performance. The site works better today for users than it did last week and we’re putting tools in place to allow us to track our progress against metrics.
The most important near term metric is response time. Response time is how fast the system responds to user requests. For the first few weeks, we estimate that users were waiting on average 8 seconds for pages across the site to load. This is an absolutely unacceptable amount of time. In fact, this metric is measured in milliseconds, not seconds. Yesterday, the average response time on the site appeared to be less than 1000 milliseconds. This is a more than 80 percent improvement and the team is taking aggressive steps to bring response time down further.
Overall, the site is getting better as we work through the punch list. Just last night we made a batch of fixes that improve the application process, increase system speed and efficiency and begin to address insurer concerns around 834 notices. And we have another round of patches and fixes planned for this weekend to cross more items off the punch list.
It’s important to remember that every day consumers are coming to the site, shopping and signing up. And while we all want the pace of progress to be as fast as possible, It is inevitable that there will be unexpected roadblocks. But we have the right team in place and a relentless focus on speed and execution.
We will measure our progress, hold teams accountable, and drive the upgrades that improve the consumer experience. The site will get better week by week, and by the end of November, the site will work smoothly for the vast majority of users.