Web Support During COVID-19

It was becoming a regular family event. In fact, there was even a drinking game made up for it, “Wine with DeWine.” I would imagine the Governor of the great state of Ohio never thought he would be a trigger for a drinking game. Each day at 2 p.m. get your favorite drink and be ready to knock a few back every time Governor DeWine says “Flatten the curve,” “Let’s not make this political” or “Wash your hands.” 

 

According to Centro, web traffic overall has seen a 22 percent increase since the start of the pandemic in the U.S. Broadcast news has become a trusted source of information and the preferred way for people to receive news.
 

Because of this regularly scheduled press conference and the heightened state of attention around COVID-19, one of our clients had a challenge on their hands. As one of the digital news outlets streaming Governor DeWine and Ohio Department of Health director Dr. Amy Acton daily, their servers on AWS + Drupal were being overloaded with concurrent connections.
 

On the technical side of this, we were ahead of this event and ready to support the technical team at ideastream. As one of our digital support services clients, the teams had been working together for months prior on setup optimizations and also where some configs needed attention. So when the weight of the audiences tuning into the press conferences hit, our team jumped in to support the traffic and shore up processes as needed. 
 

Early on, the team would get notifications from Site24x7 that the site was down due to high load (a 504 error in the browser). Our first task was to correct an issue within the auto-scale config. OpsWorks, the AWS service responsible for this automation, could no longer reach the Git repo when it needed to spin up one of the load-based EC2 instances.
 

The day after it was corrected, we still encountered a high number of 504s and concluded the lag time from when high-volume was present to when the new load-based instance was started and ready for action was already too late. Most of the high-volume traffic had dropped off since the 504 connections caused folks to stop trying, which in turn caused the load-based instances to shut-down because the resource load was back within normal range.
 

From there we adjusted the thresholds on load-based instances to react quicker and stay active for longer. A few days went by with fewer issues and then we had a mega-surge in traffic after the lockdown (Excluding liquor and alcohol locations of course. We have a game to play, remember?) and concluded the load-based servers weren’t capable of doing what we needed. Instead the team switched over to time-based instances that could be scheduled to start and stop when needed for the press conferences.
 

Ultimately, we ended up creating five new time-based instances that have twice the amount of vCPU and RAM. They spin up at 1 p.m. and shut down at 4 p.m. There are six 24/7 instances that run, you guessed it, 24/7.
 

After this, the site saw a 700% increase in the number of active connections it could serve at once. The support team did this within a short time table and with urgency as the situation changed daily.

 

And now we’re all thinking the same thing, this could and will likely happen again. Hopefully, this never happens again, but scaling to accommodate increased traffic suddenly will. And this situation certainly highlights the scary facts of digital infrastructures. Ask yourself, do we or do we not have a strategy to handle our critical and essential digital infrastructures if everything changes in a day?


 

Gone are the days of hosting critical sales and marketing properties on cut-rate products or architectures. Let’s get real. We’re not saying to not make sure you get what you pay for, but realize the importance of well-thought out and maintained hosting architectures and how valuable reliable partners are to every brand, business and news outlet. 
 

Some would argue that every brand is a news outlet these days. Or at least that is likely what your content strategists have been saying for many years now. They aren’t wrong either.
 

So, make sure to discuss and not dismiss the power of a partner that can architect and support your digital properties for where you are now and for the “what if’s” of the future.
 

And remember, wash your hands.
 

Now, take a drink.