When Shawn Harrs joined Red Lobster as EVP & CIO, one of his first discoveries wasn’t exactly comforting. The seafood chain had solid IT infrastructure spread across two data centers, but if disaster struck, it would take up to two weeks to get back online. For a company running 550 restaurants with highly perishable inventory, that’s not just an IT problem – it’s a potential business nightmare.
Harrs, who has held various CIO roles, follows a methodical playbook when joining a new organization. He carries out a 30-day assessment that covers everything from cybersecurity posture to technical debt – but it was Red Lobster’s disaster recovery capabilities (or lack thereof) that raised the biggest red flag. Harrs tells me:
I found that we were already using Pure Storage for some of our data storage – we had a couple hundred terabytes of Pure arrays. What I discovered was that we had a very robust hardware infrastructure running in a very robust data center facility in Ashburn, Virginia, in Equinix.
The problem wasn’t the quality of the infrastructure. Red Lobster had decent hardware in Ashburn and equivalent kit in a Dallas data center. But the Dallas facility, which was meant to operate as the failover site, was essentially a very expensive backup drive – storing day-old copies of data with nothing actually running there.
Building the business case
Identifying the problem was straightforward enough. However, getting budget to fix it required Harrs putting together a business case that would resonate with the CFO and board. He explains:
I began to work on two things: a business case (so that when we came up with a solution, the solution had to match the business case – you can’t have a billion-dollar solution for a million-dollar problem), and the technical solution itself.
Harrs built an actuarial model showing what would happen if the Ashburn data center went down. The numbers weren’t pretty: three to five days minimum of lost business, wasted inventory (remember, this is seafood we’re talking about), and potentially significant staff attrition. He adds:
Unless we pay our employees either directly out of cash or through an insurance claim, we will lose a lot of employees, so the rehiring could also extend that recovery time.
With restaurant industry turnover running at 100% annually and Red Lobster employing 40,000 people, even a week of uncertainty could trigger serious staffing problems.
He identified three critical systems that had to come back online quickly: ordering, hiring, and payment processing:
Even if we can’t do to-go service, even if we can’t take reservations, we need to have those three things.
The stakes of downtime
The urgency becomes clearer when you understand Red Lobster’s unique operational challenges. As Harrs points out, selling fresh seafood at casual dining prices is unusual. Most restaurants dealing with such perishable products usually charge premium prices to maintain margins. This constraint makes Red Lobster’s technology strategy – particularly any potential downtime – critical to its survival. On the overall strategy, Harrs explains:
The technology strategy is highly driven by business strategy – primarily 99% of it. We focus on the guest experience, growing traffic, service excellence and hospitality, and making the company a great place to work.
This pragmatic approach extends to build-versus-buy decisions. For anything touching Red Lobster’s core operations – supply chain, forecasting, inventory management, point of sale – the company builds its own systems:
Anything that focuses on these things – anything to do with what I call ‘farm to fork’ – we mostly and predominantly build ourselves.
The reasoning is scale and constraints that off-the-shelf solutions can’t handle. For everything else – marketing tech, analytics, labor scheduling – Red Lobster buys standard industry solutions.
Adding to the complexity is the delivery disruption that has transformed the restaurant industry. Red Lobster has seen a dramatic shift:
We have had anywhere between 20 and 25% of our business shift out of the four walls in the past seven years to the delivery experience.
This isn’t just about adding a new sales channel – it fundamentally changes operations. As Harrs notes:
Think of the cooking side of the restaurant like a manufacturing facility.
Just as Amazon can promise three-day delivery, Red Lobster needs to know it can deliver fresh food in 45 minutes without disrupting in-restaurant diners. Any extended downtime would cripple this delicate coordination.
Evaluating the options
When it came to fixing the disaster recovery problem, Harrs and his team looked at several approaches. They could abandon their data centers and move everything to the cloud. They could build out Dallas to run as a full mirror of Ashburn. Or they could find something in between. Harrs notes:
We spent a lot of time with a cloud provider who actually invested in doing the whole analysis of what our infrastructure needs are – storage, compute, networking, throughput, all of that architecture – to come up with a very robust solution architecture and quote to build it out and operate it.
But after visiting the Ashburn facility and talking to their equipment suppliers, Harrs realized the existing infrastructure was actually pretty good. The equipment might be a couple of generations old, but it was reliable and replacement parts were readily available.
The team calculated that running Dallas as a full mirror would get recovery time down to 24 hours – better than two weeks, but still not good enough:
A day is close. Why don’t we look at how to close the gap? How do we get from 24 hours to one hour?
That’s when he turned to Pure Storage:
We brought in their architects, and they said, ‘We need a faster way to keep things in sync, keep the data available, and just get this all back online within an hour.’
The Goldilocks solution
What Harrs calls the “Goldilocks solution” involved keeping the existing infrastructure but adding Pure Storage arrays to enable rapid replication and recovery. The technical implementation includes 300 terabytes of Evergreen One storage and two one-petabyte FlashArrays for disaster recovery. Harrs explains:
When you’re talking about disk replication, the ability for FlashArrays to replicate data and to bring infrastructure online is 24x faster than legacy hardware infrastructure and data on disk.
Just as important was the financial structure. While the total cost fit within the business case parameters, Red Lobster didn’t want a massive capital expenditure. Pure’s lease model spread the cost over three years. He notes:
Putting on the CFO’s hat – what does the CFO want to see? Cash out the door, not margin erosion. Even though we’re adding recurring OPEX, it’s still better from a total cost of ownership and cash out the door perspective than doing a big one-time capex project.
For context, Harrs says the full cloud option “was 6x the OPEX of what Pure is costing us.”
Learning from experience
Looking back, Harrs wishes he’d taken a different approach to evaluating solutions. He admits:
It took a long time to figure out what the different disaster recovery solutions are. What I wish I had thought of then was not to go down one pathway, but to go down multiple pathways in parallel.
This would have meant more work upfront – multiple discovery sessions, sharing infrastructure diagrams with different vendors – but could have accelerated the decision. He reflects:
I should have done solution-agnostic discovery.
His advice to others: keep an open mind about both problems and solutions:
When we approached Pure, we already kind of had a solution, but our solution needed to have a small gap bridged. Our gap was time, not a technology problem.
What’s next
Looking ahead, Harrs is particularly interested in data analytics capabilities. He says he has a big affinity for data analytics:
You shouldn’t have a favorite child, but my favorite child in the IT space, after keeping operations up and running and keeping the business secure, is data analytics.
He’s keeping an eye on Pure’s Enterprise Data Cloud announcements. Moving data warehouse infrastructure to modern platforms like Snowflake would be a major undertaking, he says, but Harrs is intrigued by anything that can accelerate time to value from data:
If I can get to value for my data faster, I’m all for it.