A food and beverage distribution company had aggressive plans to expand its business by 40% over four years. In order to achieve this goal, the company recognized that it needed to modernize and position its IT to support this growth.
This wasn’t an easy task as IT historically was viewed as a cost center. Executives had not invested in infrastructure or solutions for more than 12 years. The result was a very dated environment with applications and infrastructure no longer eligible for maintenance or support.
To help move the project in the right direction, the company brought me in as an IT leader to define a strategy and road map for the next three years. In this article, I’ll explain the steps we took to create the strategy and road map, along with some lessons learned along the way.
Defining and Selling the IT Strategy and Road Map
This was the first time this company put in place an IT strategy and road map, so it was difficult to get corporate buy-in. It was important to include the C-suite and business leaders in the planning process to make sure we had a good understanding of what was important and how to move forward.
The first initiative we identified was to implement disaster recovery (DR) and business continuity planning (BCP). The company was also working on an SAP upgrade from SAP ECC 5.0 to SAP ECC 6.0 SP7 on an IBM iSeries. Although there were daily tape backups, there was nothing to restore the backup to. The company had no plan in place to continue to operate if it lost the main data center and its SAP applications. This was a huge risk.
We had to work to explain the risk and the need for the disaster recovery and business continuity plan. Part of the process was explaining the financial impact that such a risk can have on the company. Eventually, we did get buy-in and began to implement the plan.
From Building to Implementing the Disaster Recovery and Business Continuity Plan
The IT initiative would first address the disaster recovery plan and then the business continuity plan.
The disaster recovery portion was more technical in nature. We worked with our SAP partner to implement a backup site for our SAP instances. This involved determining the size and capacity of the remote instance and making sure it would have the capacity to support the remote offices if we needed to fail over to it.
We also needed to increase the bandwidth from our primary data center to our partner’s data center. This was necessary because we had a replication agent running on the iSeries that sent SAP updates to the remote instance. The non-SAP applications critical to running the business were virtualized. We needed confirmation that our virtualized environment was being replicated to our backup data center and that the backups would work if we needed to use them.
The business continuity portion of the plan required more strategy. The VP of operations was involved to help engage and align the business units and corporate functions, including HR and finance, among others. The IT department and the business units worked together to define the procedures and steps to execute if a catastrophic event were to occur. We then communicated these procedures to the business, which included defining key roles, responsibilities, and alternate facilities where we would run during a disaster.
With all the pieces in place, the last phase focused on testing. We needed to know we were prepared to execute. This required several tests where we stopped the system to replicate an outage and failed over to our backup instance. We validated our systems by running reports and going online to check things that were defined in our test plans, such as orders and financials. Once we were successful in validating our results, we implemented the solution.
Putting the Business Continuity Plan to the Test in Real Time
With a dated infrastructure environment, one item among many that needed to be updated was the backup battery system for the data center. The system was updated, and everything seemed to operate as normal. The next day, however, my network administrators reported a fire in the data center.
We needed to shut things down but could not get to the electrical panel in the back of the room. We decided to close the doors and evacuate. We immediately implemented the business continuity plan we had just finalized and dispatched our sales teams to an alternate location. The next step was to implement the disaster recovery plan. With the help of our SAP partner, we were able to execute the plan right away. We executed the disaster recovery and business continuity plan within 20 minutes.
After assessing the damage, we found the data center was unrecoverable. We lost the servers, switches, iSeries, telephones, battery backup, and the inventory of laptops and parts. We assembled the IT team and started to bring up the non-SAP applications from our backup facility. The IT team validated the connectivity between applications, SAP, and the various databases. The sales teams were already at the remote office location resuming order entry and processing orders. The only issue we experienced was lost access to a non-critical application due to a recent password change in the active directory. We continued to monitor all the systems to make sure everything was running properly. Later that evening, the team confirmed that all orders and deliveries were processed on our normal schedule.
Resuming Business as Normal Post-Disaster
Never in a million years would I ever have expected to encounter a disaster that would require us to implement our contingency plan. Because of the work we did to plan, the company continued to operate on our backup facilities for 40 days. We needed to restore our network infrastructure, servers, and the iSeries in our main data center.
The company did roughly $1 million in orders per day. We were able to save the company a significant amount of money by having our Disaster Recovery and Business Continuity Plan in place. Something that is probably more important than money is the company’s reputation with its customers. We were able to continue to take care of our customers as if nothing had ever happened. Quite an accomplishment, with all things considered—all thanks to having a plan in place.
Learn more on how to respond to uncertainty in “CIOs Discuss How to Navigate the Uncertainty of COVID-19.” We welcome all ASUG members to submit their ideas for blogs they would like to write.