Authored By:  Douglas “Mitch” Mitchell

Contributions By:  Jackie Duty

June 10, 2003

Summary

Moving (expanding/remodeling) data centers, server farms, software development labs and test labs with minimum disruption to end-users requires precision planning and execution skills.  It is one of the most challenging single event projects an IT department can take on.  These relocations include all aspects of the IT environment including facility construction, power, HVAC, networking, security, servers, storage, databases, applications, Internet services and data center operations.  Poorly planned relocations have resulted in the demise of many project managers.

To survive you must know what you have to relocate, how it works and interfaces with other components, be able to design the new data center layout, perform the many layers of planning necessary, interface with all the external points of contact, execute the move and then deal with any resulting problems.

These data center moves are a rare event in the life of an IT department, requiring many special skills, across multiple disciplines.  It is not the kind of project where move planning is done in one’s spare time and the move is done over a weekend.  Moving a data center using this approach is very high risk. The reality is that planning and moving a data center is a very complex project with many potential points of failure.  Since these are rare events and many disciplines are involved in the planning and execution of a Data Center Relocation, the internal staffs simply do not have actual experience with or exposure to this type of work.  Ideally, you need to engage someone with the project management skills, technical expertise, as well as experience in the individual IT disciplines to be successful in data center relocations.

This paper provides tips and tricks on data center relocations and expansions.  It is not intended to be a complete blue print on how to relocate data centers.  Some concepts may seem like mother hood and apple pie but it is surprising how many times they are not followed resulting in problems and schedule delays.  For more detail or help in relocating your data center please contact the author of this paper.

Relocation Approaches

The relocation approach selected depends on business strategies, requirements, cost constraints and acceptable business risks.  These business directions are critical in selecting the most appropriate relocation approach.  Once the approach has been defined the team can proceed with developing the relocation approach, analyze the current environment and develop a plan for the relocation.  Following are the basic approaches used for these relocations.

• Lift & Move:  This approach uses existing hardware components.  Equipment is de-installed, transported to the new site and reinstalling in the new data center.  Although this typically requires more down time it is less costly since it uses existing equipment.  It also can be less risky if the systems can be moved with out the need for system and data restores.  This requires that the new data center is located close enough to allow time for the transporting with in the required time frame.

• Backup and Restore:  This approach acquires and pre-installs equipment before the actual move.  Unit testing is completed for all pre-installed components in preparation for the data and application loads.  Backups are transported or transferred to the new center and restored on the new equipment after application shutdown.  This provides the ideal opportunity for upgrades and expansion in a controlled environment with minimal down time.  Although this approach requires the least amount of down time, it is a higher cost approach and inherently has more risk as compared to the Lift & Move approach.

• Consolidation:  Data Center Consolidation relocates applications and data into a new data center on existing systems.  These systems may be running other applications or applications may be consolidated into existing applications.  This approach can significantly reduce ongoing costs but requires more time for testing, business and operational process change and has a higher risk of business and operational disruptions.

• Move & Restore:  This approach uses a combination of Lift & Move and Backup & Restore.  Time critical systems that require limited down time or are already in need of system upgrades would be relocated using the Backup & Restore process.  Those less critical systems would use the Lift & Move approach.  This approach helps balance costs, business processing requirements and risks.

Note:  If at all possible, pre-install and test all network components well ahead of the move.

Safety First

People’s safety should be the number one priority.  Data Centers have many safety hazards including heavy equipment, talk racks, high voltage electricity, holes in the floor and many sharp edges. Nothing can ruin relocation faster then someone injured or worse.

• Appoint safety manager to ensure all safety precautions are followed.

• Ensure that everyone knows where the first aid kits are located, the location of all emergency exits and who is CPR trained. At least one person with CPR training should be available at all times.

• Mark open floor tiles and holes with safety cones and close as soon as possible.  While the move is in progress, the safety manager or their delegate should be watching this at all times.  The term break a leg should not be used any where near a relocation project.

• Only certified electricians should work with electrical panels and high voltage electrical connections.

Analysis

As with any project, the first phase is to develop an analysis of why you are doing it, what you are dealing with and what the expectations are of all parties impacted by the relocation.  

• Understand why the Data Center is being relocated, the relocation objective, the business requirements, cost constraints and the acceptable risks.  These are key components in determining the relocation approach and developing a relocation plan.

• Inventory all hardware and software; also, label all equipment with a relocation data base number. After these are in place, implement a change tracking process to ensure that the relocation inventory is kept current.  The same goes for all software and applications.  You must understand the scope of components being relocated at all times.

• All external accesses to the data center need to be identified including users, vendors and customers.  You also need to know all application components and interfaces between applications.  This latter is especially important if you are planning on bringing the applications up in stages rather then all at one time.  It is surprising how many applications are not totally understood in this regard.  If documented properly and maintained this results in extremely valuable application documentation.

• Data center relocation offers a good opportunity to expand or upgrade your current systems.  Perform a capacity and performance analysis to assess the need to upgrade and where appropriate install replacement equipment in the new data center, before the scheduled move.  This reduces the risk, complexity and outage timeframe required for the final move.

Planning

Planning is a key data center relocation element.  It includes all aspects of the relocation including communication, organization, task plans and testing.  Without a detailed plan the relocation has little chance of succeeding.  

• Have a structured plan.  Write down the obvious then test to make sure the obvious works.

• Start your planning using the Data Center Disaster Recovery plan.  Although you will not be able to use it in total it is a good place to start.  In many ways data center relocation is a controlled disaster recovery.

• A good rule of thumb is to have 3 backup sources for all data and applications.  Plus, if at all possible, keep a version at the starting site until the destination site is up and running.  Even though backup media is much more reliable now, I have seen several versions of the same backup fail during restores.  Also test all backup and restore processes before the actual move to ensure that they perform as planned.

• Although a mid-level task plan may be sufficient leading up to the actual relocation, for the move period a detail task plan is required.  Typically this requires detail tasks identified and scheduled by the minute.  All participating parties need to know their tasks and have copies of their tasks including required start and finish times.

• Keep everyone informed.  When the team members, support staff and users know what is going on and when, they will be much more supportive.

• Involve users and suppliers.  Let them know what is going on and solicit there support in planning, preparation and testing.  Get their buy-in by showing them how it is going to work and that their areas will be secured.

• Perform dry runs to ensure that every detail is taken core of.  You do not want anything to happen by accident.

• Do as much up-front work as possible.  Pre-install everything you can to make up for short windows.

• Shut down applications and data bases before final backups are made.  Lock users out of the systems to ensure that they do not modify data during the backups.

• Run benchmarks on the existing systems before shutdown.  Use these to verify that the new systems are up and running at least at the same level as when they were shutdown.

• Carefully understand your systems maintenance agreements. Some agreements are null and void if your data center is not moved by a certified professional.  

• Ensure that systems are isolated and off of the network during the move process.  This is required to accommodate application and user acceptance testing before opening the systems up for production work.

Teams

The relocation team is critical to the success of the relocation.  They need to be comprised of the right resources, to be well informed and work together to ensure the success of the relocation.

• Data Center Relocation teams need to include people from most parts of the organization impacted by the relocation.  These teams should include all affected organizations in the company including business users, application support, data base administrators, technical support, help desk and operations.  

• The team should also include vendors and suppliers for hardware, data bases, application systems, transportation, tape libraries, networks and others as appropriate to ensure that the proper expertise and facilities are available for the relocation.  

• A well feed team is a productive team.  Don’t forget to include food services as part of your team.  The actual move frequently stretches out over 24 hours a day for several days; ensure that the people have food available at all times.  

Transportation

The actual move of the equipment and data can make or break data center relocation.  If the equipment and or data does not reach the new site on schedule and in one piece the relocation will be a failure.  All necessary precautions need to be covered so that the equipment and backups arrive in tact and on schedule.

• The actual transporting of the equipment must be smooth and with minimum jarring.  Servers, disks and other equipment moved using inappropriate transportation frequently require components replaced and cards reseated resulting in more time and risk.  Make sure that the vehicles used to transport the equipment have air rides and that the equipment is prepared and packed properly.

• Trucks should have hydraulic gates that have a capacity to lift the heaviest pieces of equipment.  This is especially important if there are no loading docks or if they are of a size that does not mate with the truck heights.

• When transporting data backups long distances, send separate versions via different routes to ensure that at least one copy arrives on schedule.  Many things can happen during transportation that can delay an already tight schedule.

• Select a carrier carefully.  A carrier will pass on their insurance based upon the valuation of your equipment.  Also, make sure that they are experienced in moving electronic equipment and have the right equipment.

Facility

There are many facility aspects associated with a Data Center Relocation including construction, remodeling, layout, power, security, environmental controls and fire suppression.  All of these are required to be completed and tested before the actual move.  In general they need to be done by professionals who are experienced in working with data centers.  But, their work still needs to be overseen by the Data Center Relocation team as what they do and their timing directly affects the schedule and success of the data center relocation.

• Total power and air requirements are dictated by the vendor specifications for all equipment being installed in the data center.  There should be enough additional capacity to support all planned additions and expansion capabilities for future growth.

• Verify that building permits are in place as this can shut down construction for extended periods of time.  This seems like a simple fact but it has happened.

• Ensure that all floors to ceiling heights meet code including areas under ducting where equipment will be installed.

• Verify that equipment ceiling clearances meets code including sprinkler head clearances.

• Equipment layout is a critical area and needs to be done with care.  Requirements for this include; local codes, vendor specifications, operator efficiencies and expansion capabilities.  It needs to be done in a detailed way and reviewed by all appropriate parties.  Operational and support processes also need to be taken into consideration when laying out the data center.  Once the data center is up and running it is very difficult and risky to re-arrange equipment.

Testing

Follow the standard testing processes of unit testing, system testing and acceptance testing.

• Unit testing should include testing all component items including networks, hardware and operating systems before loading applications and data.  It also includes testing individual applications as they are loaded.

• System testing includes testing of applications and data as they are loaded including their access and interfaces to other applications and systems.  System testing should also include initial performance testing to confirm that systems are performing at the same level or greater then they were when shutdown.  Benchmark data comparisons should be done before shutdown of the old systems.

• Acceptance testing is done before production turnover and uses benchmark data generated before shutdown of the old systems.  Acceptance testing should be performed by users, applications, DBA’s, outside vendors and selected customers.  It also should include performance verification based on previously identified performance criteria.

• Where ever possible install and test all network components before the move.  Make sure that all users, vendors and customers can access the new network before the actual move.

• Test your backup and recovery tools and processes before the move to ensure that they work properly.  This includes bringing the systems up and running using the restored copies.  Also, if restoring to different hardware or operating systems, verify that the restores work in these new environments.

Communications

Effective communications are essential to a successful Data Center relocation effort.  The Communications Plan needs to address requirements for the internal move team and all affected stake holders including:

• Business Management/Leaders
• Business Users
• Business Partners
• IT Management
• Vendors

• Prepare and present the move approach, schedule, risks contingency plans and key contacts to management and all affected business groups.  It is critical that everyone knows what to expect and when it will happen.

• Establish Command Centers at both the old center and the new center.  These Command Centers serve as a central point for communications, status and problem reporting for the internal move team.  If possible, engage a dedicated Help Desk person to manage the entry and status tracking of problems using problem management tracking tools.  The Command Centers serve as the focal point for reporting status of critical path/milestone activities.

• Establish a toll free number for Business users, Management and Relocation Team members to get the latest move status.  Provide recorded status updates at least every two hours.

• Setup a hotline for key move personnel and IT management.

• Prepare a key contact and escalation sheet with telephone, cell phone, home and pager numbers for all management and team members participating in the move.

• Provide relocation team members with communication tools (cell phones, pages, etc.) so that everyone can stay in touch at all times.

• Provide a Relocation Manual for team members that includes schedule of events, meal service timeframes, contact lists and other pertinent information.

Relocating a data center is a major undertaking.  But, circumstance may require that it must be done.  With enough analysis, planning, a good contingency plan and a strong team, you can relocate your data center with minimum disruption of service and avoid unexpected down time and data loss.