Case study. Relocating Servers to New Data Center, Frankfurt: Minimal Costs

When moving to a large and well-known data center, you normally do not expect any problems with the IT infrastructure or force majeure. What you do expect is that the equipment will be safe and sound, all current requests will be processed promptly and emergencies will be resolved quickly with no losses.

So did the character of this story and our future customer, the head of the IT department of a large online store (let’s call him Alexander). He believed that he did not risk anything by renting a rack in one of the largest German data centers (DC). Unfortunately, not all service providers in Germany are customer-oriented, and Alexander got into a very unpleasant situation putting at risk the operation of the entire e-commerce company, which could suffer huge financial and reputational losses simply because of some service aspects. But first things first.

In this story, we will tell you about the difficulties to expect when concluding a rental agreement with a DC, the reasons why the customer decided to find a new service provider and eventually move from one data center to another with no Plan B, the migration process details and the money saved.

The background: what was wrong with the data center

The servers of the IT system and the network equipment of the online store were located in one of the largest and most popular German DCs in Frankfurt am Main. It would seem that both the level and the organisation of service here were impeccable. But in fact they were not.

Problems faced

The customer with only one rack in this huge center was categorised as a small customer to be guided only at the beginning of cooperation.

The operating procedures in the DC require that a certain sequence of actions must be taken when a problem occurs:

• go to a special portal with a very complex structure,
• create a request with a detailed description of the problem,
• wait for a response from the support specialist.

It could take three days to get a response. It’s not about immediacy.

The ticketing system not only had a confusing structure but also — like the rest of the DC tools — was made in a language that the customer did not know well. Accordingly, he had major communication problems because of the language barrier preventing him from describing the problem, while the support service did not seek to understand.

At first, Alexander had an account manager who communicated with him in his native language, so he received the minimum necessary assistance. However, he still had to go to the general customer portal in case of any technical issues. After his manager was replaced by an English-speaking specialist, the customer became almost isolated: he could not explain the problems clearly, his requests were not perceived correctly; it remained unclear what exactly to do even when the support specialists answered the questions.

Since the current maintenance of the data center was not carried out and the customer did not have the opportunity to come to Frankfurt to carry out the maintenance of the rented equipment, the issues with the equipment accumulated and multiplied.

It is quite natural that Alexander was nervous — he was responsible for the entire IT infrastructure of the operating company and an accident at the DC could lead to a business catastrophe.

The equipment does not require much attention when it works correctly, but when there are failures in critical structures and the support service can not be reached, the situation becomes critical and may cause damage to the business of tens of thousands of euros. Obviously, if the online store site becomes unavailable one day and customers can make no purchases, the company will lose not only money but also its 15+ years reputation.

Requests did not give the desired result. Software issues were not solved, the customer’s hardware was also not properly serviced. After days of lengthy communication and attempts to overcome bureaucratic obstacles, the condition of the equipment became critical and began to pose a threat to the business operations.

The disk subsystem partly required replacement, the hard drives failed — the first one, then the second with the third on its way. The situation became an emergency. Unable to get help from the support service of his data center, the customer started looking for a suitable service provider to solve the issues locally. In the end, he even agreed to move to another data center of this service provider.

How the customer chose its service provider

• Location in Germany, Frankfurt.
• Direct contact with support specialists and coordinator.
• Communication in the native language.
• Clear means of communication and tools.
• Online feedback.
• Prompt response to requests.
• Guaranteed level of data availability and security.

In addition, the customer wanted the provider to take care of both maintenance and contractual issues.

However, it was necessary to solve the problem with the coming equipment failure before the migration. The customer decided to contact us based on the recommendation of his partners.

Negotiation stages

The first part of negotiations was held in September when Alexander turned to us for technical assistance in replacing their disks. At the first meeting, he shared his difficulties and worries, migration plans and asked a huge number of questions. In particular, he needed a specialist from IBM to deal with software errors. In addition, he was interested in redeeming the rented equipment and involving us in the interaction with service companies.

In turn, we shared our plan on how to solve his problem and described the services we offer to the customers of our DC. We also touched on the migration process. It took us several weeks to find information, reach out to IBM, negotiate with their representatives and prepare detailed answers to all important questions for the customer. In fact, the preparation for migration began immediately after the first meeting.

Alexander was satisfied with the received information and expressed a desire to accept our offer and migrate to our DC as a customer at the second meeting. A final decision on cooperation was soon made.

The attention we paid to the interests of the customer’s company played a significant role in choosing us as a service provider.

The reasons for migration were major ones. In addition, the business owners wanted to redeem the equipment completely and make it the company’s asset. It is also important that the IBM equipment used by the company was unique and not the latest one. Some parts of it were already not serviced by the official manufacturer.

All this background meant a rather complex set of tasks for us:

1. First of all, it was necessary to solve the burning issues of maintenance: to restore the performance of the hardware, replace the failed disks and manage the existing problems with the software.
2. Then draw up a diagram and carry out the migration: transport all the server and network equipment from the previous data center and connect it in the same way at the new place. It was vital to do this in the shortest possible time not to disrupt the operations of the company.
3. We also had to find service companies ready to maintain the customer’s unique equipment and sign contracts with them.
4. The last was the issue of redeeming the equipment.

Initially, we concluded a framework service contract and took over all the works performed in the previous data center. Since the equipment condition was critical, including its hardware part, we replaced the RAID disks within the storage before starting the migration. The customer’s redundant RAID-6 provided high fault tolerance. But two disks were faulty and one more failure would become dangerous for the entire system even in such a reliable structure.

Then we started preliminary works on migration. We provided the customer with security certificates and rules of the DC. We set tasks, made a schedule, planned downtime. Main migration activities started in November. At the same time, we agreed on certain points of the agreement updated since the beginning of the year.

Due to the complexities of the migration process and the working pattern of the company, we scheduled the migration procedure to take place during a quiet business period as the customer requested. We chose one of the nights in early January during the winter holidays with fewer orders and minimal losses caused by forced downtime.

Olga Boujanova

When the preparations were completed, we met with Alexander once again for the final approval of the migration plan.

Migrating to Another Provider’s Data Center in Frankfurt: Step-by-Step

We took over the entire migration process. This is the whole story of the migration.

Stage one: preparations at the previous center

Analysis and inventory control

We requested the customer to provide a list of equipment and wiring diagrams. The first step was to perform a technical audit of the network infrastructure to understand how to transport everything at the network level. Then we analysed the condition of the equipment.

Our specialists visited the data center with the customer’s rack. We examined, compared, checked, recorded and photographed everything. The equipment was specific, so it was important to prevent any errors.

Equipment

There was one rack with six shelves with the following:

• 3 storages (each storage with an extension),
• 5 servers (4 big servers and one small server), 5 units,
• network equipment (1 router, 2 Fibre Channel switches and 2 Ethernet switches).

A migration plan was drawn up on the basis of the information and diagrams received.

Stage two: preparations at our data center

To ensure that everything works correctly, we made preparations at our data center too. This was a difficult process involving careful treatment. After the migration, it was not enough to simply plug the equipment into the socket, it was necessary to establish connections between different devices, connect the outputs to external channels — the Internet, another operator, etc. That is why we had to prepare our network equipment and connections before the migration.

Stage three: migration

The customer’s IT infrastructure works as a whole. In order not to interrupt the company’s operations for a long time, the migration had to be carried out as quickly as possible. Therefore, a lot of time was given to preliminary works, and we also had to wait for the company to be ready.

The migration itself took 6 hours from the moment of disconnection of the equipment in the previous data center to the connection in our center. It was scheduled to take place at the beginning of January because there are fewer site visitors and the load on the IT infrastructure decreases during the winter holidays.

The project coordinator kept in touch with the customer throughout the migration process and kept him informed of all the details. We explained all the nuances of migration including the following steps:

1. Marking the equipment, making photographs, drawing up a wiring diagram.
2. Turning off the servers in the previous data center.
3. Packaging the equipment for transportation.
4. Transportation (with loading and unloading).
5. Assembling and connecting to the rack at the new location.
6. Test run.

Thus, we disassembled the entire system, carefully transported the devices and interconnected all their units in the same sequence in the new place.

Moving with no Plan B: how to migrate between data center infrastructures, bugs and difficulties

There were no special technical difficulties or bugs during the launch process, but only a couple of routine issues when one of the links did not work at first. Technicians switched it, and everything was ok in 2 minutes.

The only difficulty was more of psychological nature. Alexander was very nervous, and our project coordinator had to constantly support him, convince him that everything was good and going according to plan. Apparently, after the troubles in the previous DC, he simply did not believe that it is possible to successfully migrate without any problems. Because of his experience, he was pressing on us, but migration requires pinpoint precision. Our technicians did not panic and did their job competently and accurately as always.

It was the first time when Alexander had to deal with migration, so his concern was easy to understand. He was 100% responsible for the IT infrastructure of a large commercial enterprise with an online store, orders, a huge number of buyers, payment acceptance, shipments, etc. If the servers became unavailable after the migration, the direct and indirect financial losses of the business would amount to millions and the image damage would be horrific. If something went wrong, there would be no Plan B. No steps back. But we have to pay tribute to Alexander because he trusted us despite his concerns and was satisfied with the result in the end.

Migration team

The equipment owner was in another country, the work was carried out remotely and all responsibility for migration rested with us. Therefore, a fairly large team of specialists participated in the migration. In addition to our certified engineers and an experienced technician who connected the wires, an engineer from IBM (the equipment manufacturer) was involved in the migration in case of any connection difficulties.

Hours to perform the whole range of project works

1.   Negotiations with the customer conducted by the project coordinator (including their flight, trip to the facility and business meeting) — about 11 hours in total.

2.   Analysis of the current situation of the customer:

• project manager — 10 hours (correspondence, data collection, calls with the customer),
• technician — 3 days (visit to the data center with the equipment, marking, wiring diagram preparation, other preparatory work — about 30 working hours in total),
• engineer — 20 hours (visit to the previous DC, inspection of the equipment condition).

3.   Preparation of the equipment migration project:

• project coordinator — 20 hours,
• technician — 15 hours,
• engineer — 35 hours.

4.   The migration procedure took 6 hours (from 11 p.m. to 5 a.m.). This includes all migration stages: from disconnecting the equipment in the previous data center to connecting and configuring it in the new one.

One technician, two of our engineers and one IBM engineer were directly involved in the migration.

Post-migration

Our cooperation scope was not limited to assistance in the migration and maintenance at our data center. We have solved another important problem for the company — they wanted to redeem the equipment.

Renting Server in DC with Subsequent Buyout of IBM Hardware

When a company owns its equipment, this reduces costs as well as increases its assets and authorised capital. Some companies prefer to use rented resources for 3-4 years and then make updates and modifications. Others, like our customer, redeem the equipment and record its units as their asset. The key factor in this case might be the company’s policies or security rules providing for certain infrastructure. This also might be as a way to reduce the risks and costs of operation, or simply the conventionalism of the owners, the desire to keep the tangibles. Nevertheless, it was crucial for the company to redeem the equipment and record its units as their asset.

When we first met with the customer, he already had a valid contract with the previous data center for the rental of used equipment and other services. In addition to the subscription fee, he paid a reasonable amount every month to fully redeem the equipment and become its owner at the end of the contract.

But the language barrier made communication difficult, and Alexander was unable to directly negotiate with the data center. We knew how to redeem equipment in Germany and proposed to take over this issue. It is a short phrase but hours of work: before we took out the equipment, we had to agree on this with the previous data center and guarantee that its redeem.

Negotiations were held, all contracts with service organisations were reconcluded, a corresponding contract was made with the data center and our company became a party thereto instead of the customer. We bought the equipment first and then the customer bought it from us.

As a result, we fulfilled all contractual obligations, made payments, documented the equipment and transferred it to the customer at its remaining value. Thus, it was recorded in the assets of the customer.

The customer wanted not only to resolve the technical issues but also to redeem the equipment at its remaining value. It was critically important for them to fulfil the obligations under the initial equipment rental agreement concluded with the previous data center, to become the owner of the equipment and include its units in the company’s records. But this was a huge problem due to difficulties in communication.

We, in our turn, had to deal with the previous contracts on network connection, colocation services — cabinet (rack) provision, power supply, external communication channel connection, the Internet and IP addresses, since these contracts had their own validity periods.

Expansion of the IT infrastructure

The migration to the new data center provided the company with a new development perspective. They had only one rack in the previous data center but got an opportunity to scale their IT infrastructure after the migration to our center. Now they have one more rack with several more units of rented equipment. In particular:

• additional storage with extension (one more extension was added a bit later),
• iBM’s newer blade system: Flex System X440,
• Fujitsu Storage Eternus DX200,
• several servers.

Then they switched to another VPN configured on Cisco ASA 5515-X. They also made additional infrastructure changes requiring more servers and storage. We connected them to one more Internet service provider. As a result, they now have the Internet from us and another ISP.

In fact, the company’s resources have doubled. But they have not stopped with this and carried on with the development.

Equipment is purchased and all technical issues are solved through us.

Servicing and maintenance

Equipment features

The company in question is a supporter of one equipment manufacturer. Their equipment is reliable and stable but has unique features and, in addition, is not new. So the technical part of the services is of great importance to the customer.

We signed the service agreement so everything related to maintaining the devices in working condition is now our concern. It is a quest to maintain such specific equipment (when some parts of it are already not serviced by the manufacturer). We had to contact certified professionals. We cooperated with the original IBM vendor at first and then we had to go to a successor who continues to support obsolete models not serviced by the manufacturer.

Storage disks

The disks in the data storage system are another story. Since they store all the information, they are critical for the entire IT system. The customer disks were a kind of legacy:

• IBM System Storage 600Gb 3,5″ 15000RPM LFF SAS SED 49Y1983;
• IBM System Storage 4Tb 3,5″ 7200 RPM 6G NL-SAS, T10 PI capable 00Y5147.

That is why it was necessary to monitor their condition and always have spare ones in case of an accident. But looking for spare ones is a tough task in case of outdated models. We had to change our supply channels. At some point, we even thought of withdrawing the storage support because of these disks. After all, we have found a company that produces not analogues but modern disks that fit this old equipment.

Since the equipment was outdated, we had to change our supply channels for spare disks several times during the contract period.

In general, we are ready to solve even the most non-standard customer problems, such as failures of disks, RAID controllers, main boards, control boards, optical boards, switches, other issues of servicing both new and outdated devices.

Why customers change service providers

Here we need to focus on very important aspects that many companies do not take into account when choosing a DC for migration.

In large data centers, in addition to poor customer focus, the service part creates problems for customers and leads to unnecessary costs.

Traditionally, large DCs provide 2 remote hands models:

1. Non-urgent requests are processed by support specialists within 3 days (EUR 100+).
2. Urgent requests are processed during the day (these cost 3x).

In any case, communication goes through the ticketing system. This is what causes difficulties. If, for example, a disk fails and needs to be replaced quickly, the system administrator should describe everything in detail in the request: what exactly failed, what needs to be done, what slot to insert, etc. for the technician to perform the replacement correctly with no questions. It is not an easy task to make a correct request with insufficient knowledge of the language.

According to the regulations of large DCs, technicians can not respond to the customer through external communication sources (phone, e-mail, messengers), all communication should be done through the ticketing system. This is necessary to record the working time. But communication through such portals is delayed, while time is the of essence in the event of an accident, when it is necessary to urgently restore the operation of servers, network devices, IT infrastructure, etc.

Maintenance is a critical task and the qualifications of technicians may vary — you never know who is on duty today. And each time it will be different people. It is a toss-up: if you are lucky today, you get a responsible specialist to address your request, but tomorrow you may not be that lucky and the problem will remain unresolved due to human factors.

Olga Boujanova

In addition, the on-duty technician may not know the background of the equipment and, if the request does not describe the problem in detail enough to understand the failure, they have to clarify information, find documents, etc. So it may take more time to process the request, the maintenance costs may respectively increase and, in case of an accident, the downtime period of the IT system may increase, which is critical for business.

All in all, the customer can contact the support service, but the result is unpredictable. There were situations when technical specialists dismantled the equipment they decided to be extra and then customer sites, corporate portals, online stores crashed. At the same time, the data center is practically not responsible for such incidents.

Thus, system administrators have unnecessary difficulties on the one hand and are technically not protected on the other hand. It is not surprising that customers find it more convenient and useful to get all in one: the data center not only rents resources but also fully provides for the maintenance of the system. And this is most often called a service provider. In this case, the customer does not need to write long requests in a foreign language, does not worry that something will unexpectedly fail and lead to a catastrophe.

We have organized our service in a unique way based on customer interests.

How we do it: one stop service + interactive multilingual support

If the equipment is supported by us, we fully take care of its maintenance.

1. We are responsible for the availability/operation of the devices and network.
2. We know everything about the equipment that we maintain. We have at least two sources of information. For each customer, internal technical documentation is compiled with a description of the rack, equipment, serial numbers, diagram, connection procedure, etc. All this info is also included in the contract.
3. We have simple and easy to use tools for working with customer requests + interactive communication in German, English and Spanish.
4. Each customer has their own assigned engineer who directly leads their process and with whom they can communicate directly. If the assigned engineer is on vacation or sick, another specialist can always replace them and solve the issue thoroughly thanks to our detailed documentation.
5. Each customer is assigned a personal account manager. This manager is their one stop contact to solve both organisational and any technical issues.

There is another bonus for companies that conclude a service contract with us: we have spare parts in stock at our site to provide for routine replacements of any failed parts. More details are below.

Equipment monitoring and routine maintenance (how we ensure 99.99% availability period)

In our data center, constant monitoring of the customer equipment is carried out. If we see, for example, that the disk in one of the servers in the rack starts to flash, then we immediately handle the situation and automatically perform a replacement. We always have a disk swap reserve.

The process is well tuned and in most cases the customers do not even notice these works. We inform them about the repairs made when everything is completed. If this replacement requires that the equipment is turned off, the period for the forced downtime is preliminarily agreed.

The most critical component of the data storage system is disks. And we serve many companies that have mature equipment. So disk replacement is a standard service under our service contract. We do it regardless of whether the equipment is rented or owned by the customer, is serviced by the manufacturer or not. Our task is to ensure guaranteed availability, and we are fully responsible for the operability of the nodes and elements of the customer IT system.

Migration at no additional cost: budget and expenses

• We offered the customer more favourable conditions than the previous DC. As a result, the total annual costs of the capacity rental decreased by EUR 50,000 (from EUR 173,000 to EUR 123,000).
• Migration of the equipment was carried out overnight and with no additional costs for the customer. We provided for all packaging materials, car rental, dismantling, loading, transportation, unloading of the equipment, cables, proper connection required for the migration at our own expense.
• The equipment used in the previous data center was redeemed at its remaining value and included in the company’s records. The company’s costs reduced accordingly.

The customer got free funds and justification for the management to further invest in the development of their IT infrastructure. The company was able to invest the saved funds in the expansion.

Steady work — steady prices

It is important to note that we signed a contract for 3 years with a further annual extension. Prices have been fixed and are not subject to change with the exception of electricity costs as dictated by electricity suppliers and the state. With this, the customer has used and will use our services at a fixed cost throughout the entire period of cooperation.

If you are planning a migration, feel free contact us, we will be happy to help.

Our social responsibility

• Green technologies. Environmental protection is an important aspect of any business in Germany. Like many German enterprises, we do care about the environment and make our best efforts to use safe technological processes that do not pollute nature and do not harm the human environment. For example, we use EkoBasis as a source of electricity. It is a product that allows you to get 100% clean energy from renewable sources. This is confirmed by the relevant certificate of TÜV Süd (Association of Technical Supervision), an expert organisation engaged in examination, testing and certification in Germany.
• Charity. We are also engaged in charity work by partnering with the non-profit charity association MainLichtblick eV. Together we help kids with health problems by fulfilling their cherished childhood dreams.
• Innovative heating system. The heat generated by the data center equipment is not wasted, it goes to heat an entire housing block. This centralised heat supply method can reduce CO2 emissions by about 100,000 tons per year and thereby benefit the environment.

SET UP A CALL

Article author

Olga Boujanova

Consultant on server hardware and data center organization

Case study: How to choose a certified Tier III data center for server migration in Europe?

READ MORE