The Unresolved Dilemma of ZTE’s Equipment Breakdown
ZTE faced fierce criticism over its performance by China’s Telecom operators. ZTE is one of the two Chinese companies who signed the 1.6 billion Dollar Ethio Telecom second expansion project signed last year. ZTE had a substantial deal in 2006 from Ethio Telecom in a three-year sole supplier framework agreement financed on seller credit. The project involved 1.9 billion USD of investment by Ethio Telecom—1.5 billion USD for equipment and 0.4 billion USD for engineering construction.
In China, several large-scale breakdowns of ZTE’s 4G equipments threw a gloom over China Mobile’s full-swing execution of its 4G strategy.
The first incident happened in June during Shanxi Mobile’s 4G terminal trade show when a large number of ZTE base stations broke down. Then in July, at Guangzhou Evergrande Football Club’s half time, a lot of Guangdong Mobile’s users could not make calls or access the internet. In the same month, at the “Voice of China” concert, a huge network breakdown struck. All the network breakdowns happened where ZTE equipment were used. The incident that finally drew attention from China Mobile’s senior management happened between July 14 to 16 during China Mobile’s annual market conference. In light of previous accidents, Shanxi Mobile had communicated many times with ZTE to confirm the performance of the latter’s products two weeks ahead of the conference. To avoid network breakdown again, Shanxi Mobile conducted detailed coverage optimization, made plans for project expansion, and took precautions against network risks. According to ZTE’s standard equipment specifications (400 registration users per cell), the carrier even made temporary network transformations for redundancy backup. ZTE also formally promised Shaanxi Mobile by signing a written pledge that it would ensure no disruption at the conference from high data traffic. All necessary preparations were made. However, after 10 AM on the first day of the conference, the call completion rate at the Shanxi Hotel where the conference was being held began to drop. Network experience also started to deteriorate. In the end, the network broke down due to CPU overloading. Following the accident, ZTE replaced the boards, deployed two base stations for temporary expansion, and even adopted its extreme “super-cell split” solution. However, by July 16, the problem is still unresolved. ZTE could not identify the root cause of the problem in two days, which unveiled its weakcrisis management.
During the accident investigation, Shaanxi Mobile dispatched massive manpowerto locate the problem by the way of malfunction reoccurrence. They found that ZTE equipments were fully loaded when 120 terminals were online at the same time, far less from the 400 terminal accesses that it promised in the specifications, which triggered signaling storm and equipment restartas a result.
In the following mid-year network work conference, Vice President Liu of China Mobile made a special summary of this accident. He said: “The recent ZTE device breakdown is a very serious problem and we should learn from this excruciating experience. The reasons for the breakdown are as follows: first, architectural defects exist in ZTE’s hardware design, resulting in insufficient service processing capability. Second, flow control mechanisms for system software have potential risks due to poor design. There are no proper protection mechanisms against big traffic impact. Third, ZTE’s card chips and CPUs are outdated and of insufficient processing capability. ZTE sticks to single-core CPUs and outdated chips to save cost while most other vendors have employed octa-core CPUs. On the other hand, the latest accident also revealed ZTE’s weak capabilities and poor experience in troubleshooting and network assurance. All of us should learn from this experience. Currently, we should conduct inspection and troubleshooting of the entire network. We must set a deadline for ZTE to rectify all network faults. I hope our partners should also learn from this experience.” Why is there such a big gap between the specifications and actual capabilities of ZTE’s products? Didn’t ZTE test the actual capabilities of its products before it boasts to customers? How could ZTE be so careless and reckless in assuring such a big event as China Mobile’s corporate meeting? Our reporter interviewed many concerned persons to find out why. In fact, ZTE started to cut both employees and costs after a significant loss in 2012. It has also imposed harsh employee performance assessments standards. The accidents are an inevitable result of many factors. The problems were actually doomed even in the R&D phase. “There are no other ways. We have KPIs for cutting cost in ZTE, which cannot be neglected. Therefore, to save cost, you can do whatever within your power. You can reduce backup or use inferior chips to reduce the costs of cards. If you can’t find ways to cut cost, you’ll have to quit the job,” said Mr. Wang, senior engineer from ZTE’s Nanjing R&D center, “Due to large-scale employee layoff, many talented and experienced R&D personnel have left the company. The remaining people are mostly dawdlers. Currently, ZTE has a severe shortage of seasoned 4G experts”, said Mr. Wang. ZTE’s financial reports show that the company has lain off nearly 20 thousand employees.
The comprehensive cost cut campaign across ZTE gave rise to a series of problems, which affected the testing process. Mr. Chen, a technician responsible for testing before product delivery at ZTE’s Xi’an R&D center, told the reporter: “These kinds of problems and defects should have been easily detected during testing. However, the company is now experiencing a time of hardship and excruciating cost reduction. Everybody feels a sense of insecurity. Employees are constantly worrying about being given a ‘C’ in performance assessment, which is the worst and causes a pay drop. No one is really serious about testing, even if the testing personnel found the problems and reported them. The R&D department would simply make an excuse like this: they can only be designed this way if costs are to be cut.”
ZTE’s 4G wireless devices are designed in many ways to minimize cost. On the other hand, to win in the bidding process, ZTE’s propaganda of product specifications is greatly exaggerated. Experts point out that judging from the model and numbers of the chips used in their service processing cards, the actual capabilities of the base stations ZTE deployed for China Mobile may be less than half of the capabilities they claimed. Although China Mobile has very sophisticated and standard tender/bidding processes and strict network entry tests, ZTE’s false propaganda and empty and exaggerated promises were just too much to handle. China Mobile was really hurt this time. One of ZTE’s account managers spoke to our reporter: “We had to do things we are made to do. The company imposed harsh requirements on gross profits for 4G projects. The requirements account for 78% in our KPI. As a result, we had to do everything within our power to reduce cost even if it meant the product specifications could not meet carrier requirements. For example, if an additional channel board can be added to realize 1+1 backup or load balancing, each base station would have two channel boards to manage three cells, and the previous accidents could totally have been avoided. However, the competition is so fierce that we had to come up with whatever ways to reduce cost. We just can’t afford to lose the project bidding.” He continued in a self-mocking manner: “If China Mobile wants to expand for backup, we’ll then charge a high price for the products.” In fact, in the bidding contract signing phase, the network development director of one of China Mobile’s provincial branches found that ZTE boosted the contract value by 20% by increasing the quotation for cables and canceling the discount of some cards which was previously promised. The director issued urgent internal documents, notifying other branches of China Mobile to pay attention to ZTE’s product configuration and quotation to prevent it from any mishandling and fowl acts.
Mr. Zhou, a service engineer of ZTE’s emergency assurance team for China Mobile’s Market Meeting, complained: “Field personnel were cut significantly. Excellent network maintenance resources were also transferred to the headquarters to support key markets in big cities like Beijing, Shanghai, and Guangzhou. Here in Xi’an, the limited personnel can hardly cope with the job. What’s more, our resources cannot compete with other big provinces. We should have set up a special joint team of market, R&D, and technical support staff and rehearsed several times for a key event like China Mobile’s Market Meeting. Once an accident happens, all the blame is on us.”
As we dug deeper, we found that the recent accidents exposed many of ZTE’s problems. If these problems were not to be controlled, more similar accidents will happen as the number of China Mobile’s 4G subscribers keeps increasing. These accidents will be a great setback for China Mobile’s top-quality network and 4G strategies. ZTE needs to draw experience from these accidents and strives to be honest, professional, and customer-centric. It simply cannot afford another accident or disappointing China Mobile again.
The reporter also notices that on August 2 at Shenzhen’s Spring Cocoon Stadium, a large-scale network congestion and breakdown happened during Jackie Chen’s concert. This accident happened at the heart of ZTE’s home city is a sign that the company had not solved its problems.