JOURNAL OF TELECOMMUNICATIONS, VOLUME 11, ISSUE 2, DECEMBER 2011 1

Lab Interoperability and Carrier Acceptance Testing: making the case for growing importance and emerging strategic role Dominic B. Boamah and Erasmus Addae Abstract—Wireless ecosystems continue to get complex with different network elements and connected devices manufactured by different companies. Navigating through such complex ecosystems is, and will continue to pose challenges to business professionals, and organizations around the World that want to seize the opportunities provided by both legacy and state-of-the-art wireless technologies to engage their employees regardless of their location, or individuals who want to stay connected to their jobs, friends, and families whilst on the road. This paper discusses these challenges and the growing importance and strategic role of lab interoperability and carrier acceptance testing in dealing with these challenges by vendors and carriers carries around the the world. carriers around world. Index Terms— Carrier Acceptance Testing, GSM, Interoperability Testing, LTE, Third Generation Partner Project, UMTS,

——————————  ——————————

1 INTRODUCTION connected has become a catch phrase and devices Staying like the Iphones, Black Berries, Nokias, Samsungs, HTCs, etc. that keep us connected to our families, friends, jobs, etc. are seen and used everywhere, and across generations. These devices have become an integral part of our lives. To those in the wireless business, the word “connected” in the above phrase is not just a word in a catch phrase. It means much more especially to the network operators and device manufacturers. This is because the “connected” part of the phrase happens through them. The networks operators are sometimes referred to as service providers or carriers, and the device manufacturers are sometimes referred to as device vendors. These will be used interchangeably in this paper. To these groups, the question is not how connected their customers are, it is about how well and happily connected they are. Quality of service which includes “collective services performance with respect to user expectations and network configuration” [1] is everything. That is what keeps us well and happily connected, and the level of the quality of service defines the position of these vendors and network operators in the industry. However, providing the expected level of quality of service continues to be a challenge to the industry due to the continuous evolution of wireless technologies into complex systems, and the difficulty in integrating the new complex systems into existing legacy systems. Today’s typical wireless network is made up of many different pieces from many different vendors [1]. This similarly applies to the connected devices that are used on this network for various services. These connected ————————————————

D. Boamah.is with the Wireless Measurement Solutions Unit, Anritsu Company, Richardson, TX 75081. E. Addae. is with the Department of Information Systems, Collin College,Frisco, TX 75035.

devices are based on the same specifications that are used to develop the network elements. Therefore, the technology evolution mentioned above also applies to connected devices and not limited to only the network elements. Thus, central to the challenges faced by carriers carries and vendors are: 1. Ensuring that the many different elements and connected devices that are part of the ecosystem do interoperate. 2. Ensuring that connected devices in the ecosystem do function and perform as expected by a service provider in accordance with the settings/configurations of the elements that constitute the service provider’s network Dealing with these challenges requires comprehensive testing of the interfaces among the different network elements by vendors, and carriers carries need to conduct comprehensive testing of devices against their network settings or configurations to ensure they work before deploying them onto their commercial networks. These two types of testing are very expensive and time consuming when they are done with real network. Thus, in addition to making significant impact on cost of production, these two activities can cause very significant delays in time-tomarket. Consequently, performing these two types of testing in the lab, using simulators of elements in the ecosystem are gaining growing importance and beginning to play a strategic role in the wireless telecommunications business. The remaining of the paper is organized as follows: Section two, discusses the evolved functionality of cell phones and the different ways network services are obtained by customers. Section three, discusses why Interoperability Testing (IOT) and Carrier Acceptance Testing (CAT) are needed and their role in ensuring device and

© 2011 JOT www.journaloftelecommunications.co.uk

2

network elements certification, and carrier acceptance of connected devices. Section four, attempts to provide further discussions and definition of IOT. Section five discusses lab IOT and CAT, and their impact on quality of service and cost of production, and time-to-market. Section six, presents a summary of the discussions and some conclusions.

2 MORE THAN JUST MAKING CALLS, AND THERE ARE A LOT OF THEM IN THE MARKET! Today, those in the wireless industry no longer refer to the Iphones, Black Berries, Nokias, Samsungs, HTCs, etc, as cell phones. Terms like user equipment, device, and terminal are used and that is mainly because they are not just the cell phones they used to be a few years ago. They have evolved into complex devices that keep us constantly connected to our families, friends, jobs, etc. regardless of our location in addition to making the usual phone calls they were originally designed to do. These changes in the devices have been driven by evolution in wireless networks from the basic GSM EDGE Radio Access Network (GERAN) through UMTS Terrestrial Radio Access Network (UTRAN) to Long Term Evolution (LTE), which is arguably emerging as the next generation wireless technology. Along with this evolution have been an increase in complexity and number of network nodes, expansion to cover large geographical area, support for various services and as results of these, a change in deployment environments [3]. With these changes and complexity, ensuring that devices and the networks that support them do function not just per Third Generation Partner Project (3gpp) industry standards and specifications, but also with a particular network settings or configuration, continues to challenge the industry. This challenge will only increase as device vendors and network operators turn their attention to LTE and have to integrate new systems and applications into their existing legacy network and development environments. Device manufacturers know and understand that their success in the market depends on how well their devices perform on various networks. In places like the United States of America, the success of a device manufacturer very much depends on how well its devices function on the networks for which the devices are manufactured. Thus, device vendors and carriers both have higher stakes in how well devices function on a carrier’s network. In other parts of the world such as Europe, Africa, and Asia where devices are not necessarily manufactured for any specific carrier, the pressure seems to be more on the carriers than the device vendors. Looking at how flooded the market is with devices and the quality of some of the devices especially in places like Africa, it could be argued

that all that matters to some of the vendors is meeting the minimum requirements for 3gpp certification and other possible regulatory compliance requirements. In these parts of the world, subscribers can just go to the market, get any device they are interested in, and go to a network operator to sign up for services. The moment that contract is signed; the service provider becomes responsible for everything including the functionality of the device, as far as the customer is concerned. But, the fact is that some of the issues the customer may encounter in using the device may have nothing to do with the network and everything to do with how the device functions, which is the responsibility of the device vendor and not the service provider. The above may be the case for some network operators in these parts of the world, and the vendors serving these markets may not necessarily feel obligated to ensure their devices function well with any particular application or the major networks in the markets they operate. However, due to the dynamic and competitive nature of the wireless business, it should be in the interest of all vendors to accept that whether their devices are made for specific networks or not, their survival and future ultimately depend on how well the network operators in the various markets they serve perform. Therefore, the proper functioning of devices beyond what is required for industry certifications to functioning with some of the most popular applications in the market, and major networks in the areas they serve should be as important to vendors, just as it is for the network operators. It is imperative that vendors work together with network operators to find ways to ensure devices function well before they are released into the market. In places like the United States before any device is deployed on any network, network operators verify the functionality of the device to ensure that it does not just comply with 3gpp and other industry requirements, but it also works with the specific network configuration and applications on the network for which it is made. This is what is known as Carrier Acceptance Testing (CAT), and it is done by the network operators in collaboration with the vendors.

3 ENSURING CERTIFICATION AND CARRIER ACCEPTANCE OF DEVICES To ensure that devices do comply with 3gpp specifications, and both devices and services are quickly validated and acceptable to service providers, companies in the test and measurement section of the wireless industry are working on various techniques to ensure a fast and effective validation to avoid delays in time-to-market, and also to lower the cost of device certification, and development in general. In fact, even vendors who are making investments and spending time to make quality devices derive the definition of their “quality” mainly from 3gpp com-

© 2011 JOT www.journaloftelecommunications.co.uk

3

pliance so those devices have to be validated for services on a carrier’s network before it is deployed. This is because the fact that a device has been certified for 3gpp compliance does not validate it for a particular network, which may have unique customers with some unique needs. It is due to the interoperability challenges discussed above and the need to ensure that devices operate on a particular network that IOT and CAT continue to gain attention, and being seen as a strategic activity in the design and deployment of new wireless devices [2]. By conducting CAT, vendors and network operators can ensure that a device that has passed 3gpp and other industry and regulatory compliance requirements testing is also ready to be deployed onto a particular network. Consequently, in addition to the usual 3gpp and other industry and regulatory compliance requirements for devices, CAT is gaining a lot of attention across the industry and gradually emerging as a strategic weapon in the wireless industry’s competitive warfare along side IOT.

4 THE MEANING OF INTEROPERABILITY TESTING (IOT) As the term implies, IOT is the type of testing that is carried out to verify that the interfaces between two or more different applications or systems function in accordance with relevant and acceptable standards. According to ETSI, “the purpose of interoperability testing is to prove that end-to-end functionality between (at least) two communicating systems is as required by the standard(s) on which those systems are based” [3]. “Operators consider IOT as an essential activity in verifying that new equipment can deliver the expected functionality per specification” [4]. Although wireless devices and networks are all based on the same 3gpp standards and specifications, they are manufactured by different companies, with different manufacturer profiles so they may not necessarily be compatible. Therefore, it can’t be assumed that wireless devices and networks will interoperate because they are based on the same standards. It is important to test them to ensure that they, in fact, do interoperate and the type of testing that is conducted on such different implementations to check for their interoperability is called Interoperability testing [4]. Additionally, IOT helps vendors and carriers to verify that in addition to the interoperability among system components and connected devices, the components and connected devices do provide the services they are expected to provide after integration [1].

5 WHEN ARE IOT AND CAT NECESSARY? Interoperability Testing is a device certification requirement that should be met before the device is certified by either Global Certification Forum (GCF) or PCS-1900 Type Certification Review Board (PTCRB) or both for deployment. GCF and PTCRB are the bodies that govern the development of wireless devices and ensure each de-

vice is 3gpp compliant before it is released into the market. CAT, as already stated is carried out by network operators to ensure that devices do work with their network configuration before they are deployed onto the network. Since IOT is a device certification requirement, it is done as part of the certification process. By conducting CAT before a device is released or deployed onto an operator’s network, the device vendor and the service provider are able to work together on both sides to ensure the device is not just 3gpp compliant but it works well on the operator’s network. CAT helps the two parties to fish out endto-end interoperability issues and address them appropriately before the device is deployed onto a carrier’s network. This means that CAT should be a shared responsibility between device vendors and network operators. This helps the operators to avoid a number of initial deployment hiccups that many companies face when they first deploy new devices. To the device vendors, this helps them to maximize the sale of their devices, which is what every vendor looks up to. This is especially so with new technologies like LTE and the possible issues involving handing over from LTE to UMTS and vice versa [6].

6 LAB OIT IOT AND CAT 6.1 Growing importance and emerging strategic role As indicated, IOT in the wireless telecommunication world is testing the interface between wireless devices and a particular network infrastructure and the interfaces among the different elements that make up a particular network. Thus, the external interfaces between connected devices and network elements are tested in addition to the interface among the different network elements. For the device vendors, each model released has to pass IOT on a real network before the product passes GFC and PTCRB certification, and per recommendations by the network vendor IOT (NVIOT) forum, wireless products should interoperate with at least 3 different vendor equipments [6] for it to be certified. There are a few live networks labs for IOT that scattered around the world. However, just by looking at the number of devices pouring into the market, and the frequency of new releases, it is obvious that these few labs are fully booked around the clock, and they are not cheap [6]. It gets even more expensive if a product fails IOT in these labs and has to be tested again because besides the high cost of retesting, failing means the product has to be on the tail end of the testing queue the next time it shows up for retesting, and that can lead to unreasonable delays in time-to-market. Finding ways to avoid this and to reduce the overall cost of production are part of the many challenges facing device manufacturers. As both vendors and carriers strive to deploy LTE equipments and devices with the possible inter-working issues with existing systems and applications, these challenges will only increase, as already discussed. Also, although devices have to interoperate with network elements from at least three different network vendors for and

4

certification as mentioned above, network operators configure their networks based on the needs of their customers, and geographic conditions may also impact how operators set up their network. Therefore, the IOT challenge does not only rest with device vendors. As mentioned earlier, network operators have to ensure devices interoperate with their specific configurations before mass deployment of any type of device onto their network. This has, and continues to become even more critical for large operators with different classes of subscribers who require different types of services especially with the ever increasing number of value added services and the quest by customers for high data rates. To ensure that devices do interoperate with their network, carriers who can afford may have small networks that are set aside for CAT. These are usually called testbeds or testbench, and they are configured the exact same way the commercial networks are, and they are used for CAT on devices before they are deployed onto the commercial networks. Additionally, network operators try to stay ahead of the game by trying to understand the possible scenarios that may happen on their commercial networks by simulating them, and identify possible fixes before the problem even appears on the field [4]. These sorts of simulations can’t be done on testbeds even if an operator can afford to build one. This is because they require frequent changes in configurations to match the different scenarios. To deal with these challenges, both device vendors and network operators rely on system simulators to help them with their network IOT and CAT. The device vendors rely on these system simulators to simulate the network configurations against which their devices will be tested for certification, which as already discussed, is a requirement to be met before the device can be released into the market. This initial lab testing helps the vendors to adequately prepare for the actual network IOT. It has been estimated that between 70% - 75% of device IOT compliance requirements are met by lab testing using system simulators [4]. Obviously, this reduces the chances of a device failing certification and therefore increasing the chances of the device being released on schedule. For the network operators, even if they have a testbed, the system simulators allow them to simulate as many scenarios as possible on their network in the lab to ensure devices do function with their unique settings before they are deployed to the commercial network. Additionally, system simulators allow network operators to verify that the many new features on devices function as expected, and the many data related value added services they provide do function with different devices, and with the expected data rate.

6.2 Assembling the right simulators and tools As discussed above, the elements that constitute wireless networks and their connected devices are developed based on common standards and specifications developed and maintained by 3gpp. It is against these same standards that these elements and connected devices are

tested by PTCRB and GCF before they are certified for commercial release. Although carriers may set up their networks in different ways to provide different services to their customers, as discussed earlier, these different settings are based on the same 3gpp standards and specifications. Therefore, these standards serve as the foundation for device acceptance testing by carriers carries. Consequently, simulators for wireless telecommunication networks have to be well and carefully designed based on these same 3gpp standards and specifications. Generic tools like Iperf, FileZila, Wireshark, etc. can be used to monitor traffic and data rates on the network, but the protocol simulators have to be 3gpp compliant. These 3gpp compliant simulators are commercially available through companies like Anritsu, Rhode & Schwarts, Anite, and Spirent that operate in the testing and measurement sector, which is a small but very well established and competitive sector of the industry. Reviews of these companies and what they provide as far as features and services in their simulators is outside the scope of this paper. However, it is our recommendation that a careful investigation and a well designed cost-benefit analysis must be done before a decision is made to purchase any simulator for lab testing. Some simulators may support GERAN, UTRAN, and LTE whilst others may only support GREAN and UTRAN, and just like all other things, the different vendors of these tools have their strengths and weaknesses. Thus, the best simulators for lab testing are the ones that meet the needs of the vendor or carrier making the purchase. The cost of these simulators can easily run into thousands of dollars or even a few millions and that can be a significant investment for any organization.

7

CONCLUSION

To device vendors and network operators, ensuring that customers are not just connected but happily and well connected have and continues to be critical to winning a competitive advantage in the wireless market place. With LTE fast emerging as the next generation wireless technology, there are many challenges such as integration of LTE elements and applications into existing GERAN and UTRAN elements and applications. Thus, more critical than ever, there is the need for vendors to be able to test their products throughout the development cycle and to perform some form of testing in the lab to ensure their devices are in good standing before they are submitted for PTCRB and GFC certification. Although there are certified labs to help device vendors prepare for their device certification, these labs are almost always fully booked with long queues. Besides, finding and fixing bugs during the IOT certification of devices can be very expensive because each bug that is found has to be fixed and retested. In addition to the amount of time that will be lost to the fixing and retesting of a bug that is found during testing, the testing opportunity will be lost, and the device has to join the tail of the queue after the bug is fixed for retesting. This can have a significant im-

5

pact on time-to-market, cost, and ultimate product success. Also, due to the increase and continuous changes in services provided by network operated or demanded by customers from their network operators especially data related services, and issues associated with bandwidth and data rates, it has become equally important for network operators to be able to simulate the types of configurations they may need on their network in order to support or provide certain services. Although some operators have set up testbeds to help them to ensure that UEs interoperate with their different network configurations before deploying them onto their commercial network, testbeds are expensive to set up and maintain. Besides, the number of different scenarios that may be investigated with their unique network configuration makes relying on real network testbeds difficult and expensive. Consequently, the use of network simulators for interoperability testing of network elements and their connected devices in the lab have taken the center stage of interoperability testing by device vendors across the wireless industry. Also, carriers carries around the world have resorted to using simulators in the lab to help them meet their acceptance testing challenges. This increase in focus and reliance on lab IOT and CAT using simulators is now seen as strategic in the competitive warfare across the wireless industry. However, for Lab testing to be beneficial and to play the strategic role in the competitive warfare, the right simulators that are 3gpp compliant ought to be assembled through careful investigations and cost-benefit analysis.

REFERENCES [1]

[2]

[3]

[4]

Y. Wang, C. Wu, and X. Chu, “ Quality of service management in emerging wireless networks”. International Journal of Network Management. Vol. 21, 267-268. doi: 10.1002/nem.791 O. Kone, “An Interoperability Testing Approach to Wireless Application Protocols.” Journal of Universal Computer Science, Vol. 9(10), 1220-1243. ETSI. “Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT); Generic approach to interoperability testing” [ETSI EG 202 237 V1.1.2 (2007-04)]. Retrieved from http://www.etsi.org. 2007 P. Dutta, “IOT Pre-Qualification Testing” http://www.computerweekly.com/Articles/2011/04/08/2462 50/White-paper-IOT-Interoperability-Testing-PreQualification.htm. 2010.

Dr. Dominic Boakye Boamah received a Diploma in Data Processing from Kwame Nkrumah University of Science and Technology in Ghana, in1993, M.S. in Economics and Business Administration (Information Systems) from the University of Jyvaskyla in Finland, in 1999 and Ph.D. in Organization and Management (Information Technology Management) from Capella University in Minnesota, U.S.A., in 2007. He has worked in the wireless industry for the past 12 years including 8 years with Nokia. He currently works with the Wireless Measurement Solutions unit in Anritsu Company. He has served as an adjunct instructor and a consultant on course design and delivery particularly online, for several schools

including Colorado Technical University, Grand Canyon University, Ashford University, Central Texas College, University of Fairfax and the Ghana Institute of Management and Public Administartion. He is a member of the Dallas Chapters of the Project Management Institute and Toast Masters Club. His research interest includes Regulatory Policies and Proliferation of Information Technologies, Alignment between Business and Information Technologies, Project Risk and Quality Assurance, and Organizational behavior and the development of effective teams. Dr Erasmus Addae received a Bachelors degree from the University of Ghana in 1993, a Masters in Information Systems from Tarleton State University Texas, USA in 2000 and Ph.D. in Educational Computing from Nova Southeastern University in the USA, in 2007. He is currently an Associate Professor of Information Systems at Collin College in Texas, USA. He is a member of ASTD and Merlot.