The characteristics of the cloud create a new set of considerations for the entire lifecycle of performance testing from test planning to results analysis and reporting.
Test planning:-
When it comes to test planning, user stories and success criteria are very similar. You still must be concerned with defining objectives for end-user response times, numbers of users, and the ability to meet the requirements of service-level agreements (SLAs). But the way your test bed is designed and constructed can be very different.
Let’s walk though some of these concerns:
• Test bed issues— Your test bed may need to be designed to scale up a number of nodes, scale out a number of nodes, and scale across multiple cloud vendors. What’s more, it might all be located off premises, so you’re not setting up the physical test bed in your local test lab; instead it’s out there in the cloud.
• Response times—If you’re used to testing response times with wide-area network (WAN) emulation, your issues are going to be compounded by the cloud. When your application is hosted on the cloud by external resources, latencies are beyond your control. That means you’re going to have to do extra work to include calculation of real WAN latency combined with emulated WAN latencies.
• End-user experience—Growing numbers of companies are hosting web services in the cloud. Testing the performance and scalability of these cloud-based services is important, but that alone is not enough. To understand the exact user experience, you must test end-to-end response times for the client applications that used these remote web services. This is very important when it comes to hosting an end-user client on a web server in the cloud, such as a rich Internet application (RIA) that is accessing the web service.
• International requirements— You may need to consider certain international requirements and issues, including currency calculations and units-of-measure calculations. These can compound performance issues that require additional background processing or real time computations and translation.
• Green computing— As we noted earlier, cloud vendors have different approaches to optimizing system efficiency. So you can expect green computing criteria to differ from cloud vendor to cloud vendor.
• Test data regulations— You might have test data regulations that limit your ability to put test data out into the cloud. This is an important issue. It suggests you need to pay attention to privacy issues and compliance issues surrounding test data management.
Test development and construction Scripts that worked on-premises might not work at all when you move an application to the cloud. There may be different security requirements, firewalls, network routing, and permissions and access. For instance, the directory services that you are using for the application might change, or be completely unavailable out on the cloud.
Authentication type might change from internal certificates or keys to simple user names and password combinations. This means that when you take a script that worked fine on-premises and move it to the cloud, you might need to either rebuild it or update it to make it work properly in a cloud environment. This can happen even if the application under test is still hosted from your own company’s Internet connection, via your Internet service provider (ISP). Also keep in mind that in the cloud, the test and development environment is more dynamic. Because you’re running on a virtualized platform, the test or development image might suddenly “move” to a different execution location. In the middle of recording a script you are developing against one host and one ID, your image moves to another data center. This can happen automatically, behind the scenes under the control of the cloud service provider. Whether you’re on virtual IP or physical IP, your test bed, and even your development environment, will be more dynamic. Because it’s remotely hosted and it’s on a virtual platform, that AUT is not exactly the same physical, solid, local application that you had before.
Budgetary considerations
The cloud application under test should be included in your budget for testing. There’s a real-time cost associated with your test environment now, in terms of bandwidth, system utilization, storage utilization, and actual utilization for the AUT. If the application is running on third-party infrastructure, there are charges associated with renting those machines. This means you have to budget differently from how you would budget if you were buying your own machines and running your tests locally.
One other consideration: You might want to virtualize costly or impossible services from the cloud. This could be the case if you have a third-party web service that is available only for production use, such as a shipping lookup or a geographic lookup service. You have to pay a transaction cost to the third-party vendor for a service like this, and the contract might preclude you from using the service for anything other than production uses. So you might want to virtualize, or sub out, those costly third-party calls to other web services that are running in the cloud.
Test execution:-
When it’s time to execute the test, you are likely to find that it is extremely difficult to monitor the application under test from on-premises. Many monitoring solutions require specialized opened ports, and there could be firewall implications that affect your ability to monitor the AUT.
Another consideration is the need to remotely monitor usage or instances of the cloud. While you are running your test, some cloud vendors will give you information about how many bytes per second you are moving. So there is actually a type of external monitoring of the cloud infrastructure itself. This allows you to see what’s going on beneath the AUT, just as you would with a physical machine in your own data center.
Another good technique in test execution is to use a baseline transaction, or something that is very generic, such as a ping, that hits each node in the architecture. This baseline transaction gives you an idea of latency. This is a “canary in the coal mine” idea. Because the cloud AUT environment is so dynamic, you want to have at least one baseline transaction for relative comparisons. If you can’t have true visibility into certain factors that affect performance—such as bandwidth limitations, the distance and latencies between zones within a cloud vendor, the place where a VM is actually running, and latencies due to movement in the cloud—you at least want to be able to make relative comparisons.
As we noted earlier, your AUT can move around within the cloud infrastructure, because of the elasticity of the cloud. When resources can scale up and scale out dynamically, and when your AUT can suddenly move, best practices call for some type of dynamic monitoring.
In addition, we recommend real-time investigation and optimization. When the AUT is up and running and a bottleneck occurs, it’s important to have diagnostics, to have a profiler, to have the ability to drill down and find out what query is running. In the cloud, the AUT has an on-off switch, so you might find that at the end of your testing, the AUT turns off and the virtual machine is gone—completely. And the next time you run the same text you might be in a slightly different zone or using a slightly different set of resources. So it’s really important to be able to investigate the cause of bottlenecks while you are running your test.
Test results analysis and reporting
Cloud “weather” can affect the accuracy of your test results. Dynamics such as movement in the cloud or changes in the cloud topology are important. It’s also important to use trending across cloud vendors to compare costs and benefits. Sometimes performance is better with one cloud vendor or another, depending on your application; the location of your users; and the type of operating systems, kernels, and platforms you’re using. It’s OK to test them to identify the best fit for your application.
Another consideration: Large test results stored in the cloud can increase costs, because a large set of test results can be many gigabytes. So it makes sense to think about downloading your test results onto a local drive and archiving them locally. And as we mentioned earlier, root-cause investigation can be difficult when the cloud application under test is turned off, is not running, or is simply gone. So how you manage the cloud AUT while the test is running can make a big difference in terms of your analysis capabilities.
