a ST3PP in the right direction
a ST3PP in the right direction
We will be at STAR Canada next week on the demo floor in a booth. Please stop by and introduce yourselves! Its a annual chance to meet some of our users and meet many people face to face for the first time. We look forward to seeing you.
Its at the Hilton Downtown Toronto and we will be there Tuesday 8th all day and Wednesday 9th till 1:00pm.
For those of you who attended my TASSQ presentation, this post is not new, but for those that did not, I wanted to so a post to give you some background.
The example we used at TASSQ was;
“If you have an assignment to test 500 services, each service has 10 functions and each function requires 4 tests, upper boundary, lower boundary, and 2 negatives. Using service plan costing, the total number of test cases you would need to test is then 500 x 10 x 4 = 20,000.”
Now before you say, there hundreds of ways to reduce that number, let me agree and say, yes there are many ways we could. But lets leave that discussion for other posts, or another presentation (was actually something I was going to speak to) and pretend that this number is is the result AFTER all optimization is done. The Reducing Amount of Test Coverage series of posts will address ways this can be done.
The TASSQ numbers were numbers in the second column, I got polling the audience, and I believe are pretty conservative.
The result is a project cost of $1,473,500 which would take 10 resources at 8 hours a day 450 days to test.
Now for the not so obvious Truth about Testing.
So regardless of if your business uses service plan costing or not, it is a valuable tool for understanding and dissecting QA costs. For focussing attention on the matters that really effect costs. In fact, I believe KPI and pareto analysis of these KPI is important for any QA manager. I always like to hear ideas that reduce testing time or test cycles, so please, feel free to share your thoughts.
If you have not, please do Starting with SOAP and REST Tutorial first.
Automation lets you test the business flow scenarios. So now lets model a very simply business flow. Lets take an example, the company uses a 3rd party API for calculating distance between cities. This is a REST service. They have a second API that they use to calculate. The Client application makes use of both. The test requirements are that users are expected to calculate the time it would take to ride a bicycle between Toronto and Montreal if you travelled and average speed of 20 km/h. To do this we set up out test cases in project view and run it in Run View. But to automate this, we need the response from one service to be the input into the second. This we at ST3PP refer to chaining services for automation.
TUTORIAL
1. Open SOAPSonar, and lets first add the REST distance service. In this case I am going to use Google Maps. File, New, Test Group. Then right-click on Tests in the Project Tree and select New REST Test. It’s always good to give names to things, so right click on New Test_1 and Name it Distance.
2. Paste the following in to URI field.
http://maps.googleapis.com/maps/api/directions/json?origin=Toronto&destination=Montreal&sensor=false&avoid=highways&mode=bicycling Make sure Method is GET and then Commit and Send.
3. In the Response section we interested in the distance. Unless Montreal and Toronto are floating apart, you should get 621 km at 1 day 8 hours. But at what speed? Select the tab Runtime Variables, then scroll down till you get the distance value field of 621476 (meters). Select this value number and right-click, Add Variable Reference. Name it Distance. OK, Commit.
4. In Tutorial 2 we used a SOAP calculate function, lets add that. Since SOAP comes with WSDL, just paste
http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx?wsdl into the capture WSDL bar.
5. Now on Divide_1, right-click and Clone, twice. Renaming the the first to KM and the second to Hours.
6. Select KM and we need to enter the response from distance in Meters, into a = and b= should be 1000, the response needs to go into hours. in a= right-click, [RV] Runtime Variable, New Custom Group, distance. Set b= to 1000. Then Commit and Send.
7. In the Response tab, select Runtime Variables, then Divide Result and right-click. Add Variable Reference and name it Distance_KM. Notice the tab shows Runtime Variables (1) now?
8. Lastly we need to divide that response by 20 km/h. Select Hours in the project tree, in a= right-click, Select [RV] Runtime Variable, Calculator.asmx, Distance_KM variable. then b=20. This should be the result we have being asked to model.
9. Switch to Run View and drag Hours test case under the DefaultGroup. We should name that our scenario.Notice that Distance and KM are known prerequisites and so get added automatically.
10. Make sure we use the Test Case Success Criteria as we have not captured a baseline and lets Log Fail Only. Make sure you not Publishing this to HP Quality Center Project. Commit and Run suite.
You automatically placed into Real Run Monitor, but we talk about Report view and logs in next post. Save your project as Tut3.
You have just automated a test scenario of 3 unit tests across both REST and SOAP API’s that reside on different systems and at different locations (external) to your network. Yes you could have done this faster manually the first time, but we will look at ways to baseline and monitor as well as use this example for more locations or different speeds.
Comments? Please share any thoughts or provide feedback so others can learn.
So you have installed SOAPSonar and it’s registered as is detailed in this earlier tutorial. Were now do you start? First let me explain how SOAPSonar works. SOAPSonar is device independent, and requires no client or client application in order to test the Web Services API. It sends requests directly to the Web Services API, and receives the response.
The SOAPSonar Interface is divided at the top into 3 Views. Project View, Run View and Report View. In Project View we create the individual test cases, defining data source, variables and success criteria. In Run View we combine these test cases into automated test scenarios, by dragging or linking the independent test cases created.We can also define virtual agents, for load testing and logging option. In Report view we see the results of the independent test cases that are run, compared against the success criteria we established. The logs provide addition detail on for further trouble shooting errors. On the right is 4 modes to run the tests in.
QA Mode, Performance, Compliance and Vulnerability. SOAPSonar uses the same test cases created in Project View, and Test Scenarios in Run View, to run in any of these modes, yet provides different reports.
SOAP Web Services API, provide a WSDL, which can be captured by SOAPSonar and used to automatically generate a UI. For example if you put http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx into your browser, you will see a free basic SOAP service example for calculation.
If thiis service seems to be down try http://www.w3schools.com/webservices/tempconvert.asmx?wsdl
1. Make sure you are in project view and QA mode. Place that same link into SOAPSonar’s Capture WSDL bar and add ?wsdl afterwards or paste http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx?wsdl Then select Capture WSDL or press enter, you can automatically import the available services. If you look on the left, in the project Tree, you will now see a service called Calculator.asmx which has 4 services, Add, Subtract, Multiply and Divide.
2. If you select Documents you can view the WSDL and Embedded Schema.
3. Now lets see the automatic UI for the Add service. Select Add_1 service. In the main section, there is a Request section on top, and under that a Response section. Lets start simple and enter a= 2 and b= 2 then the green check mark to commit the test settings and the blue arrow to send current request to server. If you look at the Response section, you should see the result is 4.
4. If you select the XML tab at underneath the Request Section, you can view the XML Request that was sent. Changes in the Schema Fields or here are mirrored. Change the a field in the XML by changing the 2 to a 3 in <tns:a>3</tns:a> then then the green check mark to commit the test settings and the blue arrow to send current request to server. Does the response now read 5?
REST based API do not provide a WSDL that can be captured. Their lightweight nature, requiring a bit more knowledge or documentation.
1. To start a REST test, select File, New, Test Group. You should then get NewCustomGroup1 in the project Tree. If you right click in it you can rename it REST Demo.
2. If you Select Tests, then right click and and select New REST Test, you will see a similar but slightly different Request and Response section.
3. Lets Check the Bitcoin price. In the URI in the rest section, paste https://btc-e.com/api/2/btc_usd/ticker and leave the method as GET. then the green check mark to commit the test settings and the blue arrow to send current request to server. What did you for a response? My high I got was US$584 today.
4. Notice that by pasting in the URI, it is then broken down into Protocol, Host, and Path. Right click on the New Test_1 and rename it BTC_Bitcoin.
5. Please save your test plan at this point for the next Tutorial on Run View.
Well now that you have some idea how to start a REST or SOAP Test case development, take a few minutes to try a few additional service or services from your own environment. Try a BBC Music artist
http://www.bbc.co.uk/music/artists/ada7a83c-e3e1-40f1-93f9-3e73dbc9298a
or SOAP Soccer information service
http://www.xmlsoccer.com/FootballData.asmx?WSDL.
For that matter, Facebook, twitter and many other companies offer API’s that can be used, although some require a developer registration and key first. Perhaps you have a service you familiar with you would like to try?
We now have some idea of project view and how to start with a SOAPSonar project. next we will look at Run View and how to use some of these unit tests.
Comments and question below will help others learn.
I want to thank everyone for their participation and input in last nights TASSQ presentation. Although its was a very simplified scenario, using a LOT of Averages, and the fixing of the 20,000 test cases required limited options, the limitations were set to demonstrate the impact of change more clearly. If we had more time, we could have looked into changing coverage ratio’s and changing the number of test cases required per Release or test iteration. I encourage everyone to do just this, using the own projects, to see what personal insight it will provide.
I have put the slides and spread sheet used for My TASSQ presentation on Google Docs for view or download.
Much of what was presented is in more detail on this blog, which you can subscribe to receive a monthly newsletter of updates. I also encourage all to make comments and share thoughts. Only a screen name is displayed.
Thank you again, and I hope you found it as interesting as I did.
In an earlier series, I went over a number of costing models that I have come across, that are used by different organizations. Service Planning Costing being one of the more detailed and specific, and often used by outsourcing companies. What is most interesting around service plan costing, is that a change of a single iteration, or minute in testing time can have major impact to the total price for the project. That said, I have seldom met an organization that has statistics or KPI on test times.
To understand this better, lets look at Testing as just one part of the Software Development Life Cycle. The majority people experience higher testing costs during product maintenance than development, yet at that point expenses have settled and are generally known. During the testing phase however, there are many more unknowns.
During software development testing, there are a number of different levels or test types. The first and most common test is the basic unit test, testing a function in the service to ensure that it does what it is intended to do. Each time the code is revised, a unit test is usually run to ensure that nothing unexpected has resulted. At some point, code is consolidated in a “release drop” and Integrated testing is done. Integrated testing, ensures that each unit interoperates with the others. Systems Testing is the next level and is around testing the entire system. This includes things like security testing, Performance testing under load, and other specialized tests to cover different environments. At each of these testing levels and each test iteration is usually compared to the previous baseline, in what if normally referred to as regression testing. The final testing level is user or business acceptance testing. Depending on the complexity of the application and the development approach and the maturity of the organization, there can be anything from 5-50 Testing level iterations. For example, in Agile environments, the sprints are usually shorter, yet need to include each of these test levels, while in waterfall, a company may only begin integration testing once all unit testing is completed.
Now each test Iteration at each testing level follows a process. A Test plan is developed, and the expected outcome along with intended inputs is created. This then is executed in a given environment, and the outcome is compared and analysed against what was expected. Any unexpected results or differences require further analysis. Afterwards a test report is completed, detailing what was done and the result The test cycle can be brief, in a unit test or lengthy as it may be in acceptance testing. In a single web service, there may be 10-20 function and each function may need to be tested. Possibly with a positive and negative or some other boundary requirement. The result could be hundreds of test cases for a single web service. Thousands of test cycles can make up a single test level and each test level can be repeated with each code drop or systems change. Since employee salaries are the largest costs in software testing, its makes estimating these times essential.
Since each of these processes is repeated exponentially, a small variance of a few seconds allocated to each test cycle or to the number of test levels in costing, can result in huge impacts to the costs. This is especially difficult to manage, when environments, code releases and other parties invested n the process may not appreciate the costs impacts.
This touches on core issues with QA and testing costs. I will be speaking more on this Tomorrow at TASSQ meeting in a workshop, and in future posts. Drilling down on some thoughts on how to reduce the costs and increase the accuracy of costing. In the mean time please feel free to comment below.