2. SOAPSonar – Starting with SOAP and REST

So you have installed SOAPSonar and it’s registered as is detailed in this earlier tutorial. Were now do you start? First let me explain how SOAPSonar works. SOAPSonar is device independent, and requires no client or client application in order to test the Web Services API. It sends requests directly to the Web Services API, and receives the response.

The SOAPSonar Interface is divided at the top into 3 Views. Project View, Run View and Report View. In Project View we create the individual test cases, defining data source, variables and success criteria. In Run View we combine these test cases into automated test scenarios, by dragging or linking the independent test cases created.We can also define virtual agents, for load testing and logging option. In Report view we see the results of the independent test cases that are run, compared against the success criteria we established. The logs provide addition detail on for further trouble shooting errors. On the right is 4 modes to run the tests in.

QA Mode, Performance, Compliance and Vulnerability. SOAPSonar uses the same test cases created in Project View, and Test Scenarios in Run View, to run in any of these modes, yet provides different reports.

A SOAP Based API

SOAP Web Services API, provide a WSDL, which can be captured by SOAPSonar and used to automatically generate a UI. For example if you put http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx into your browser, you will see a free basic SOAP service example for calculation.

If thiis service seems to be down try http://www.w3schools.com/webservices/tempconvert.asmx?wsdl

1. Make sure you are in project view and QA mode. Place that same link into SOAPSonar’s Capture WSDL bar and add ?wsdl afterwards or paste http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx?wsdl Then select Capture WSDL or press enter, you can automatically import the available services. If you look on the left, in the project Tree, you will now see a service called Calculator.asmx which has 4 services, Add, Subtract, Multiply and Divide.

2. If you select Documents you can view the WSDL and Embedded Schema.

3. Now lets see the automatic UI for the Add service. Select Add_1 service. In the main section, there is a Request section on top, and under that a Response section. Lets start simple and enter a= 2 and b= 2 then the green check mark to commit the test settings and the blue arrow to send current request to server. If you look at the Response section, you should see the result is 4.

4. If you select the XML tab at underneath the Request Section, you can view the XML Request that was sent. Changes in the Schema Fields or here are mirrored. Change the a field in the XML by changing the 2 to a 3 in <tns:a>3</tns:a> then then the green check mark to commit the test settings and the blue arrow to send current request to server. Does the response now read 5?

REST Based API

REST based API do not provide a WSDL that can be captured. Their lightweight nature, requiring a bit more knowledge or documentation.

1. To start a REST test, select File, New, Test Group. You should then get NewCustomGroup1 in the project Tree. If you right click in it you can rename it REST Demo.

2. If you Select Tests, then right click and and select New REST Test, you will see a similar but slightly different Request and Response section.

3. Lets Check the Bitcoin price. In the URI in the rest section, paste https://btc-e.com/api/2/btc_usd/ticker and leave the method as GET. then the green check mark to commit the test settings and the blue arrow to send current request to server. What did you for a response? My high I got was US$584 today.

4. Notice that by pasting in the URI, it is then broken down into Protocol, Host, and Path. Right click on the New Test_1 and rename it BTC_Bitcoin.

5. Please save your test plan at this point for the next Tutorial on Run View.

Conclusion

Well now that you have some idea how to start a REST or SOAP Test case development, take a few minutes to try a few additional service or services from your own environment. Try a BBC Music artist

http://www.bbc.co.uk/music/artists/ada7a83c-e3e1-40f1-93f9-3e73dbc9298a

or SOAP Soccer information service

http://www.xmlsoccer.com/FootballData.asmx?WSDL.

For that matter, Facebook, twitter and many other companies offer API’s that can be used, although some require a developer registration and key first. Perhaps you have a service you familiar with you would like to try?

We now have some idea of project view and how to start with a SOAPSonar project. next we will look at Run View and how to use some of these unit tests.

Comments and question below will help others learn.

TASSQ Presentation

I want to thank everyone for their participation and input in last nights TASSQ presentation. Although its was a very simplified scenario, using a LOT of Averages, and the fixing of the 20,000 test cases required limited options, the limitations were set to demonstrate the impact of change more clearly. If we had more time, we could have looked into changing coverage ratio’s and changing the number of test cases required per Release or test iteration. I encourage everyone to do just this, using the own projects, to see what personal insight it will provide.

I have put the slides and spread sheet used for My TASSQ presentation on Google Docs for view or download.

Much of what was presented is in more detail on this blog, which you can subscribe to receive a monthly newsletter of updates. I also encourage all to make comments and share thoughts. Only a screen name is displayed.

Thank you again, and I hope you found it as interesting as I did.

More Detailed look at Service Planning Costing Part 1

In an earlier series, I went over a number of costing models that I have come across, that are used by different organizations. Service Planning Costing being one of the more detailed and specific, and often used by outsourcing companies. What is most interesting around service plan costing, is that a change of a single iteration, or minute in testing time can have major impact to the total price for the project. That said, I have seldom met an organization that has statistics or KPI on test times.

To understand this better, lets look at Testing as just one part of the Software Development Life Cycle. The majority people experience higher testing costs during product maintenance than development, yet at that point expenses have settled and are generally known. During the testing phase however, there are many more unknowns.

During software development testing, there are a number of different levels or test types. The first and most common test is the basic unit test, testing a function in the service to ensure that it does what it is intended to do. Each time the code is revised, a unit test is usually run to ensure that nothing unexpected has resulted. At some point, code is consolidated in a “release drop” and Integrated testing is done. Integrated testing, ensures that each unit interoperates with the others. Systems Testing is the next level and is around testing the entire system. This includes things like security testing, Performance testing under load, and other specialized tests to cover different environments. At each of these testing levels and each test iteration is usually compared to the previous baseline, in what if normally referred to as regression testing. The final testing level is user or business acceptance testing. Depending on the complexity of the application and the development approach and the maturity of the organization, there can be anything from 5-50 Testing level iterations. For example, in Agile environments, the sprints are usually shorter, yet need to include each of these test levels, while in waterfall, a company may only begin integration testing once all unit testing is completed.

Now each test Iteration at each testing level follows a process. A Test plan is developed, and the expected outcome along with intended inputs is created. This then is executed in a given environment, and the outcome is compared and analysed against what was expected. Any unexpected results or differences require further analysis. Afterwards a test report is completed, detailing what was done and the result The test cycle can be brief, in a unit test or lengthy as it may be in acceptance testing. In a single web service, there may be 10-20 function and each function may need to be tested. Possibly with a positive and negative or some other boundary requirement. The result could be hundreds of test cases for a single web service. Thousands of test cycles can make up a single test level and each test level can be repeated with each code drop or systems change. Since employee salaries are the largest costs in software testing, its makes estimating these times essential.

Since each of these processes is repeated exponentially, a small variance of a few seconds allocated to each test cycle or to the number of test levels in costing, can result in huge impacts to the costs. This is especially difficult to manage, when environments, code releases and other parties invested n the process may not appreciate the costs impacts.

This touches on core issues with QA and testing costs. I will be speaking more on this Tomorrow at TASSQ meeting in a workshop, and in future posts. Drilling down on some thoughts on how to reduce the costs and increase the accuracy of costing. In the mean time please feel free to comment below.

Installing a SOAPSonar License Server

SOAPSonar comes in a per seat and floating license models. Floating license are popular for occasional users and offshore sharing as they allow SOAPSonar to be installed on as many users machines as you would like, and only counts concurrent users.

The license controller application is only used briefly on checking out of the license and checking the license back in. The instance of SOAPSonar still runs on the local machine. As such minimal “server” load and network is required. The license controller is in fact a small process that runs on a windows machine, which requires a static IP address. On my machine its says its using 20,000k of memory, about 33% of chrome and the entire file is only 4.2 MB.

1. To install the license Controller start by downloading it from here. Install this on your server with Admin Rights. Once installed, although I try keep this post up to date, please check immediately if this is the latest version. To do this, run the license controller and select Updates, Check for Updates.

2. Make sure the the process is running and automatically starts on your server. If the service is unavailable, license will not be available to check in or out. Right-Click on the icon in your tray, and select Manage Licenses.

3. Every server is fingerprinted, and your licenses tied to that server. Select File, Show Server ID, and copy this value. Send this FingerPrint ID to [email protected]

4. Your license will then be generated by Crosscheck Networks for that server ID, and we will send you back your license key. Select File, Add Licence Key and paste your license. Select OK. Each group of licenses should now show as a line item.

5. Select File, Preferences, and confirm and on the License Server Settings tab, confirm your port number. Run cmd, then enter ipconfig to get your IP address. These are needed for users to connect to the server.

6. On the second tab, Client Installation Version Control, you can set the minimum or maximum version and the download location to ensure that your team uses the same release. This is helpful if you sharing projects.

7. Statics offers a way for you to see which users use what licenses and how often. Enabling planning between seat and floating licenses.

8. On a workstation, download and install SOAPSonar. Its important to do this by right clicking on file and run as admin, for access to .Net libraries. Select Registration, Use License Server.

9. Enter the IP address and Port of your license server, Select Key Type and Request License.

If you have any question, please just reach out to me.

ST3PP Presenting at TASSQ in March

Join us at TASSQ this March the 25th in Toronto at the The Albany Club – King & Church Streets 5:30 pm networking 6 pm dinner 7 pm presentation. March’s presentation will be done by ST3PP’s me, Ivan Flowerday. Visitors and new members are always welcome at TASSQ and attendees include anyone who has an interest in software quality, or software testing, whether they work for a large corporation, a small software house, or whether they are independent consultants.

ST3PP’s mission is to Align the domains of Tools, Process and People, to achieve excellence. At TASSQ, I will be sharing some thoughts and ideas collected by meeting with a number of QA and Development organizations. The session will also be interactive, with attendee participation. Although it will cover all 3 domains, no specific organizations Tools, People or Processes will be used. The intent being to generate ideas or thought that members can further investigate, and perhaps apply in their teams. Similar to much of ST3PP ahead newsletter.

To give you some idea of what I will be discussing you can read a detailed look at service plan costing

We look forward to seeing you there and please stop by an introduce yourself if you have not already.

KPI – Identifying Areas in Process that Requiring Tuning

Both Six Sigma and CMMi focus on creating and managing a feedback loop. Assessing the current status in your organization and then identifying areas for improvement. To do this, KPI or Measurement Objectives are needed to measure the effectiveness of your organizations Tools, People and Processes. Measuring details like Defect Density, Test Coverage, Total defects, reliability etc. Then collecting the relevant data and analysing these measurements to provide insight into the the results, set goals and track progress.

One of my favourite tools or methods to visual and analyse KPI is via Pareto Graph. Its simplicity makes the 80/20 rule stand out to any level of an organization. In the above example the first 7 groups of defects is clearly well over 80% of the all defects. The impact of addressing the others being minimal, while a small change in the first 3 could have major impact to your organizations quality.

Potential issues with Pareto Graph.

Although extremely helpful in identifying areas that need to be addressed, there are some potential pitfalls on relying on Pereto analysis too much. These include

It is important to define and group the categories correctly. They need to both be precise, so that there can be little doubt as to which category the defect fits into and they need to be of a suitable granularity. Making larger groups, can make addressing the sub issues in the groups difficult, yet making too many sub-groupings can result in loosing the visual effect and intent of the pareto.
Selecting appropriate time periods is important. Measuring only a single release cycle or team, can result in wild fluctuations in results. Consolidating multiple teams, releases etc can cloud visibility into a particular team or release issues.
If changes are made to the Tools, People and Process, to address the issues identified in the pareto chart, adding them to the same period will not offer any insight into the effectiveness of these changes. These changes need to be compared to a previous baseline. Baselines however can become rapidly dated.
Pareto analysis is draws focus on areas of greatest need, but a laser focus only on those areas, can rapidly destabilize others. Some of these other fixes could be potentially low hanging fruit requiring little effort to resolve.
Although Pareto analysis identifies common areas that may require focus or consideration, it does not supply solutions.

Poll

What I have learnt is that when someone sits down and actually gathers the information and generates a Pareto chart for analysis, they are almost always surprised by what is revealed. For benefit of all, I wanted to take a short Poll. First off, looking at teams that do use pareto analysis and teams that don’t. What I would be interested in seeing, is if these teams see common or different set of issues. In other words, does doing a pareto analysis change peoples list of most common defects. The results will be collected privately and consolidated by me, and posted at a later date. Comments at the bottom of the page posted before hand for public, but comments in the poll kept private unless you stipulate in them you wish to be mentioned.

Please take a minute to weigh in and help others ST3PP ahead.

[contact-form subject=’pareto analysis’][contact-field label=’Screen Name’ type=’name’ required=’1’/][contact-field label=’Does Your Organization Use Pareto Analysis’ type=’radio’ required=’1′ options=’Yes – we review them frequently,No – I have never seen one for my organization’/][contact-field label=’1 – Most Common Defect Group’ type=’select’ required=’1′ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’2 – Second Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’3 – Third Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’4 – Fourth Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’5 – Fifth Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’Private Comment To Me’ type=’textarea’/][/contact-form]

Monthly Archives: March 2014