Search for:

Robust API

In the Data Economy, the currency is information. The near default method of accessing information today is via the developing and exposing of Web Services API as Providers of information. Most applications are developed to Consume more than 1 API, increasingly from more than one location, source and even organization. SaaS, cloud services, supply chains, payment clearing, shipping information, social media etc are all examples that rely on API’s. A good quality API is essential for success in the Data Economy and corporations need to define an approach to API quality in much the same way as they would any other product quality. ST3PP refers to this need for a strategy to ensure Robust and Sustainable API.

Traditional approach to development architecture, tightly coupled web services development with the application client development. Business gave requirements and developers created all the services to expose these requirements. Developers then developed the client UI to access these services. When they felt they were “done” they passed it to quality assurance, which tested the “application” as a whole via the client UI. Often manually entering keys into the client’s UI’s fields in an attempt to ensure functionality. If anything was identified by QA, it usually went back to development to “fix” and development decided if it was easier to be “fix” the service or the client. The next time business issued new requirements, the entire process started again.

In the Data Economy, the client application needs to be treated independent of the Web Service API. API’s are designed as re-usable components to stand independently of any client or other application that may Consume the API. The various API each Provide some portion of information witch the Client application may consolidate or refine. These API could come from multiple locations, organizations or delivery models, like SaaS, BYOD, Cloud, Open API etc . API are no longer something IT deals with, but considered as core business asset, differentiating one organization from the next in a competitive information based economy. Better API = Better ability to establish corporate value in the economic chain. To get the most form API assets, a new approach to development and QA is needed. API need to be treated independently, like an end product. Developing API to Provide information yet unknown future consumers, requires that API be Robust.

1) Functional Testing.

In the Data Economy, the need for each field in each API to be functional still exists. Since API are no longer being developed for a particular client, and independent method of testing the API to ensure no functionality, format or other limitation exists in the API. Automated testing using broadest possible data sources can further ensure Robustness.

2) Compliance Testing

Developing an API, for unknown Consumer applications requires that the API meet with certain standards, to avoid versioning based on client applications. Testing of the API needs to include the compliance of the API to accepted standards in order to ensure that a new Consumer, perhaps for a new native smart phone application, will operate in the same way a web browser client in Chrome does or another server refining the information.

3) Security Testing

Robust API needs be secure API, independent of client application Consuming the API. SQL Injections, Cross Scripting, Improper key or session management and other OWASP top 10 vulnerabilities need to be tested for. “Cloud”  Identity structures like WS, SAML and OAuth along with key management become key components of testing for Robustness. Additional information leakage though API’s with “forgotten” exposed information fields and Metadata can be filtered using a governance gateway.

4) Performance and Scalability

Performance and scalability are not only a function of hardware, but of location, encryption, message signing, network, location, wait times, retries load throttling and many other application design criteria. An application that Consumes information from a variety of API’s on different networks and managed by different teams, requires additional hardening to ensure performance and scalability. How long should I wait if one API is not available? Do I require a resend after how long? What if someone is on a poor quality mobile network, how would that effect my  performance? What if I required higher level of encryption? How many concurrent clients can I support with my current infrastructure? What if I split servers or added a second location?

Visionary organizations have started by creating “Information” or “Data Management”executive to extracted value from corporate information for the Data Economy. This involves treating API as we would an application core to the corporations success. Poor quality API, limit access and make extracting value from data near impossible. These executives need to ensure that business, development and QA structure the right process and approach to creating more Robust and Sustainable API.

Reducing Scope – the Amount of Software Testing

Our example used at TASSQ and in our a More Detailed look at Service Plan Costing we used a fix number of test cases (50,000). At the TASSQ event, many immediately wanted to discuss ways to reduce the number of test cases. This is just as important an aspect as streamlining the test process once this is done. Due to time constraints, I decided to leave it till now. But, what kind of things can be done to reduce the sheer scope of testing needed?

This is too long for a single post, but over the next few weeks, I will build it out each area. Looking at high level areas to focus at though, we have 4 key areas to focus on

  1. SDLC Strategies
  2. Test Iteration Strategies
  3. Test Cycle Strategies
  4. Maintenance Strategies

1. SDLC Strategies

What is your corporate mandate? Is these a free internet service, that is best effort, or do errors have potentially huge financial risks? How long is this software expected to be in use (next release), and how mission critical is it to your business? The way we approach testing should reflect our business needs.

The second aspect is more architectural. I have already posted one post on API Versioning Strategies. How the service and the client are designed and managed in the SDLC, can greatly impact the amount of testing needed.

2. Test Iteration Strategies

This looks at way to reduce the number or effort required in each test Iteration. Can you share the same test case for Functional and Performance Testing? How can you ensure that development did fix the issues in the last release? Do you really need to retest code that was not changed?

Strategies here vary a great deal depending on if you using AGILE or Waterfall or some other Development methodology.

3. Test Cycle Strategies

This area looks at way to reduce the number and the complexity of doing individual test. A lot of this has to do with desired percentage coverage, but automation, data sources and regression are all aspects.

4. Maintenance Strategies

Far to often we focus on getting the software into production, yet its generally accepted that testing during the maintenance cycle can be well over half the testing costs. This is about, automation,  regression and continual testing strategies that can reduce the maintenance testing costs or coverage.

I cant poll the audience here, but as usual, we share and learn. So please if you have thoughts or suggestions on the subject of reducing the amount of testing required, please let me know.

 


 

Calculating Percentage Coverage

I wanted to discuss some confusion of percentage Test Coverage. I have noticed that different organizations calculate test coverage very differently.  This can be very confusing when using contractors, off-shoring and simply comparing best practices. Lets say you have a simple service that returns Name, Phone Number and Address and you asked to create test cases for 100% Test Coverage. What exactly does that mean?

Would a simple unit test entering the following be considered 100% Coverage

  • Bob Smith
  • 555.555.5555
  • 55 street
  • City
  • QC
  • M5M 5L5

Or would you need to break the service down into each of its functions. Name, Phone Number Street, City, Province, Postal Code. Testing each of these independently?

But how many test cases do you need to perform, for you to consider it 100% coverage. Lets take Postal Codes. Would a single Postal code be considered 100% coverage? Or would you need one from each of the 18 starting letters ( Y, X, V, T, S, R etc)? Perhaps you require some random number of say 10 or 100 postal codes? Or do you need to enter every defined Canadian Postal code.  Lets consider testing name function, how long a name does the app need to support, how many names can a person have, what if we include title, what if the persons name has a suffix.

What about negative scenarios? Do you need to test postal code that does not exist, or one in the wrong format before one can consider the test coverage to be 100%? With space after first 3 letters, without space, or with a hyphen. What about letter number letter or what if all letters, number or some other possible combination? How many of these negative scenarios does one need to run to say you covered 100%?

What about testing these functions as they relate to each-other or as this service relates to other services?  Do you need to test that a Postal Code starting with letter V, is not used for a city that resides in Quebec? Do you need to confirm that this address service when used in one chained request, responds the same was as when used in another? So often I hear of companies unit testing services as they developed, but never running a final systems and integration end to end test. What if one service requires that postal code to have a hyphen and the other a space?

Understand if your organization is manually testing a service, entering even 18 postal codes will take significant time directly impacting costs. Entering all positive, negative scenarios including chained services is just not feasible. Does increasing the number of test cases actually effect the percentage coverage, or is a single test case enough? All the possible boundaries for a simple service like postal codes could result in a large number of tests. Does testing  each service once, without considering all the boundaries and negative scenario’s constitute 100% coverage? More importantly perhaps, is when QA testers give a percentage coverage, does it really mean the same thing to the everyone?

I would like to invite everyone to weigh in and share their thoughts on the subject. Please select and option and comment if you will below. So far the majority selected test every function once. So I broke this into boundaries and positive and negatives to see if we can get further clarification.

***Please note The form is submitted privately and is not automatically published. If you wish your response published, use the comment link at the end of any post***

Warning: strpos() expects parameter 1 to be string, array given in /home/content/13/11164213/html/ST3PP/wp-includes/shortcodes.php on line 193
[contact-form to=’[email protected]’ subject=’percentage coverage’][contact-field label=’What does your Organization consider 100%25 Test Coverage?’ type=’radio’ required=’1′ options=’Whatever We have Time for,One Test for Each Service,Test Each Function of the Service only once,Boundaries for Each Function,Both Positive and Negative Boundaries for Each Function,All/Many (Data Source) in Chained Workflow’/][contact-field label=’Comment’ type=’textarea’/][contact-field label=’Screen Name’ type=’name’/][/contact-form]

Continuous Testing in Agile

Along with performance testing, there were 2 other themes that continually came up in conversations during STAR Canada.

  1. How should QA integrate in a Agile environment
  2. The need for “Continuous Testing”.

While there are thousands of articles about Continuous Testing, and hundreds of thousands on Agile, there seems little on both. Perhaps due to some apparent conflicts.

Lets look at a theoretical QA in an agile environment. Say your organization Sprints are 2 weeks in length, each scrum having 8-10 members for manageability. Due to project time constraints,  there are 5 scrums working concurrently, each focussed on a different component of development of your application. What test cycles are done as part of the sprint and what cycles are done outside or as cross functional teams?

Agile Testing Levels

It was pointed out that although common, doing only unit tests and integration testing on your Sprints code, then jumping to acceptance testing of that sprint, is not Agile. Agile should in fact have all test stages built into the Sprint. Many companies ignore various test cycles like load, integration and security testing of the end to end system as there simply is not time in each Sprint.

An alternate approach is to created independent teams outside of the Agile development. Their role is to test integration, load, security and systems in integrated environments. Defects identified are then fed back into the scrum meetings and allocated to particular Sprint. This also is not really Agile, falling into some kind of hybrid. The challenge here is that issues are often identified after sprints are finished and so not really continuous testing either.

A second approach, was to create cross functional roles were the scrum masters and one or more members of each sprint were allocated to doing systems level testing and possibly fixes. These cross functional teams, would near the end of each sprint, break out of their old scrum into the new role. The challenge with this approach is that on shorter sprints, and large systems, they can land up spending more time in the cross functional role than in their own scrum.

Continuous Testing

Continuous Testing is somewhat the same as Baseline and Regression Testing, but need not only testing against a Baseline. Its about continually testing while developing through the entire SDLC. The benefit that issues can be identified far earlier (Shift Left approach) resulting in lower costs to address. Agile environments at first glance, seem to favour continuous testing, but does that include, regression, integration and systems testing across Sprints? If each test case takes 9 minutes to complete, 1 tester can only run 53 test cases in a day or 533 tests in a Sprint. This is simply not enough coverage to test all systems and other tests continuously. The result, is partial or low test coverage.

Enter Automation

If as part of each Sprint, a fully developed set of test cases are developed by each Sprint in the same application (eg SOAPSonar) covering their development efforts. The incremental work to role these up into test cases for integration, load etc would be minimal. Each sprint then shares a set of Integration, performance, load and regression etc tests that they simply run as part of their sprint. Being automated, these can even run after hours. The result is continuous testing of both at the Systems level and the Sprint level, without the heavy resource requirements of manual testing. issues be the system wide, or sprint level can then be addressed in Sprint.

Conclusion

The concern with this is the same as with any automation project, “Will The Development of the Automation Scripts Not Take More Time than the Resulting Benefit.” This is a tool selection question. Finding the right tool for your team to use to minimize the time taken developing and maintaining various test cases from function through load and regression, to acceptance testing.

Would you like to weigh in with your thoughts or comments on the subject?

MORE DETAILED LOOK AT SERVICE PLANNING COSTING PART 2

For those of you who attended my TASSQ presentation, this post is not new, but for those that did not, I wanted to so a post to give you some background.

The example we used at TASSQ was;

“If you have an assignment to test 500 services, each service has 10 functions and each function requires 4 tests, upper boundary, lower boundary, and 2 negatives. Using service plan costing, the total number of test cases you would need to test is then 500 x 10 x 4 = 20,000.”

Now before you say, there hundreds of ways to reduce that number, let me agree and say, yes there are many ways we could. But lets leave that discussion for other posts, or another presentation (was actually something I was going to speak to) and pretend that this number is is the result AFTER all optimization is done. The Reducing Amount of Test Coverage series of posts will address ways this can be done.

Service Plan Costsing

The TASSQ numbers were numbers in the second column, I got polling the audience, and I believe are pretty conservative.

  1. Number of Release Iterations – This is how many times you get a code drop from Development that is tested by QA. Each time, QA runs through a series of tests, be the manual or automated.
  2. Integration Testing is testing integration with Identity systems, 3rd parties and existing systems.
  3. System Testing includes testing the system as deployed. Testing the environment. Load vs performance, DR, preproduction, and production etc.
  4. Time per test case, is a breakdown of time taken to run a single test
  5. Hourly Rate all in, needs no introduction – we tend to have a slavish focus on this number which often translate into salary cuts.
  6. Tools and Infrastructure costs is the capital expense for buying software etc. to test
  7. Training – far to often Zero
  8. Additional overhead, any number not previously in there, like recruiting costs etc.

The result is a project cost of $1,473,500 which would take 10 resources at 8 hours a day 450 days to test.

Now for the not so obvious Truth about Testing.

  1. In Column 3, we changed 1 number, test time, to understand the effects it has. If you could save 1 minute a test case, the multiplier effect translates this into $160,000 savings or some 11%. How do you like that for a reason for optimization or Automation. Time and Time again I am told, “but we can do the same thing in-house without buying tools” and time and time again, I ask, “but does it take more or less time”. I heard QA staff complaining about no training, and yet, a course would be cheap in comparison. More Importantly for some we cut 50 days off testing time!
  2. In Column 4 we dropped one test cycle for a saving of $120,000 or 8% or 37 days. Remember the last time development gave a code drop that was not fully baked? Why do people use different tools and test case’s for functional and performance tests, when they could wrap them into one?
  3. In Column 5 we reduced the hourly wage by $5/hour. The effect, $180,000 or 12%, yet time remains constant…. Not as much effect as you thought perhaps?
  4. Column 6 we tripled tool costs, in an effort to reduce testing time or release cycles. The effect is some 4%

Conclusion

So regardless of if your business uses service plan costing or not, it is a valuable tool for understanding and dissecting QA costs. For focussing attention on the matters that really effect costs. In fact, I believe KPI and pareto analysis of these KPI is important for any QA manager. I always like to hear ideas that reduce testing time or test cycles, so please, feel free to share your thoughts.

TASSQ Presentation

I want to thank everyone for their participation and input in last nights TASSQ presentation. Although its was a very simplified scenario, using a LOT of Averages, and the fixing of the 20,000 test cases required limited options, the limitations were set to demonstrate the impact of change more clearly. If we had more time, we could have looked into changing coverage ratio’s and changing the number of test cases required per Release or test iteration. I encourage everyone to do just this, using the own projects, to see what personal insight it will provide.

I have put the slides and spread sheet used for My TASSQ presentation on Google Docs for view or download.

Much of what was presented is in more detail on this blog, which you can subscribe to receive a monthly newsletter of updates. I also encourage all to make comments and share thoughts. Only a screen name is displayed.

Thank you again, and I hope you found it as interesting as I did.