Software Testing and Evaluation Thoughts of Kelvin Ross

2011-07-05T17:48:00.000-07:00

The Army's $2.7 Billion Computer Designed To Help Troops In Iraq And Afghanistan Doesn't Work

http://www.businessinsider.com/armys-billion-dollar-cloud-computer-in-iraq-afghanistan-hurts-troops-2011-7

Cannot perform simple analytical tasks
has trouble finding reports
the mapping software is incompatible with the search software.

This post provided to me by Rocket, one of our rising star graduate testers.

Questions Regarding the Ross Report

2011-06-17T06:19:00.000-07:00

I thought I would post a recent conversation I had regarding the analysis of metrics. The Ross Report collates software testing benchmarks for Australian organisations. The discussion with Michael De Robertis, Test Manager at ISoft, highlighted a mutual interest in testing metrics, and the deeper questions raised as you start to peel back the layers.

Hi Kelvin.

Thanks a lot for your detailed response.

Yes please feel free to post on your blog, and you’re welcome to include my name/title/company.

From my personal perspective, it is a breath of fresh air to establish good, constructive dialog with any like minded people who have a deep understanding of Testing and who wish to challenge the boundaries of what it all means.

One of the things I love about testing is that despite having 15 uninterrupted years in the profession I still have so much to learn, and good philosophical discussions about Testing still inspire me.

Whilst I can’t speak for regions outside of Canberra (and Sydney to a lesser degree where I used to live), I am deeply concerned with the general quality of Testing professionals out there, and perhaps this is my biggest driver for having a good, mature Benchmarking process which I refine and improve religiously.

I am seeing all types of Testers out there – those who are only politically driven and insist on a requirement having a dot on the “ i “; those who don’t understand business priorities and how to manage risk or to plan; those who only want to break systems and drive wedges with development; those who lose motivation quickly; and those who don’t understand why solid process needs to accompany what we do.

I am seeing these qualities in “certified” testers (both new and experienced), and in graduate IT professionals, many of whom do not even study one single subject on testing. In my final year of my Electronic Engineering degree, testing was a big deal. I even majored in subjects on testing (through what was then called the Australian Centre for Test and Evaluation, ACTE). However unfortunately I don’t see a passion for testing driving the Test industry as it did when I began my career. Instead it seems to be only market driven (ie we need more testers, even if they are no good at it).

Apologies for the drawn out background. In my view, this is where I see publications such as the Ross Report adding real value because the Testing industry needs proper guidance through skills validation programmes complimented with independent evaluations (benchmarks) about how well the industry performs.

Refer below, where I have responded to your follow up questions.

Thanks again.

Regards,

Hi Michael,

Thanks for your comments and feedback on the Ross Report. I really appreciate your email, as it gives us valuable feedback on the way the industry benchmark is being used and what improvements and additional information users are seeking.

Apologies first of all for not responding sooner, it has been on my long list of to do items for a little while, and I have been waiting for a little time to put some real attention to it.

I will add some responses inline with your feedback below.

Also would it be possible that I could post your comments and my responses to my blog page? I found this discussion quite interesting, and I would like to share it with others.

Cheers

Kelvin

Hi there.

Can you please forward this email to someone who is familiar with the content of the Ross Report?

I found the report very comprehensive and particularly useful to compare to our benchmark…..love the metrics! J

I must say I am a numbers person myself. I love looking at what data shows, and using that to gain a deeper understanding of the testing process. Also I have the view that "without evidence, you are just another person with an opinion".

What I did find though, was the more that you look at the results, the more questions that open up to you. You tend to question: how did the number emerge, what does it mean, what influences it, how to I improve it... And I think that is the point you make below. Certainly after scratching the surface, I want to know more.

Getting the balance right in the survey is hard. To better understand I want to ask more questions, but the tradeoff then is participation will fall off if respondents find the survey too onerous. What we probably need to do is to periodically run other more detailed surveys that drill down, so long as participants dont then suffer "survey fatigue"! Anyhow a challenge to build the just right "goldilocks" survey.

Compiling the Ross Report turned into a much larger job than I anticipated. We spent well over 500 hours, with multiple people involved. It was a big job, but we want do go deeper and wider (more respondents) next time.

We are just about to kick off the 2012 Ross Report, so your feedback is very timely.

In my personal view, there were three areas that I was hoping someone could comment further about (purely for the benefit of comparing philosophies of benchmarking).

1) More proof about Automation Tools effectiveness – The report rightly separates automation and performance testing from manual testing, however I couldn’t find measurements that specifically reported on whether automation testing (for example) improved regression fail rates or resulted in bugs being found. All that I could see was that organisations were happy with their choice (but not the backup metrics as to why). I also couldn’t find metrics on downtime needed for training, setup, configuration, execution, and maintenance of those tools.

Very good point. I think the industry is still uncertain on the real benefits they hope to obtain from automation. Few organisation measure their effort, tests automated/executed, defects found, etc. Automation ROI expectations seem to be poorly understood in terms of their economics, and the metrics to show that.

There are questions around metrics for automation, for instance, that we should measure manual test reduction. Many times we count execution reduction from automated tests, that would perhaps never have been chosed to be executed if it was manual. We probably still expend a lot of effort running tests that shouldn't be continued, as they are not providing much insight.

There were some metrics on coverage, regression re-execution specific to automation, and these indicated some surpising insights - many organisations are poorat re-executing automated tests. Obviously this requires more in-depth follow up! There were also some metrics in relation to

Your comment about defect discovery rates from automation is also pertinent. I think this is a very important metric. Also perhaps to be combined with effort, to deduce defects per hour (from automation). On page 49 you should see we report that only 6% of regression tests fail per test cycle. Regression tests have lower defect yield than new feature or defect correction tests. I would expect for many this would also apply to automation, as in most organisations automation is focussed on regression testing.

I think your comments to extenddefect detection, testing effort metrics from general testing, to specifically look at automation, is a good suggestion, and one we will look at.

2) Skills validation as opposed to Skills Certification – There is still a view amongst many testing professionals that existing certification programs are not value-adding other than to say that you have the certification. Although I have certification, I lean towards this view because I personally feel that certification could do more to prove that testing professionals have worked the hours, gained the experience, delivered the services, and have been endorsed by former and current peers. Hence I think the report would add value if it considered skills qualification and validation rather than through certification alone.

I agree, certification falls short in a number of areas in terms of skills validation. See http://kelvinross.blogspot.com/2010/10/certification-is-failing-do-we-need.html and http://dorothygraham.blogspot.com/2011/02/part-3-certification-schemes-do-not.html.

I think to go beyond the certifications that organisation's staff are attaining, we would need to get some measurement on the competencies of the staff. I am just not sure how we can measure this from an organisational perspective??? It ispretty hard to assess competency of individuals, doing it in a meaningful way for organisations, hmmm, I need to think about that a bit more.

We tried to assess where staff are sourced, in terms of career entry, e.g. from thebusiness, development, etc. and impact of staff attributes, e.g. tools, techniques, theory, etc.

In other work we have undertaken skills assessments, including with our own consulting staff, scored skills for different skill categories, and tried to role that up to a organisational profile score to aim for improvements, etc. I say tried, because getting a precise skill profile was a challenge. While this profile is important to the individual organisation, I am not sure how this as an average trend would benefit, nor how we would measure individual organisations in a consistent way.

I think also if we take a organisation norm, it would perhaps look similar if we surveyed individuals regarding their skills and competencies across a large survey set. It is also possibly easier to do it at an individual level. What it might not show us is what skill areas are in most demand, and where are the most significant shortfalls.

That being said, what you say about skills validation is definately correct. Could you perhaps give me some more suggestions on what you would like to see here from a benchmarking point of view?

You are right, I’m not sure how you would present this in a benchmark / survey form and I’m not sure how respondents would react. However, if I look at this from a real day to day issue, my problem is that I cannot be secure in knowing that a Tester is fit for the job on the basis of having certification. I recall the early 90’s when Cisco Certification was a big deal. Now it isn’t, and everyone has it. In my view Testing is heading down the same path, and therefore I rely on my own benchmarking to substitute “certification” for “validated skills”.

I think your report almost has the structure in place to do this when you look at the Perceptions section. For example instead of a Test Manager answering a question about what their perception is about something, why not have a section whereby a number of disciplines are required to answer (ie Developers, Dev Managers, Project Managers, BA, etc)? At least once per year I obtain this feedback from my peers. Perhaps respondents might not do so if their team culture is not solid, I don’t know.

In terms of actual certification, I think something like CBAP (for business analysis / systems engineering) would be a good model.

3) Respondent Profile & the influence of the report due to their Methodology – I couldn’t relate to the average team size being up to 25. In my 15 year testing career spanning small and large organisations, I can recall only two occasions where test teams were this big (let alone the whole project team), however they were also subdivided and very independent of each other. Therefore I was slightly concerned that the report leaned towards the perspective of more strictly controlled / less-progressive teams running old methodologies. In fact, your Methodology section confirms that given that Waterfall appeared to be one of the most prominent methodologies used.

Team sizes vary dramatically, you can see some organisation at 1-5 members, the norm around 11-25, and also several in the 100+. The team size though refers to the overall testing team size for the organisation, and hasn't been reduced to project testing team size. Some of the 100+ organisation I know have over 500 testers, some at 700 - 750 testers.

When we looked at the Project Landscape you can see it is common to have 10 - 20 projects per annum. Also common was a 6 month project duration. This paints a rough picture that on average you are probably going to have a project with 2 or 3 testers. Larger projects with longer durations may had 5 - 10 perhaps. We didn't specifically ask though the average number of testers on a project. As you point out that could change a lot based on methodogy, archecture, team composition, etc.

I think the survey does have a bias towards "testing aware" companies, as we less likely to have contact with "testing ignorant" companies. Also as you point out, this is likely biased to larger organisations. I think in benchmarking yourself, you need to think in terms of benchmarking with the testing industry, which wouldn't include "testing ignorant/weak" organisations.

Waterfall is still most common, however Agile has been increasing dramatically over recent times. Still waterfall is prevalent in large teams/organisations, however we are finding the understanding is improving, and the barriers are coming down to adoption of Agile. What I still see in organisation is misconception of what constitutes "Agile", and hence why we tried to drill down into some of the Agile practices. We need to do this further again in the next survey.

Some other areas that didn’t appear to be covered which might add value in my opinion are:

1) Exploring more deeply into the skill sets of testing professionals. Examples:

a. Are they more closely aligned with the business side (in scope & analysis) or with the development / technical side of things (design, tools, infrastructure, etc)

b. What classifies an Automation or a Performance Test Specialist? Is it a Tester who can capture/playback and make basic adjustments to automation scripts, or are they at the other end of the spectrum and are fully competent in reading/writing in scripting and programming languages?

I think this relates to your point 2 above.

2) More detail about the value adding of test professionals (in addition to perceptions of testing) . Examples:

a. How quickly can they adapt to the job at hand?

b. What is their contribution to process improvements, finding important bugs, and identifying new features?

I think we should be able to look at some of these in future perceptions. We just need to be able to phrase it into a perception questions, and one that has qualified results, and one where the benchmark is likely to provide some comparable/actionable results that organisations can compare their own position against.

As discussed earlier, perhaps we need to split out a survey for individuals, and get their bottom up input into the testing approach, rather than the top down perception from the organisation view as well.

3) More information about how Test Managers estimate. Example:

a. If a Test Manager has a wealth of metrics at their disposal, what are the best ones to use when estimating for future projects, and what are their reasons?

Michael, is this something that you wanted surveyed, or something that needed some further discussion?

In my experience, people tend to use rules of thumb at the early stage, supported by % of project budget, and developer/tester ratios (2 metrics we did survey). That is when they are varying their budget, for many it is what you had last time. As more detail comes in they tend to move towards work breakdown, and that's when overheads must be factored in such as test management, test environments, etc. (again distribution of test effort metrics provide some guidance here). I guess we could survey whether people use other things such as function points (or test points, etc.), and other estimation approaches.

If you think we should survey this further, some example of what you would like to see would help us.

Good question. I probably have to think about this one in more detail, but basically, I am constantly challenging myself to get the perfect estimate. That is despite the scope creep, changing priorities, higher than expected bug rates, etc etc. My view is that these ‘unplanned’ events should always be expected and therefore planned. However this should be done with a proven technique rather than to simply double the time required (for example). I have detailed methods which I use and which consider these events, but I’m not sure if others do likewise particularly in organizations where they have to stick to the fixed number of deliverables or timeframes (even if quality is not up to scratch). In my group, we react according to the need and therefore the estimation process needs to have lots of options. Stressful? Yes. Good end result? Also yes!

Hence I’m after guidance which tells me that my estimation methods are good or not good (etc). From a mathematical perspective, I would think this is achievable.

4) Post Deployment and measurement of success - How accurate was testing implemented to a plan, what criterion was used (and by whom) to determine if the project was a success, and does this criterion ever change depending on the priorities? Example:

a. If the test plan could not be implemented accurately (due to other project issues out of our control) AND the project ran late (due to a larger number of bugs needing to be fixed) AND the client was very happy with the end result, is this considered a failure or a success? In my industry (safety critical, I would be happy with this result).

Yes, you point he is quite right. Completing a project on schedule is not the only measurement of success. Finding defects late in a project is always a huge frustration, and I have experienced it first hand where despite the testing evidence the project has steamrolled into production, in some cases costing 100's of millions of dollars. So success could be a cancelled project due to test evidence.

The Standish study doesn't reflect that point. I think IT portfolio management should be like share portfolio management. Some investments will not acheive the return, and better to abandon them then to stick with it.

I guess we (as an industry) need to think about measuring success more effectively. Quality ultimately needs to include the end user / customer perception. While testing we tend to focus on the pre-deployment metrics, it is the post deployment metrics which really count! It is like we are measuring inputs to quality, and we need to focus more on outputs of quality.

Here we could look at specific issues, like you suggest above. It might be worth understanding:

- projects delays caused by testing, did post-deployment quality improve. Ie. are testing results beneficial in influence project schedules

- do project plans effectively use testing results to measure plan milestone completion

- and so on

Something also for our benchmark team to focus on for 2012, and one where your suggestions are very welcome.

Is it time consumers get tough on software warranties?

2011-04-19T23:55:00.000-07:00

A recent Australian Federal Court ruling has imposed a $200k penalty on a hardware and software vendor for false and misleading warranty claims made to its customers.

Most software vendors have limited understanding of their obligations under the Trade Practices Act 1974 (section 52 and 53g). Section 53g states: "[A corporation shall not...] make a false or misleading representation concerning the existence, exclusion or effect of any condition, warranty, guarantee, right or remedy."

Indeed, there are many clauses under the Trade Practices Act that software vendors should be aware of in relation to software quality, such as:

Section 68 - Application of provisions not to be excluded or modified
"... purports to exclude, restrict or modify or has the effect of excluding, restricting or modifying: ... (c) any liability of the corporation for breach of a condition or warranty implied by such a provision; or ... is void."
Section 70 - Supply by description
"... there is an implied condition that the goods will correspond with the description..."
Section 72 - Implied undertakings as to quality or fitness
"... there is an implied condition that the goods supplied under the contract for the supply of the goods are of merchantable quality..."
Section 74D - Actions in respect of goods of unmerchantable quality
"... the consumer suffers loss or damage by reason that the goods are not of merchantable quality; the corporation is liable to compensate the consumer or that other person for the loss or damage and the consumer or that other person may recover the amount of the compensation by action against the corporation in a court of competent jurisdiction."

"Goods of any kind are of merchantable quality within the meaning of this section if they are as fit for the purpose or purposes for which goods of that kind are commonly bought as it is reasonable to expect having regard to:
(a) any description applied to the goods by the corporation;
(b) the price received by the corporation for the goods (if relevant); and
(c) all the other relevant circumstances."

So what constitutes "merchantable quality" and "reasonable to expect" in the software industry? What testing prior to deployment is reasonable to expect? How do we ensure that the "goods will correspond with the description"? What will be the outcome when some of these clauses are tested in court?

I am not a laywer, but I believe that many vendors are likely short of their responsibilities in these areas. However, for many consumers and acquirers, they are not prepared to legally challenge for their entitlement to quality.

Bridging the Software Testing Skills Gap

2011-04-03T21:52:00.000-07:00

As the Australian economy improves, and pressure to obtain skilled resources grows, the pressure to find skilled software testing resources will be even greater. This is the most critical challenge currently facing Test Managers in large organisations throughout Australia. Ultimately the ripple effects will be felt by IT and Project Managers, and up to the CIO’s software delivery pipelines, as inadequate access to testing resources will place successful releases at risk (see Testing is IT's Elephant in the Room).

In the Ross Report the evidence collected led me to conclude that Australia will need to find 1500 – 2000 new testers to fill the gap. The reasoning for this is that not only will IT projects continue to ramp up, but the IT project budget apportioned to testing is continuing to grow, and this translates into a significant increase in testing expenditure by 2012. Evidence for this can also be seen in the ramp up in job vacancies, as shown in our Software Testing and QA Job Index.

Sourcing of testing resources is changing, with a shift towards increased outsourcing. Furthermore, testing is becoming increasingly analytical, where testers are now expected to provide technology advice at a similar skill level to that of others in the development team, such as programmers, business analysts and solution architects.

The implications of the Skills Gap are two fold:

Testing resources will become a suppliers market. Salaries will increase, there will be greater competition for available good resources, bidding wars will take place on experienced resources, and organisations will face greater churn as resources leave when tempted by other opportunities.

Shortage of resources will limit what project demands can be met. Projects may be delayed, or released with less than desired governance and quality control. Organisations will face greater risk.

On April 14^th and 15^th, Australia’s leading Test Managers will meet for the 9^th Australian Test Managers Forum on the Gold Coast. At last year’s Forum, attendee feedback confirmed resourcing as their most critical challenge, further reinforcing the evidence above. Hence the key focus for this year’s forum on the Skills Gap, and what organisations are doing to address the challenge.

This article explores the considerations, and seeks feedback as to what organisations are doing to bridge the Skills Gap. A number of strategy questions that organisations may be considering are:

Sourcing Strategy

Where will your resources come from?
Whether to use permanent, contract or outsourced resources?
Whether to undertake testing onshore, offshore or a blend of both?
Will you rely on a constant permanent team, or do you need to be flexible with your resource demands?
Are resources full-time or part-time?
What supplier relationships need to be established to secure access to resources when needed?

Attracting Resources

Where can we find new testing resources?
Do we look at immigration to access a greater resource pool?
Is testing seen as an attractive career path to potential recruits?
Are Australian universities producing graduates to address the testing skills gap?
Are resources from other parts of the business being reskilled successfully to address the skills gap?
Are resources from other parts of development being reskilled successfully to address the skills gap?
Are graduate, intern or traineeship programmes being used as a mechanism to develop new testers?

Retaining Resources

Are organisations able to offer the right environment to retain testers?
Are salaries competitive?
Are career paths in place, and do testers see advancement and personal development opportunities?
Do testers have opportunities for promotion without having to leave the discipline?
Is testing a good pathway for developing leadership in other areas of the business?
Are appropriate skills development programmes in place to continue to advance skills and provide greater value?
What is the best way to develop skills: classroom or online training, mentoring, project placement, etc.?
Are organisations providing a pleasant workplace environment?
Are testers recognised and respected by their peers?
Are team structures supportive of the testing role?

Reducing the Workload Demand

What efficiency improvement can be undertaken to reduce the requirement for skilled testing resources, yet still produce a quality outcome?
Are team structures well balanced?
What level of testing needs to be undertaken by independent testing?
Are processes efficient, in terms of most cost-effectively finding and fixing defects, using the right skills and resources, at the right time?
Can less testing be done and still achieve the successful outcome, e.g. through prioritisation and risk-based testing?
Can Test Tools be better utilised to reduce the workload?
Can scheduling the development lifecycle manage the workload, in terms of spreading testing more evenly?
Can we achieve sufficient reuse of test assets to improve efficiency?

A lot of tough questions here! Please post comments regarding your thoughts on what organisations are currently or are planning to do to address the skills gap. I will take your comments to the Forum for discussion, and also post a follow up article here after the Forum on what alternatives were suggested.

Bank error reveals Student's Ebay Fraud

2011-03-01T22:11:00.000-08:00

As reported by Computerworld, there are times when IT failures do some good. In Brisbane a schoolboy's Ebay scams were uncovered only after a glitch incorrectly deposited $2 million into one of the fake scam bank accounts.

Virgin Glitches Again, But No Queues

2011-02-15T20:03:00.000-08:00

Virgin has again had glitches according to Computerworld, but this time no long queues as the system was down only for a few minutes.

Last time failures resulted in massive delays across the country.

Payment problems at the NAB for 2 days

2010-11-25T17:35:00.000-08:00

For 2 days the National Australia Bank has had problems processing customer payments and transactions, causing delays to EFTPOS, ATM and other electronic payments.

Source: http://www.dynamicbusiness.com.au/articles/articles-finance-cash-flow/nab-electronic-payment-problems-eftpos-update-2248.html

A follow up similar failure occurred on the 8th Dec but for only about 1 hour of service disruption. See http://www.computerworld.com.au/article/370720/nab_electronic_banking_woes_strike_again/?eid=-6787&uid=19517

Certification is failing, do we need Professional Accreditation?

2010-10-22T17:06:00.000-07:00

Recently a question was posed by a colleague "Certification as a concept has lost it's meaning... The future as I see it, in line with other professions is not only to be certified, but accredited by some national body".

With volunteer lifesaving though Surf Lifesaving Australia, for the bronze medallion, for instance:

1. You must demonstate pool swim proficiency (400m in less than 9 minutes)

2. You must complete the training course (about 5 days)

3. You must pass the written exam

4. You must be able to do a run, swim (400m ocean swim), run within time limit

5. You must be assessed for practical components (rescues, lifts, resuscitation, first aid, observation, radio)

Then each year you have to redemonstrate proficiency, including:

1. Above again

2. Recap on changes to procedures (about 1 - 2 days)

3. Above - cut down written proficiency exam

4. Above again

5. Above again (criteria selected by assessors)

And you must maintain a minimum number of patrol (or water safety) volunteer hours per year.

Last year I did effectively 5 days of training. This year for proficiency, it probably took me another 2 - 3 days. The practical assessment alone was 0.5 days.

It is a lot for doing around 50 volunteer hours per annum, but they are very professional in their approach, and one reason why ultimately in our own company culture we aspire to Surf Lifesaving Australia's values. Many of them are skills that I may never have to use, but when I do need them I am confident I will get 90% of it right, and that will make a difference. The other key is you work as a team, and that way you support and remind each other on learned skills, they focus a lot on communication. It comes out in assessment, as they encourage team work in assessment, and look for the communication and reminding each other of procedures. That should be a goal in our work also.

So what is reasonable for someone doing this 1000 - 1500 hours per annum in their profession. Obviously if you are doing it everyday, the training and assessment is quite different. Is revisiting 5 days of training and assessment each year appropriate? This would exhaust the average training budget industry currently provide (see our industry benchmark).

So why can we expect such a high level of training in lifesaving (which is based on volunteers), but in a professional discipline, which also has some significant impact on people's lives and livelihood, skills expectations are not at a similar level.

I think the training provide by SLSA is great. I find the time commitment a big challenge though. And sometimes the reasoning on procedure changes is confusing.

So yes, I agree about stepping up to accreditation. Foundation level certification being the current bar is way to low, and the way much of the assessment is done is just pathetic. The outcomes are disappointing. I am certain with multi-choice exams done online, I would guess that around 30% of answers come not from the assessee, but from some other coach. I have seen this first hand with other multi-choice proficiency exams in other disciplines.

Certification is a step in the right direction, but failing to improve skills and competency outcomes, means that our best intentions may ultimately be threatened, and certification will be seen as a farce and a failure, and our intentions are reversed.

How to get influence???

It needs to be influenced via an industry body like the ACS, or a National Training Authority. Ultimately in needs commitment from employers that it makes a difference to the outcome and productivity.

I think there is a long way to go...

Faulty Speed Cameras Suspended

2010-10-18T00:46:00.000-07:00

According to Computerworld, faulty point-to-point speed cameras in Victoria have been suspended after a fault was detected which recorded incorrect readings on nine occasions.

Deputy Commissioner Ken Lay said the cameras would not be switched back on until they had been thoroughly tested by independent assessors.

Upgrades Cause Delay In Internet Banking

2010-09-26T01:41:00.000-07:00

According to Computerworld, Suncorp has admitted that recent upgrades to internet banking has caused delays, while some customers have complained of loss of functionality.

Virgin Glitch Strands Passengers

2010-09-26T01:35:00.000-07:00

According to ABC News, a crash of the check-in system at Virgin Blue has stranded thousands of passengers around the country.

Maritime Software Sinks

2010-09-07T04:58:00.000-07:00

Customer Service Officers claim that a new registration software system for NSW boat owners is a failure (as reported in CIO).

The software has failed to send out letters for license renewals, sent out incorrect expiry letters, and registration processing time has grown from 5 minutes to 30 minutes.

Jonathan "Peli" de Halleux

2010-09-01T19:28:00.000-07:00

We have had the pleasure of Jonathan "Peli" de Halleux visit from Microsoft Research visit us as keynote for the Test Automation Workshop and a presentation to the ACS Special Interest Group on Software Testing in Brisbane.

Peli presented work he and his colleagues have undertaken on Pex and Moles. Pex and Moles are tools which support Unit Testing and Stubbing in the .NET applications.

Pex will generate Unit Tests directly from the code under test. Moles is used to dynamically stub out interfaces. Both these tools were quite impressive, and will truly accelerate unit testing activities.

You can try out Pex online in a virtual environment (no installs!) at http://www.pexforfun.com/, otherwise you can use the full version in Visual Studio.

How to be "Better" as a professional

2010-04-03T01:32:00.000-07:00

I have just read "Better", by Atul Gawande. In his book, Gawande describes examples of surgeon's endeavors to improve their practice. The book is highly engaging, provoking thought with real-world examples and human compassion.

I have taken a strong interest in E-Health in recent years, and I see myself over the next decade or so seeing where my work in Quality Assurance can be applied to the Healthcare sector to reduce adverse outcomes. I can see elements of Total Quality Management and Demming applied in a clinical setting. Indeed, Gawande ponders the question "is medicine a craft or an industry?". Again a question the IT profession is asking of itself over the past couple of decades.

I thoroughly recommend this book as reading to those that are aiming to improve professional practice. Even though dealing with a domain which I have had no clinical experience, only observations from one of IT enablement, I found this book inspirational.

In Gewande's afterword, he pondered on how a surgeon may make a difference. His five propositions I believe are aspirations for any professional. Here they are :

1. Ask an unscripted question

Stepping outside of the professional script, both with the patient and colleagues. Here Gewande see the value of the human relationship to build upon the professional relationship. After nearly 13 years in business I have customers that I count as close personal friends. Professional service requires trust, and through building the relationship and empathy, we become a little bit more reliant on each other to step up when needed.

2. Don't complain

Gewande points out we all have something to complain about, but doing so doesn't actually advance us, and it depresses us and the team. Look toward the positive. Look for what we can do to prevent these problems in the future.

3. Count something

In this book, Gewande gave numerous example where measurement of individual and group practice had led to revolutionary advances. Examples he cited were Battlefield Surgeons in Iraq, Agpar scoring in Obstetrics, and comparison of Cystic Fibrosis centres across the US.

This idea I reflected upon a lot while reading this book. Indeed I am keen for us to measure our practice. This insight in most cases doesn't give me the answers, it leads me to ask more questions, but in the process I think we understand more of what is going on. Indeed I often quote Lord Kelvin, "when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind".

Gewande discussed how the Cystic Fibrosis Foundation is publishing the performance of individual centres, and how this may appear controversial, yet at the same time essential to get us to do better and become a "positive deviant". I reflect on how this is recently undertaken here in Australia with schools via the MySchool website.

Recently we have commenced the Software Testing Industry Benchmark, which aims to measure Software Testing practice across organisations. I think this is essential so that we understand our performance in relation to others. However we need to go further in our own company, measuring projects and individuals, not as criticism or punishment, but so that we strive to improve.

4. Write something
Gewande says this is how we contribute, and become part of a larger community. I agree. I found that contribution is a duty to give back to the community, to the profession. Whether it be a blog like this, a newsletter article, a quick presentation to colleagues at lunch, or a short idea to the staff mailing list. I found in laying down those thoughts, it also helps me to better understand what I am proposing and in the process gain greater insight.

5. Change

Here Gewande points to three adopter profiles: early adopters, late adopters and skeptics. [For more discussion of adoption profiles see also "Crossing the Chasm", by Geoffrey Moore]. He says we should seek to be early adopters, be willing to recognise the inadequacies in what you do and to seek out solutions.

The above five points I think are useful concepts for us also as Software Testing professionals to take into consideration. Certainly attributes that personally I will aspire to.

Business doesn’t see value in $900 million spend on software testing

2010-03-24T17:53:00.000-07:00

Industry research recently released has shown despite significant and increasing proportions of IT budget being assigned to software testing and quality, businesses are not convinced of the investment value.

Dr Kelvin Ross, CEO of KJ Ross & Associates who conducted the Software Testing Industry Benchmark study said: “Part of the problem is the expectation that IT should just get it right. Why should you have to test if development just built it properly in the first place.”

Spending on software testing has increased to 22% of IT project spend. Annually this equates to an Australian enterprise spend estimated at over $900 million.

The proportion of IT project spending on testing will continue to increase. Higher reliability expectations, greater integration, larger projects and increased complexity are putting greater pressure on software quality, and more effort is required to check the software is right.

While Test Managers are always wanting to spend more on quality, they perceive Project Managers and IT Executives view the current spend as about right. Surprisingly though, greatest pressure to reduce software testing spending comes from the business. Over 50% of respondents indicated that business considers spending on testing too high.

The industry benchmark found business knowledge was amongst the most sought after skill in the testing team. Yet 30% of test managers found customers and business users are not happy about being involved in testing. It remains a major challenge to develop and retain the business skills in the IT testing team.

The research further showed over 20% of all defects found in testing were problems traced back to poor requirements. Yet these requirements problems are found very late in the project, often causing rework, slippages and delays.

“The key to improving software quality outcomes is engagement of the business”, said Dr. Ross.

“First, the business must be informed software development is complex, and failures are inevitable. Then the business needs to work with testing and quality more closely to accurately define the user expectations, and trap failures early, thereby reducing downstream costs and project delays.”

The Software Testing Industry Benchmark study provides a detailed analysis of software quality and testing practices within Australian organisations, and covers aspects such as stakeholder perception, budgets, resourcing strategies and costs, adoption of tools and techniques, defect discovery profiles, and key performance measures.

Further information:

Request a detailed white paper discussing the above issues
Test Manager Forum (TMF) 2010 will include more detail investigation of Business Engagement
Participate in the Software Testing Industry Benchmark to receive a complimentary report
Feeling Tested? Find out about other test management information events to get organised and manage testing

400,000 Prius Recalled to Patch ABS Software

2010-02-09T03:36:00.001-08:00

Toyota have officially recalled almost 400,000 Toyota Prius and Lexus HS250h vehicles.

An update to the ABS software is required to fix a problem where brakes have been reported to momentarily stop working as the braking system switches between regenerative and conventional braking.

The fault has been attributed to several accidents, but no deaths.

Toyota have always been solid reputation for quality, showing leadership in the Automotive sector. They were the pioneers of Lean Manufacturing and Just-in-time manufacturing processes.

However this recall follows two other recalls of 8 million vehicles (for slipping floormats and sticking accelerator pedals), and has raised a media frenzy.

Further links:

MySchool strained under the load

2010-01-28T01:41:00.000-08:00

It has been pretty much headline news all day today here, MySchool, which publishes comparative ratings for all Australia schools, has been hit by load issues as it went live today.

It was highly publicised, following the controversial debate regarding school performance comparisons using National Assessment Program Literacy and Numeracy (NAPLAN) test results.

The site suffered availability issues after go live as the public started to pound the application. While the site was designed 1.75 million hits per day, the site was reported to experience 2.5 million hits between 1:00am and 10:00am this morning.

Bank of Queensland's EFTPOS's role over to 2016 rather than 2010

2010-01-04T21:59:00.000-08:00

Thanks for pointing to this one as well Alistair (http://watirmelon.com/2010/01/03/software-quality-metrics/).

News.com.au reported Bank of Queensland EFTPOS machines rolled from to 01 Jan 2016 rather than 01 Jan 2010. Retailers reported losses as unable to process electronic transactions.

As one commentor noted - it is a Y2.01K bug!

Black Screen of Death

2009-12-02T15:46:00.000-08:00

While the "blue screen of death" has made it into the IT failure and quality vocabulary, could we see a new addition - a colour change perhaps!

In Computerworld article, "Microsoft denies blame for black screens of death" (2 Dec 2009). User have claimed that November windows updates have caused lock outs of users' PCs.

Microsoft have denied claims, and investigation of failure reports have found that none of the behaviour is related to that its November Windows updates are causing a widespread "black screen" lock-out of users' PCs. Furthermore technical support have not seen this as a broad customer issue.

Microsoft have some extraordinary approaches to configuration and patch testing. It is a huge challenge when you consider the miriad of configurations and applications dependencies that exist. In the whole scheme of things I think they have done a good job in recent years, and a testament that we dont see these kinds of problems proliferating every day.

While most of us have experienced the "blue screen of death" and enjoyed the quality awareness that this colourful metaphor has raised. From the article, it sounds like a widespread mutant of the term is not yet ready to surface???

Computer glitch delays Qantas flights

2009-11-17T00:17:00.000-08:00

I thought I should find a place to log interesting software failures. So here is the first one, which hit the news last night and this morning.

"Computer glitch delays Qantas flights"

http://au.news.yahoo.com/thewest/a/-/breaking/6480146/computer-glitch-delays-qantas-flights/

Amadeus (http://en.wikipedia.org/wiki/Amadeus_CRS), a reservation system, failed for multiple airlines, and Qantas reportedly had to switch to manual processes. This failure was reported to have affected many other airlines internationally.

Immature in Using Metrics to Support Investment in Software Testing

2009-07-21T11:22:00.000-07:00

In the current economic environment, one would think that we would be using all means to secure investment in software testing. However at a recent industry forum it became apparent to me that the level of maturity in using metrics is somewhat limited.

At a recent ACS Testing SIG meeting on metrics, we discussed what metrics could be used to support the business case for investment in testing. The meeting was largely an open forum discussion seeking opinions and contributions rather than a presentation.

Of the participants, many admitted to using defect metrics, but rarely used other metrics such as time and budget allocation, effort, coverage, and test outputs. This is somewhat disappointing to have a poor measure of the work that we undertake. It leaves us vulnerable to cutbacks, as other managers who may be more verbally gifted may take away our budget. Metrics provide a useful way of supporting our benefit.

The meeting went on to review the kinds of metrics that testers use. The following list provides some suggested metrics:

Defects

Type
Cost
Age
Status
proportion found in stage

Coverage

automated v manual
planned v executed
requirements
code

Effort, time, cost, schedule
Test outputs

test designed
test executed

Test inputs

size
complexity
developer hours
project budget

Risk

Some comments were made that metrics can be broadly grouped into two categories:

Efficiency - can we do more testing using less effort
Effectiveness - can we acheive a better outcome from our testing effort

While the discussion focused on supporting the business case for software testing in tighter economic times, it is important to note different uses of metrics. Metrics are used by test managers for the following:

Managing Progress - producing estimates, how complete is testing, how much more to go, ...
Quality Decisions - is the product good, bad or indifferent, are we ready for release, ...
Process Improvement - what are the areas that future improvement should target, how do we know our process has improved, ...
Business Value - what benefit has testing produced for the business in terms of reduced costs, increase revenue, reduced risk, ...

The choice of what metric to use can be daunting. It is not a good idea to collect everything, as you get overwhelmed in what data to use to make a management recommendation. It is worth looking at Goal-Question-Metric (GQM) promoted by Victor Basili to help choose appropriate metrics.

There was a lot of discussion that metrics assessing productivity can be dangerous. My personal view is that we should you use metrics to influence productivity, but that the following points need to be kept in mind:

Productivity or performance measures do change behaviour. People will start aligning behaviour towards meeting the metric target. A poorly chosen metric or target could create behaviour that you never intended.
One metric wont tell the correct story. Measuring productivity in terms of tests per hour completed, may mean that people run poor tests that are just quick to run. You may need several metrics to get a better perspective, for instance collecting information on defect yields, defect removal ratios from earlier phases and so on to get better picture.
Given that metrics will change behaviour, you may change your metrics from time-to-time to place emphasis on improving or changing other parts of your process or performance.
Metrics should be used by the manager to ask more questions, not a hard and fast rule to make a decision. A metric may lead you to make deeper enquiries with individual testers or developers.
Managers need to build trust with the team in how they us metrics. If the team don't trust how you will use the metrics the will likely subvert the metrics process.

When we were discussing using metrics to justify the business case for testing it is very easy to get caught up in the technical metric, that is important to the test manager. However, when discussing with other business stakeholders, you need to talk their language. You may need to explain the metric in terms of what it means in terms of cost saving or increased revenue for it to have its impact. Don't explain it in terms of how many more tests cases are executed per hour, instead explain it is $'s per hour saved. Other business may need it explain in other terms, such as risk reduction or satisfying compliance.

It appears that we still have a way to go with software testing metrics. As a profession we need some clarity about how as Test Managers we can use metrics to gain support, and provide evidence of our benefit. Lets hope that as we mature we will start stepping up to sell our return-on-investment more effectively.

Certification - Should be more than a piece of paper

2009-05-14T04:22:00.000-07:00

I just saw a link to "Expert Software Testing Certification" for only US$9.95. http://softwaretestingwinners.blogspot.com/2009/05/expert-rating-certification-in-software.html

It cracked me up!!! Will 40 multiple choice questions ensure that I am an expert?

There is a real danger with certification becoming meaningless, and it WILL damage our profession. The key to a profession I believe includes some of the following elements:

Common terminology and standards
Shared body of knowledge
Competency development programmes and recognition schemes (e.g. certification)
Representative bodies
Recognition by employers and legislators

Yes, I created a Certification program a long time ago and we continue to offer it widely (http://www.kjross.com.au/page/Training/Certified_Software_Test_Professional/). And I believe Certification can add value, but if Certification doesn't help to develop and assess competency then it will be doomed. If people become certified and they are not competent, then the recognition of the profession will be damaged.

I think focusing purely on the exam is a big problem. For individuals this is the main driver, they want the piece of paper! I remember as a university student, the endless swotting & cramming, and many students not caring about the subject they were studying, avoiding lectures, and doing whatever it took to pass the exam and get the credit.

For the most part, participants in our certification are funded by the employer. For the employer, they want to see skills development and recognition. If their employees sit in a training course, they want them coming out being more competent. They want to ensure that the certification process ensures that recipients exhibits and develop competency in specific areas.

The exam should be considered a secondary goal, just an assessment method that measures competency. Yet for many it is the primary goal. The primary goal should be competency development.

I dont like multiple choice exams, but any exam on its own is not enough. There needs to be multiple methods to assess competency, this should include:

participation in practical exercises, understanding and demonstrating skills
demonstration of workplace competency
proof of understanding through completion of individual assignments
engagement with experts where they can demonstrate understanding of key concepts

It is costly to build these into a certification programme, but that should be the goal. We need to make sure people dont slip through, those that do will devalue the certification, and others that have completed it.

It really frustrates me when I hear people say "let's do the 1 day course and then sit the exam". I would be happy if they felt they were already competent, and they just wanted the recognition. But for most, they do not have the competency in the specified areas, and they are just looking for the fastest way to swot for the piece of paper. Will they be competent???

SOA Challenges

2009-04-30T19:08:00.001-07:00

Service Oriented Architecture (SOA) places signficant priority on software quality and testing.

I just read a vendor whitepaper about SOA. The paper emphasised a number of points about SOA and software quality, as indicated by the quote:

"Developing an SOA that guarantees service performance, scalable throughput, high availability, and reliability is both a critical imperative and a huge challenge for today’s large enterprises."

This whitepaper from Oracle confirms for me my experience that SOA is placing greater emphasis on the importance of software quality and testing.

Testing becomes the pivotal role in integration activity where SOA is used to glue together larger systems. It is at testing where we see all the components and the fabric that binds them come together into a working solution.

Not only must testing support function, end-to-end, and acceptance testing of the SOA integration, but as this paper highlights other quality attributes are critical to the success of SOA:

Performance
Scalability
High availability
Reliability

These are difficult areas to test and evaluate particularly where SOA is integrating many and large separate applications.

My experience on testing projects involving SOA is:

Building end-to-end test environments is difficult, and we must use stubbing and harness to contain the scope of testing.
Performance testing is a must, as SOA communication between applications impact user response times significantly.
SOA is distributed, which means vulnerability to failure of a single component can bring down other interdependent systems and applications.
Performance monitoring should run tests of user experience periodically. Many of these tests can be drawn from automated tests used for performance or functional testing.
Testing is more technical, as tests are constructed around SOA messages. Tool support (either commercial or open-source) is essential. Frequently manual / business testers aren't suitable in this areas, as there is no user interface, and instead tests must be coded to send and retrieve messages.
Asynchronous behaviour of messaging makes functional testing more complex. We must deal with messages coming back in different orders, it is not always send then receive, other messages may be interleaved.

When adopting SOA, testing teams will need to evolve their strategy regarding how the evaluate and test. New methodology and technology will need to be incorporated to address the points above.

7th Australian Test Managers Forum

2009-04-25T07:32:00.000-07:00

The 7th Australian Test Managers Forum concluded on Friday 24 April.

Numbers were down on previous years. Previous years have always been a sellout 5 or 6 weeks before the event. This was clearly a sign of the current economic conditions, with most frequent attendees citing tight travel, training and conference budgets, as well as reluctance on staff to be out of the office in uncertain times. However we had pretty close to the same number of representative companies, just less multiple attendees. Nevertheless, the interaction was the same as in the past, and a rewarding time was had by all.

Particularly enjoyable for me each year is the ideas exchange with people who are equally passionate. What I like best is the information interchange and sharing of what has and hasn't worked for each other.

The most critical challenges this year (from the participat challenges survey) :

Requirements - still number 1 from last year. We often experience poor quality requirements leading into testing, and no doubt the same is happening for development. Testing needs to work more in the requirements phase to have clear expectations on what is being tested. Similar our role in this phase means that we test the requirements specifications and find defects earlier.
Return-On-Investment: the global economic crisis is putting more pressure on us to demonstrate business value and to justify costs. With ROI, if we dont show the "R" (the return to business), all they see is the "I" (investment) as a "C" (cost).
Environments: with the introduction of larger integration and technologies such as SOA, environment management is becoming more complex and downtime wastes testing effort. While virtualisation (like VMWare and others) is changing test environment management significantly, there is still significant challenges in specifying and controlling testing environments.
Early involvment: the biggest cost savings that testing will make will come from working earlier in the lifecycle. Yet it is a challenge for test teams to get engagement in early phases. We need to use test results to show defects coming out of early phases, such as requirements bugs, then offer more cost effective ways that testing can find these bugs, e.g. through requirements and specification evaluation (such as inspections, modelling, walkthroughs).
People: while the pressure has come off staffing with the economic downturn, selecting the right candidate, and having succession plans from existing staff leaving remains a critical issue. Here focus on skills definition, evaluation and staff development techniques are required.
Scheduling: the schedule squeeze on testing at the end of the lifecycle remains. Development still slips, yet the release date doesn't. Many organisations discussed the greater move towards agile, which has other challenges for testing, to help alleviate this. Also discussed we techniques are sourcing strategies, flexible resourcing, and risk-based prioritisation of testing.

The were other challenges discussed, but the survey showed these as not as critical to the audience as a whole:

Resourcing
Estimation
Automation
Process Improvement
Methodology
Coordination
Stakeholder Engagement
Governance
Agile
Release Management

Much of the information, surveys and presentations will be collated and progressively added to the website at http://www.kjross.com.au/page/News_and_Events/Australian_Test_Managers_Forum/Test_Managers_Forum_2009_