Virtual QA Tools

Functional Test Planning: An Overview

2008-09-01T01:54:00.000-04:00

Keep this simple and to the point and you have a guaranteed winner. All too often people get bogged down in detail when making a test plan, I suggest having an overview to eliminate this pitfall. Don't skip the plan, although that is what normally happens when one feels bogged down or overwhelmed. Planning reduces test development time and increases efficiency and coverage. Planning also helps you identify standards and techniques to use when using automation, such as initial and end conditions.

Here is a little checklist that will help you to keep this simple and cover all your bases in your test plan:

Identify the Project Resources. The types of resources you want to identify are subject matter experts (SME), project managers, team leads, functional experts, database experts, other stakeholders (business analysts, etc.).
Identify and Document the Application Under Test. Focus on the application that will be tested. Some examples might be: IE, Firefox and Netscape for the web
Identify application details that may be relevant to your test (hardware, software,
client version) and any exceptions (will test on both XP and W2K, etc.).
Identify Scope of Business Processes. Evaluate business processes with respect to business requirements. Ask the questions: Who writes the test cases? How can transactions be linked together? Do they require specific data? Is data passed from one transaction to the next? How much data is required for each process? Document all steps in the business processes that will be automated. This is the point at which the test cases are created. Each test case should include all the user’s steps, sample data values, expected results for each step, and any checkpoints for verification.
Manage Required Data. Identify where the source for valid data will be, who knows what data to use. This step focuses on data management. Data requirements are documented and required data is collected for each test case. Ensure that all data to be used for testing is valid and will not cause any errors during test execution.
Results, Defects, and Changes. Establish a baseline value against which to compare test results. In order to determine what results are to be expected, you need to be able to compare the results of the test with a baseline value that will determine if the system passed or failed the test. These values should be known and documented at this step.
Organize a system for logging test results and each problems you encounter. Monitor the progress of finding and correcting problems in the system. As a tester, you may need to know where to store tests, now use appropriate naming conventions, document results and defects, and schedule test runs.
Document the Final Test Plan. The contents of the test plan should include the schedule, objectives of the test, what’s in and out of scope for the test, functional testing environment, defect tracking, change management system.
Build The Test Lab and Create Tests. Be sure to include caveats and processes as well as provide this report to those stakeholders who you have identified as those needing or wanting this information. Support of the business and stakeholders is critical to your test, keep everyone in the loop who wants and/or needs to be there.

Getting Started with your Virtual QA Lab

2008-07-12T16:05:00.001-04:00

You love the idea of a virtual QA Lab but wonder how to begin. At the beginning, as with any project, with the high level requirements and the business systems design review. Get your ideas in line with the needs of the business.

Define and Prepare

The solution must be designed to provide stable services to the Quality Assurance and Development Departments by implementing network servers and updated software purchased from approved vendors. This will have direct impact on reducing business risk, increasing availability of the applications in line with current projections for the QA and Development Departments and compliance with SoX audit trail requirements. Not worried about SoX? Someone in your organization will be concerned with audits, be is SoX or something else. Having an audit trail protects the business from anything that may occur in the future so plan for it now.

Anyone for Scope?

Gain a complete understanding of what exists today in your world. Do you have any single point of failures? Lack of failover systems? Operating systems that are not part of a security patching schedule and are therefore out of date posing a security risk. Have the needs and volume changed since the original equipment was put in place? What are those? How do you plan to address that?

The Virtual QA lab will aims to resolve all the outdated and growth issues now and in the future by upgrading hardware, implementing adequate security polices and adding support and training. Be sure your plan:

Specifies clearly the needs and offerings.
Is clear and focused to avoid misinterpretation.
Measureble to be quantified and compared to other data.
Attainable: achievable, reasonable, and credible under conditions expected.
Realistic: fits into the organization's constraints and is cost effective.
Timely.

Phase It!

This is not a simple undertaking for an organization that has not walked down this road before. Avoid the overwhelming feeling by phasing your plan. Make it simple and understandable to avoid the "this is too complex" thoughts your customers have in their minds before it hits their lips.

For example, in the initial phase, new hardware may be purchased and installed in an approved data center. New Application software may be installed and configured. Be sure to include assets and how they will be accommodated.

Be absolutely certain to include savings, this is a critical part of this plan. Include those costs associated with exposure due to outdated and aged out hardware, as well as growth both immediate and in the future.

Get Specific

If you have done your homework up to know, you are fully aware of the pains within the QA organization and the organization overall. Address those pains in specific, real terms. What if you are new or trying to do this as an outside vendor with no specifics? Ask questions! Most clients are more than happy to talk about their pain. They need your help, after all, that is why they are talking to you in the first place. Don't miss any opportunities to learn about the pain and pleasure of the current systems and environments and address them.

Need Help?

Give me a call, an email or contact me in whatever way you feel most comfortable. I am very happy to help and am certain I can be of help to you in your plan. I have implemented this solution ins both large and small organizations and can customize this cost saving solution to fit any environment and company size.

LoadRunner 9.0 - A Review of What's New

2008-06-14T15:44:00.001-04:00

LoadRunner 9.0 is released, debugged and ready to sere the needs of performance testers. It's been out there since August, but had some "issues" that are reportedly fixed. So let's take a look and see what we get!

New Features

The best improvement in VUGen is the most basic and simple - we finally have a FIND feature!
The most notable in VuGen is the user defined Business Process Report in Microsoft Word format, that summarizes the VuGen script for various protocols including the infamous and troublesome Citrix_ICA. This works well, but text would have been a far choice since this is to be a template that is reusable and Word adds extra formatting codes you are going to have to hunt down and get rid of before you can actually use this.

In the Controller, a new scheduler is available with a GUI to actually see the firing off of Vusers to that schedule. This is a help for when you are in the process of taking down a server, you can show it visually to the impacted team with little explanation necessary. One less meeting is always good in testing.

New Functions

Five star new function, IMHO, is at long last, the support for RDP testing with a run time viewer! Now if we could only get the application itself to run with VNC and RDP it would be great!

For you Citrix fans, the new ctrx_execute_on_window function (replaces ctrx_set_exeption) provides for the use if wildcard characters in the window caption, less scripting for you!

The SLA Wizard gives you the opportunity to define performance metrics against established SLAs. Purely a business reporting tool that will help when it is analysis and reporting time, Loadrunner will go out and perform the measurements for you and report them on demand. A great time saver if you are in regression performance testing mode. Works well, I had no issue getting the metrics I asked for even when the SLAs were not possible (negative numbers).

Improved

Java support has been improved, we now have JMS (for Weblogic 8.1, Websphere MQ 6.0 and TIBCO EMS), JACADA and the ability to add custom as always.

Still needs work

The RMI support still needs work if we are going to use it without winsockets. I know the technology is old school, but we have tons of legacy apps alive and well that use this function.

Visual Studio integration is still not quite there. You will find some of the old errors and mis configurations still exist.

=============================================

Citrix

Citrix Vuser scripts emulate the Citrix ICA protocol communication between a Citrix client and server.

Improved script replay stability and predictability.

New bitmap synchronization replay error management compares bitmap synchronization errors as they occur, and adds the necessary changes to the the script in a single click.

Improved text trapping and text synchronization algorithm, including showing the synchronization area in a snapshot.

Improved tree view, including replay snapshots and additional options in the context menu.

New ctrx_execute_on_window function (replaces ctrx_set_exeption) enables usage of wildcard characters (*) in the window caption.

Supports Metaframe server 3.0 and Metaframe server 4.0

Supports Citrix client 9.x and 10.0

WAP

Supports recording using the new multi-protocol mechanism.

Records any type of WAP application/simulator.

VuGen automatically recognizes the application/simulator settings and uses them during recording.

Supports WSP, HTTP proxy, and HTTP direct modes as configured in the application/simulator.

Winsocket

Added support for SSL and UDP.

Mailing Services

Added SSL support for LDAP, FTP, POP3, SMTP & IMAP protocols.

Microsoft Visual Studio 2005 Add-In

Full support for creating, replaying, and debugging scripts from Microsoft Visual Studio 2005 - C#, VB.NET or C++.

Debunking The Cost Of Quality And Productivity

2008-05-23T01:35:00.000-04:00

I hear an awful about the cost of quality and how it impacts production in such negative terms. I thought maybe it's time to debunk the myth that investing in quality negatively impacts production.

Quantify This

the cost of quality is not defined as the expenses in creating, deploying and supporting a quality system as most think. It is rather the cost of not adopting a program. How much does it cost you to loose a contract for low quality? What does that do to your credibility? How does your bottom line suffer from your inability to gain more customers and secure more contracts? Actions speak a lot louder than words, as do bottom line realizations.

So, how much does it really cost?

It is a well know fact that poor quality costs the average business 15 to 30% because its product or service was not done right in the first place. So what is in the 15-30%? Rework, returns and big dollars. Is it worth the risk of loosing 15 or 30% for low quality? I am sure each of you can find something better to spend that money on.

1-10-100

So how much does a quality program really cost? An accurate rule of thumb suggests that a quality problem costing $100 to resolve in the field would cost only $10 to correct if discovered during design review and merely $1 to prevent in the first place. Surely, your reputation and bottom line are worth a dollar. There are four basic areas that will need your attention in developing your cost of implementing quality are Prevention, Inspection or Appraisal, Internal failure and the dreaded External failure cost. That makes them about a quarter each, a bargain at twice the price!

Investment

You have invested in the best machines, hired the brightest and smartest programmers in the industry to work on what you believe, it the best product or service around. I know you believe that or you would not get up every morning to go in to work. With all your dedication, hard work and effort, your investments are depreciating by the hour and being over utilized to clean up the mistakes made. This is a vicious cycle that feeds itself, the harder you work, the more you push, the deeper you get. Would you like a return on this investment? Quality programs actually appreciate with time. Your initial dollar ends up being two dollars in a year!

Lessons Learned

In conclusion, it is fair to say that we have witnessed quite a few failures recently due to poor quality programs. Had Mattel implemented quality standards into their contract with the firm who manufactured the toys that are responsible for the death of children with high lead levels, their stock would be riding high. Imagine if Topps had done better inspections of the beef and internal process checks, they might still be in business today instead of bankruptcy court. The cost of poor quality is high on many levels. It's worth a dollar.

LEAN Software Quality Assurance

2008-04-01T16:46:00.002-04:00

Quality Assurance is perceived as an extra, as overhead and generally, a cost center. To some extent, this is true. It is an investment in the long term and quality of your product or service in where the value can come back to you in any number of ways and most are not immediate or instant. Let's take a look at applying LEAN principles to Quality Assurance and see if we can improve it just a little. [more]

Ten Ways to Improve Quality Productivity

2008-03-19T12:09:00.007-04:00

Have a "Rock Star" QA Team with some of these suggestions to get your started.

Agile and Free to Learn Tools in the Dot Net Arena

2008-02-29T13:19:00.001-05:00

Agile development and test have finally arrived in the .dotNet programming environment, just as the open source movement has entered into the commercial development of Microsoft and some other top tier software organizations. Typically, dotNet has been solely a development tool with very few quality tools to speak of in any real terms. Quality teams have been almost isolated and collaboration between development and quality assurance are tenuous at best, which is quite the standard in traditional development environments. There is now an opportunity emerging with the launch of Visual Studio 8.0 to a more agile and cooperative team environment.

Development tools for building both desktop and team-based enterprise Web applications are standard for this and many other IDE’s on the market. What this release has to offer is providing the testers the ability to collaborate and fine tune those applications in real and measurable terms. Testers have available to them a complete studio of tools in the Test Edition of Visual Studio 8.0 that are integrated into the development environment including unit, Web, load, manual and code coverage tests. In the past, the use of the Web Application Stress Tool was the only opportunity in this space for configurable load and performance testing of applications. While this tool was effective, it did take a fair amount of expertise in the tool itself as well as load testing in general. This was time consuming and did not lend itself to rapid development and as such, was relegated to the luxury bin and rarely performed well, if at all. Much of the guesswork and pain of learning a new tool is alleviated in a familiar interface and standard toolbars in this tool.

The challenges facing quality assurance in the testing of web based applications has been addressed with the advent of the WebTesting namespace that provides specific classes to enable Web testing. The base class for all web tests is the WebTest class, which is available out-of-the-box as are the classes WebTestRequest and WebTestResponse classes for simulating HTTP requests and responses. Gone are the days of hand coding tests for http responses and posts, this is now a point and click operation with an expert view if you feel so inclined.

Building and testing high-performing desktop applications in a simple team-based design enables swift and high quality applications and deployment of enterprise solutions. Add the ability to communicate and provide to the development team valid and reproducible errors seamlessly and you have the pure divinity of agile. The opportunity to learn the functionality of this tool are available at no cost on the educational portion of the developer network site called channel 8, also known as DreamSpark to students everywhere. You can download the tools below at no cost: Channel 8, DreamSpark, Test Center 90 day Trial Download.

Configuration Status Accounting

2007-12-13T13:45:00.000-05:00

Configuration Status Accounting of Application Software Configuration Items? What is that? Why do I need it? The Configuration Status Accounting system will provide for you a record of the following information for each application software configuration item (CI):

1. The planned and actual dates the:

application software specification was released;
allocated baseline was established;
application software Test Plan for the CI was released;
application software Test Plan for the CI was approved;
application software design document was released;
application software design document was approved;
application software CI testing took place;
application software test report for the CI was approved;
application software Product Baseline was established.

2. The Configuration Status Accounting system will record all change requests made to the application software specification (the allocated baseline):

the date of the change request;
the title of the change request;
the status of the change request;
the approval date of the change request; and
the planned and actual dates of change implementation.

This will ensure you have the latest and greatest version of your software within your archive and deployed to your customers. It also won't hurt you a bit if you have a SoX audit either :)

Software Configuration Managment: Configuration Identification of Software Configuration Items

2007-12-13T13:30:00.000-05:00

Some good Policies for Configuration Identification of Software Configuration Items will greatly improve the success of your software archive. Suggested policies include:

1. Each application software configuration item must have a specification document, the following are recommended for discussion:

Business Need/Requirements Statement
Software Requirements Specification
Software Design Description
Source Code/Executable Code
Test Plan Procedure
Software User Manual and Run Book
Training Plan
Software Product Specification

2. Application software specifications should be contained in separate documents.

3. Each application software specification will be baselined (as part of the Allocated Baseline) and subject to formal change control.

4. Application software specifications should be traceable to the system-level requirements specification.

5. The content of the application software specification establishes the acceptance criteria for that software configuration item.

6. Each application software specification will be numbered with the configuration items identifier as part of the document number, for example, the specification for application software configuration item XYZ Project will be XYZ Project-SPEC.

7. The naming convention used applicable to application software configuration items is to have a maximum of the first six characters be the CI number; for example, XYZ Project-OPEN-WINDOW links the open window module to the XYZ Project CI.

Agile Testing Types and Process

2007-12-13T12:33:00.000-05:00

Types of Testing and the steps to be followed in sequence to assure your customers receive a quality product. For example, performance testing can only be accomplished successfully AFTER functional testing has been performed and the software passes that step. To do so otherwise would result in functional errors delaying the success of performance testing

Step 1: Unit testing
Performed by the Developer. Unit test case design begins after a technical review approves the high level design. The unit test cases shall be designed to test the validity of the program's correctness. In other words each decision statement in the program shall take on a true value.

Step 2: Test Planning
Performed by QA. Review requirement specifications, architectural designs and use cases to develop test plans, test steps and test scenarios. Identify any gaps or issues that need to be addressed.

Step 3: Requirements Traceability / Testability
Performed by QA. Traceability is the ability to show how requirements are derived from higher level (or "parent") requirements. Conversely, traceability helps to identify all downward requirements derived from parent requirements.

Step 4: Integration Testing
Performed by QA. Integration testing proves that all areas of the system interface with each other correctly and that there are no gaps in the data flow. The final integration test proves that the system works as an integrated unit when all the fixes are complete.

Step 5: Build Verification / Smoke Test
Performed by QA. When a build has met completion criteria and is ready to be tested, the QA team runs an initial battery of basic tests to verify the build. If the build is not testable at all, then the QA rejects the build. If portions of the build are testable those are tested and the results documented.

Step 6: Functional Testing
Performed by QA. Functional testing assures that each element of the application meets the functional requirements of the business as outlined in the requirements document/functional brief, system design specification, and other functional documents produced.

Step 7: Performance, Load and Stress Testing
Performed by QA. Non-functional testing proves that the documented performance standards or requirements are met. Examples of testable standards include response time and compatibility with specified browsers and operating systems.

Step 8: Defect Fix Validation
Performed by QA. If any known defects or issues existed during development, QA tests specifically in those areas to validate the fixes implemented by the developers who have also unit tested these fixes prior to sending them over to QA for validation.

Step 9: Regression Testing
Performed by QA. Regression testing is performed after the release of each phase to ensure that there is no impact on previously released software. Regression testing cannot be conducted on the initial build because the test cases are taken from defects found in previous build in initial there will be nothing to test against.

Step 10: Error Management
Performed by QA. During the QA testing workflow, all defects will be reported using the error management workflow. Regular meetings will take place between QA, development, product and project management to discuss defects, priority of defects, and fixes.

Step 11: QA Reporting and Readiness Review Performed by QA. QA states the results of testing, reports outstanding defects/known issues, and makes a recommendation for release into production.

Step 12: Release Management
Performed by QA. Releasing software refers to the process of providing some named (or otherwise uniquely identified) files to others for use. The others may be your department at work, your classmates, or The World. Managing the release means you know, understand and can explain what went into it.

Step 13: User Acceptance Testing
Performed by All concerned Product Managers, Project Managers and a Customer Focus Group (if applicable) to verify the new system, data and software changes meet customer expectations and meet usability requirements.

Step 14: Release into production
Performed by QA & Project Team If the project team determines that the build is acceptable for production, the configuration/version management team will migrate the build into staging for the implementation team to move into the production environment.

Step 15: Post Implementation
Testing Performed by QA. Testing performed after the software has been deployed to ensure proper implementation.