Test Levels, Test Types, and Test Approaches in CTFL

The ISTQB-CTFL (Certified Tester Foundation Level) curriculum details 5 levels of testing, 4 test types, and 3 testing approaches.

The five levels of testing are as follows: Component Testing, Integration Testing, System Testing, User Acceptance Testing, and Maintenance Testing.  They are missing a needed sixth level, Smoke Testing. Perhaps one day they will add it.  Here is a description of all of them.

Smoke Testing: A quicks run through of all of the screens and reports within a system to ensure the developers gave you everything.  This ensures you don't get 2 days into testing a large app only to discover a ".dll not found" error due to the developers not including everything in the install package. Smoke testing should take now longer than 20 minutes or so.  Two added benefits once an organization has smoke test in place is that the implementation team can use the smoke test to test a new install, and if a production system goes down and is brought back online a smoke test and ensure all of it was brought back online.  But, I digress....

Component Testing: Checking each component of a system. There are 5 types of comports in software: 1. Screens, 2. Reports, 3. Interfaces, 4. Logic Modules, and 5. Data Structures.  Typically, component testing in software is checking each screen, and each report. The interfaces are tested at the next level.  While screens and reports are tested using requirements-based "black-box" testing, logic modules are best tested using "white-box" tools like logic trees and logic tables.

Integration Testing: There are two types of integration testing.  Internal integration testing is testing module to module (screen to report, for example).  External integration testing is testing system to system (your app to Paypal interface, for example).

System Testing: The best way to system test an app is to create a set of "typical day in the life of a user" flows representing different ways a user would use an app.  Providing all of the previous test levels worked reasonable well, system testing is a way to ensure everything is working together.

User Acceptance Testing: Once the system has been tested enough that the release success criteria has been met, the system should be shown to a real end-user for a final approval.  This should not be a surprise to the end-user.  In a healthy environment, the end-user should have been working closely with the development team as the software was created. Since the product should match requirements, any changes at this point would require updated requirements, and likely a new work order.

Maintenance Testing: This type of testing is used after an OS updated, or database change, or some other type of refactoring has occurred.


The Four Test Types found on the CTFL Exam are:

Functional testing (Black-box Testing): Requirements-based testing. When you cannot see the code, but all you have is a set of requirements, or user-stories. This is traditionally what people think of when they envision testing software.

Non-functional testing: Non-functional requirements are "implied" requirements which apply to the entire software app.  For example, security requirements, or HIPAA requirements, or PCI requirements.

Structural Testing (White-box Testing):This is used for component testing logic modules, where the developer share the code with you, and together you create flowcharts, logic trees, and logic tables.

Change Testing: This testing happens after a bug is found and the software is sent back to the developer, and the developer fixes the bug and returns the updated app to be tested again.


The Three Testing Approaches are:

Black-box Testing: For testing components with requirements, such as screens and reports. 

White-box Testing: for testing components where you can see the code, like logic modules.

Ad-hoc Testing: for testing an app when no requiems are present.  This can be due to time constraints, or perhaps you were just given an app and have to learn about it to create an approach and set of test plans to test it adequately.




In August of 2021, PMI released the PMBOK 7th Ediiton.  I have a previous post describing what's new. Surprisingly, is a book filled with professional soft-skills, and not really a process-guide for executing projects.  Wihile PMBOK 1-6 Editions contain step-by-step process guidance for project execution, PMBOK 7 is all about the soft skills.

I tell my PMP and CAPM students they will need both, and an easy way to think about it is to use PMBOK 6 for learning how to manage projects, and PMBOK 7 for learning how to be an excellent project manager.  One analogy I used in a recent public presentation is that PMBOK 6 is the bones, and PMBOK 7 is the meat.

Mike Berry  Red Rock Research



What do you learn from a CTFL Certification?

I've been instructing students on the ISTQB-CTFL Certification curriculum for a number of years now.  I like to tell students the CTFL program is really a mini project-management training program.

Somebody hands you a software app to test, what do you do now?  You decompose the App in to smaller portions which make sense to you, constructing a Work Breakdown Structure.  Then you identify items to test within the WBS, organizing them by Test Plans. They you prioritize all test plans into a Test Execution Schedule so and your test team will know in what order to perform the tests.

You'll keep track of test coverage, and results, reporting on progress as you work through the app. Here's where it's important to communicate with your developers thoroughly so they can fix bugs you find and get the updated app install back to you for confirmation testing.

finally, when your testing matches the success criteria, you'll report of the final results of your findings and wrap up the testing project, adding lessons learned, estimates vs. actual, and other information into a test historical repository.

All of these activities are the foundation for project management.

PMBOK 7 Announced!

In August of this year (2021), the Project Management Institute will release PMBOK 7!  This edition will be different than past editions of the PMBOK as PMI is emphasizing a descriptive program versus a prescriptive program.

The new Project Management Body of Knowledge will contain 12 Project Management Principles, which help guide decisions, 8 Project Performance Domains, and 7 Model Sets.  They are as follows:


12 Project Management Principles:


  • Stewardship
  • Team
  • Stakeholders
  • Value
  • Systems Thinking
  • Leadership
  • Tailoring
  • Quality
  • Complexity
  • Risk
  • Adaptability and Resilience
  • Change



8 Project Performance Domains:


  • Stakeholders
  • Team
  • Development Approach and Life cycle
  • Planning
  • Project Work
  • Delivery
  • Measurement
  • Uncertainty


7 Model Sets


  • Situational Leadership
  • Communications Models
  • Motivation Models
  • Change Models
  • Complexity Models
  • Project Team Development Models
  • Other Models
In addition, some definitions are expanded upon. One example is thinking of your organization as a System of Value Delivery.

Something a little awkward is that the book is actually 3 separate texts, glued together--complete with their own page numbering systems and indexes.


Book Review: Motivating Employees

Employee motivation is an ever-present concern for most proactive managers.  Interestingly enough, motivation can come from both functional and dysfunctional sources.

I've seen employees motivated for many different reasons: recognition, financial incentive, empowerment, personal growth, tension release, fear, and finally there's that weird Lord of the Flies thing where employees get motivated together against another employee.

In their book, Motivating Employees, Anne Bruce and James S. Pepitone describe the most effective ways to motivate a team.  They describe the three C's which are vital to functionally motivating employees:

1. Collaboration: Be sure to involve employees in decisions and discussions where their efforts are involved.

2. Content: As they produce suggestions, act on those suggestions immediately.

3. Choice: Be sure to offer choices to your employees--even if you can predict what they will decide.

These three techniques actually empower your employees.   Involving employees in decisions that affect them, or the outcome of what they are working on produces a level of buy-in that is hard to match any other way.

Bruce and Pepitone continue with an examination of Theory-X and Theory-Y motivation and management styles.  These styles were originally presented in the 1960's by Douglas McGregor.

McGregor states that Theory-X managers proceed from the assumption that their employees are uninformed, lazy, and needy of high-structure.

Theory-Y managers, however, proceed from the assumption that their employees are qualified, intelligent, and capable of making proper decisions provided they are given proper goals, accountability, authority, and resources to accomplish their tasks.

Although Theory-X is the most effective approach during some situations, if you consider the amount of college-educated employees in the workforce today, it's easy to see how Theory-Y, if applied properly, yields much higher performance.

The authors continue with a formula for encouraging Entrepreneurial Thinking.   Their five-step formula is:

1. Explain the organization
2. Demonstrate how the organization operates and generates income
3. Help your employees understand the competition
4. Encourage intelligent risk-taking
5. Inspire innovative thinking

Another great idea the authors present is to link motivation to performance.  They suggest you develop a written-list of performance standards for meeting and exceeding the expectations you've agreed upon during collaborative sessions with them.

The authors talk about how important it is to weave fun into everything your organization does.   This may sound like a unusual suggestion at first, but the authors point out that there is a direct correlation between fun on the job and employee productivity, moral, creativity, satisfaction, and most importantly--retention.

The final few chapters in the book discuss de-motivating factors (or individuals), and how to deal with them.  There is also a good chapter on conducting effective employee-reviews.

Overall I recommend this book to any manager.   It's a great book to re-read every so oft r/>
Mike J. Berry

A History of ITIL

ITIL, the Information Technology Infrastructure Library, was created by the British Government in the 1980's to address growing complexity in the management of IT infrastructure.  It is based on many of W. Edward Deming's quality principles, and Walter Shewhart's Plan-Do-Check-Act Cycle, which the Western world was just finding out about.

ITIL originated as a voluminous set of books which were simplified during versions 2, and 3, down to 5 books. In 2019 ITIL 4 was released, adding considerations for cloud-based management, DevOps, and Agile.

ITIL 4 emphasizes looking at an organization as a Service Value System (SVS).

Four dimensions that comprise a SVS are:

  1. Organizations and People
  2. Information and Technology
  3. Partners and Suppliers
  4. Value Streams and Processes

In addition, ITIL 4 covers governing the SVS, Continual Improvement, and multiple areas of practice, in which ITIL principles can be applied.


Mike Berry Red Rock Research

Book Review: Freakonomics

I just read Freakonomics: A Rogue Economist Explores the Hidden Side of Everything, by Steven D. Levitt and Stephen J. Dubner.

This New York Times bestseller is an analytical exploration into social cause and affect.  Using analytics, Levitt shows how he was able to detect administrative cheating in the Chicago school districts, prove that sumo-wrestling is fixed, and suggested which baby names will be most popular in 2015.

I can't say enough good about this book.  It was compelling to read, and a great mind-opener about the value of the relatively-new field of business analytics.

Mike J Berry www.RedRockResearch.com

PMBOK 6 Announced!

September 6, 2017,  PMI will release Version 6 of the Project Management Body of Knowledge (PMBOK) Guide.  The updates in the new guide reflect slight adjustments to the framework expanding it to encompass managing equipment as well as human resources, and changing terminology from “control” to “monitor” in many areas.  In addition, PMI will publish the new Agile Practice Guide which will become PMI’s definitive resource for all things Agile.   

The PMP and CAPM tests will not change until January, 2018.


Knowledge Area Changes:

• Project Time Management > Project Schedule Management

• Project Human Resource Management > Project Resource Management


Process Name Changes:

• Plan Stakeholder Management > Plan Stakeholder Engagement (Planning)

• Plan Human Resource Management > Plan Resource Management (Planning)

• Perform Quality Assurance > Manage Quality (Executing)

• Control Communications > Monitor Communications (M&C)

• Control Risks > Monitor Risks (M&C)

• Control Stakeholder Engagement > Monitor Stakeholder Engagement (M&C)


New Processes:

• Manage Project Knowledge (Added to Executing)

• Implement Risk Responses (Added to Executing)

• Control Resources (Added to M&C)


Deleted processes:

• Close Procurement’s

PMI Announces the PMI-PBA Certification!

The Project Management Institute (PMI) announced a new certification for Business Analysts.  Called the Professional in Business Analysis, the PMI-PBA is a program for business analysis, and many other professionals who perform business analysis activities.


Think of Business Analysis as what happens before a project, the Scope development and change management happening during a project, and the post-release RIO work that may happen after a projects.  

Before a large project concept is approved, a Business Analyst may be commissioned to perform a feasibility study, and assuming the concept is feasible, create a Business Case Analysis showing the most recommend option and a few lesser options.

Assuming the Business Case is approved by the proper authority, a project is created and the Business Analyst transfers their information to the new Project Manager and may or may not become part of the team.

If they become part of the team, they likely will manage the Scope baseline, while the Project Manager manages Schedule and Cost baselines. During this process they will also work closely with the Project Manager to implement change control.  They will also makea good 2nd witness for assessing how well the project team is working with other stakeholders.

When the project is complete, they will help evaluate the projects success criteria, which was put into the Project Charter from the Business Case they created. If the success criteria is met, the project can be Closed.  

Any post-solution deployment ROI work can be measured and calculated by the Business Analyst to determine if the target ROI was reached with the solution. This target ROI, and the pre-determined techniques to measure it are found in the Benefits Measurement Plan, also created by the Business Analyst to compliment the original Business Case. Multiple measuring points and a Statistical Process Control chart, and an efficiency frontier chart can be used to assess ROI.

Mike Berry Red Rock Research

Advice for Passing the PMP Exam

The PMP, or Project Management Professional certification, indicates a person possesses years of industry experience participating in projects, and they understand the PMBOK framework.

The PMBOK, or Project Management Body of Knowledge, is a framework comprising 42 processes useful to managing formal projects.

Legendary in the industry, the PMP exam is one of the toughest professional exams out there.  It consists of 200 questions and takes most people the entire 4 hour allotment of time to complete.

The test is put together using Blooms Taxonomy, a learning framework that describes different ways people process learned information.   Recalling lists versus selecting the best option from a set of viable options are examples of categories in Blooms Taxonomy.

One major tip for passing the PMP exam is to expect to be reading questions in street lingo.  For example, the question may be a short story about a manager asking for something from her project manager.  The question will contain no terminology...you will be expected to translate the street lingo into the particular framework component being described and select the correct answer from the choices given.

I teach a PMP Exam Prep class and have many students who pass, and a few who don't.  What's the difference?  Quality study time.  Be sure you take this exam seriously so that you can benefit from the PMBOK framework concepts.

John C. Maxwell Leadership Coach