RSS-Feed
Four Steps to a Successful SCAMPI A - Practice Implementation Indicators Made Practical
Many people perceive the creation of the Practice Implementation Indicators (PII) as painful work with little value. But Practice Implementation Indicators can be the best friend of your process improvement initiative. They can guide your process improvement efforts and help you in tailoring CMMI to your organization's needs. PII database entries can come out of your PPQA work with little or no extra effort, and they tie in nicely with a continuous identification of strengths, weaknesses and non-compliances. This has the additional advantage that if this is done right it leads to efficient, fast and no-surprise SCAMPI A appraisals.
In this article we tell step-by-step how this can be done - and we demonstrate it with examples from our practice.
Prerequisite: Model of the Standard Process’ Components
As with any product, a good model of the components of the organization’s set of standard processes is essential to ensure usability, separation of concerns and maintainability. A model of the standard process’ components is a prerequisite for our approach. It allows for identifiable process elements that can be mapped to CMMI requirements.
Our model of the standard process’ components has three main elements: artefacts, roles and activities. Artefacts are all tangible results of a process (also called work products), e.g. “UseCase” or “Trained Team”. Roles describe a set of skills and responsibilities and can be realized by a person or a group, e.g. “Requirements Engineer” or “Change Control Board”. Activities are steps within a process.
All these three elements are assembled to process flow patterns, e.g. a process flow pattern for change request management, or a process flow pattern for a solution outline phase. We use the term “pattern” deliberately to denote that the organization’s set of standard processes is a set of proven solutions to recurring project situations, and that these solutions must be tailored to establish the project’s process.
We expect projects to use the organizations’ standard process as their common product engineering language (see figure 1). This means that in the plans and the repository of the projects the organization’s artefacts’ names must be used. It also means that project plans must use the names of the roles and the activities of the process patterns. The usage of the organization’s set of process patterns is evaluated by the PPQA team. This ensures practical process descriptions – if the direct use of the process descriptions in the projects is not possible, either the process descriptions are not correct, or the project is not following the best practices. Unstructured process descriptions with more or less text based procedures will not support the approach described here. It requires identifiable process enities which have identifiable instances
in the projects.
Overview of the artefacts
We propose a clear traceability from CMMI requirements, to policies, to the organization’s set of standard processes, to the project’s process, and to the practice implementation indicators. This traceability builds upon the above model of standard process components. In the following paragraphs we give an overview over the core artefacts needed to implement this traceability. The artefacts explained in the next paragraphs (with their names set in italic typeface) are shown in figure 1. In the following sections we will then explain our approach step-bystep.
The policies are the organizations’ requirements, which concretize the CMMI requirements. Good policies are specific enough so that practitioners will understand the senior management’s expectations, and they are general enough so that they can be memorized and that they do not specify any specific practice implementation.
The organization’s set of standard processes is established as a set of process patterns which assemble artefacts, roles and activities as described in the previous section “Prerequisite: Model of the Standard Process’ Components” on page 3. There are two different types of process flow patterns: procedures for event-driven processes (e.g. “Handle Change Request”) and lifecycle models & work breakdown structure patterns (WBS patterns) for project workflows (e.g. the phase “Solution Outline”). All process elements as well as the process flow patterns are mapped to the policies and CMMI.
As explained in the previous section we expect the projects and the organization to use the names of the elements of the organization’s set of standard processes to plan and execute the work done in the projects and the organization. This usage rule establishes a clear traceability from the organization’s standard process to performed processes and work items of each project.
To objectively evaluate whether the projects are using the standard process, the PPQA team gathers direct artefacts, indirect artefacts and affirmations as objective evidence of the implementation of the organization’s standard processes and of CMMI2. The direct artefacts, indirect artefacts and affirmations that show that a project is implementing the organization’s standard processes and follows the policies are called standard process implementation indicators. If the traceability between the organizations’ set of standard processes and CMMI is correct, the direct artefacts, indirect artefacts and affirmations also show that the project is fulfilling the CMMI requirements, thus they can also be practice implementation indicators at the same time (see figure 3 on page 11). Each implementation indicator has a statement3 capturing the affirmation or a judgement why the artefact was considered appropriate. Additionally we collect significant strengths (relevant to the organization) and non-compliances (weaknesses). Once the implementation indicators have been collected and annotated with a statement, a characterization and rating for the observed projects/instances can be produced. We organize the structure of ratings, implementation indicators and strengths/weaknesses in a statement breakdown structure. See section “Step 2: Perform continuous Mini-Assessments in your companies’ own language” for an example statement breakdown structure.
To help the PPQA team to establish the practice implementation indicators, we use practice implementation guidances which map CMMI, the policies, and the elements of the organizations’ standard processes. The set of practice implementation guidance sheets are a handbook for the PPQA team. The practice implementation guidances can roughly be described as expected practice implementation indicators. See section “Step 1: Map your standard processes to CMMI” for details.
In the following sections we will describe the steps of our approach to capture and use practice implementation indicators step-by-step.
Step 1: Map your standard processes to CMMI
What to do: The first step is to map your organization’s set of standard processes to CMMI. You do this at the beginning of your CMMI initiative in order to map your existing organizational process descriptions to CMMI. You also do this at any time when you define new organizational processes.
Output artefact: For each process area and for each specific and generic practice within a process area we list the corresponding policy as well as the expected direct and indirect artefacts and activities. The result are practice implementation guidance sheets. We also amend the practice implementation guidances with the roles which are responsible for realizing the practice, and with questions that can be asked by PPQA teams. To help the PPQA team check the projects and the organization we additionally give interpretation guidance. (See table 1 for a template and table 2 for an example of such a practice implementation guidance.)
The practice implementation guidance sheets provide a clear traceability from CMMI to the process descriptions. The artefacts of the organization’s set of standard processes are used as the expected direct and indirect artefacts in the practice implementation guidance sheets. The activities and procedures are used as the foundation for the affirmations. The roles of the standard processes are used to indicate who is responsible for the implementation of the practice and organizational policy and who will thus be asked by the PPQA team.
The mapping of the standard process to CMMI should be reviewed by a Lead Appraiser to ensure at an early stage that the process solutions of an organization can help the projects fulfill CMMI requirements once deployed. However, even if the standard process fits CMMI, this does not guarantee that the projects implementing the standard process fulfill CMMI requirements. This must be checked by the PPQA – see the next step for this.
Benefits: There are several very important benefits of the practice implementation guidance sheets for several stakeholders.
First, the practice implementation guidance sheets give the process team, e.g. the engineering process group (EPG), a traceability between the organization’s set of standard processes and CMMI, and they allow the team to judge whether the process descriptions cover all CMMI requirements or not. This traceability is also useful if an existing set of standard processes must be mapped to understand their coverage of CMMI and to identify weaknesses of the process descriptions with regard to CMMI. Moreover, the practice implementation guidance sheets allow a Lead Appraiser to evaluate the standard process of an organization at an early stage.
Second, the set of the practice implementation guidance sheets is the handbook for the PPQA teams. The handbook lists the questions the PPQA team can ask, and it lists the expected answers by means of direct and indirect artefacts (“look-fors”) and affirmations (“listenfors”). Note that expected direct artefacts and the indirect artefacts listed in the practice implementation guidance sheets are taken from the organization’s set of standard processes. The practice implementation guidance sheets do not list the actual instances of direct and indirect artefacts. This listing is produced as a result of the PPQA work – see the next step for this.
Third, the PPQA handbook – consisting of all organizational practice implementation guidance sheets – is communicated to the whole organization. This makes expectations from the organization clear and helps the projects and the organization to prepare for PPQA audits.
At an early stage of process improvement – when there is no standard process yet – the practice implementation guidance sheets can also be used to give the projects guidance on how to fulfill the policies. The sheets list for each policy guidance, artefacts, and they possibly link to support material like templates.
Fourth, the practice implementation guidance sheets allow the PPQA team to perform assessments in the organization’s own language. The PPQA team uses the practice implementation guidance sheets as the basis for the assessments. With these sheets the PPQA can check the performed process against both the standard process and CMMI. (See the next section and figure 3 on page 11 for details.)
Our approach can also be applied if you start with incomplete process descriptions that do not yet are a complete organization’s standard process. If the standard process is constructed from components as we have described, these components can be developed iteratively, and the practice implementation guidance sheets can be extended accordingly. The following gives an example: The organization starts with a mapping of the policies to CMMI, and the PPQA team starts to check the policies (actually this can already have quite an impact on the organization). In the next step the organization defines the artefacts and adds them to the practice implementation guidance sheets. After that the roles are added in the same way. The PPQA team picks up these new elements and starts checking them as they are added into the practice implementation guidance sheets. This allows the engineering process group (EPG) to release the process patterns that comprise the organizations’ set of standard processes step-by-step as the organization matures, and there is a defined process how the PPQA team picks them up and starts checking them.
Step 2: Perform continuous Mini-Assessments in your companies’ own language
What to do: The second step is to continuously assess the organization, based on the PPQA handbook that consists of the practice implementation guidance sheets. These assessments are done for projects as well as for the organization, e.g. to assess PPQA, MA or management tasks. We call assessed organizational units “instances” as in the SCAMPI method definition documents. There are several methods possible to conduct the mini-assessments (e.g. SCAMPI, own organizational method, audit techniques).
The assessments evaluate the projects and the organization against the standard process as well as against CMMI.
The PPQA team looks for standard process implementation indicators (direct artefacts, indirect artefacts and affirmations) that show whether projects are using the elements of the organization’s standard processes and follow the policies. If the traceability between the organizations’ set of standard processes and CMMI is correct (see previous step), the implementation indicators also show that the projects and the organization are fulfilling the CMMI requirements, thus they can also be practice implementation indicators at the same time. This requires the PPQA team to evaluate the implementation indicators against the standard process as well as against CMMI. In case the implementation indicators show that the standard process is fulfilled, but not CMMI, the PPQA team raises a process description issue for the EPG (see figure 3 on page 11).
In order to enable the PPQA team to check against the standard process and against CMMI, the team should be trained in both.
The assessments have several goals. They objectively evaluate the actual process improvement progress of the organization, and they identify strengths, weaknesses (non-compliances) and further improvement opportunities. Moreover they show that the organization cares about the implementation of the processes. Either the PPQA team or another support group should help the projects fix their noncompliances. This way the mini-assessments can also be a driver of the process improvement actions within the projects. The method used for the continuous mini-assessments could be a “lightweight” SCAMPI B.
Output artefacts: The output of the assessments is a statement breakdown
structure for each assessed instance with ratings, characterizations, direct artefacts, indirect artefacts, affirmations, significant strengths and weaknesses. The weaknesses are handled as non-compliances. (See figure 4 for an example of such a statement breakdown structure.)
The statements should be captured in a list which allows for easy and fast writing of statements. The team should have an easy overview of all statements for one practice.
The assessments should be calibrated by a Lead Appraiser in order to ensure that the CMMI model is interpreted correctly and that the ratings are correct. For this calibration the Lead Appraiser should regularly review the work of the PPQA, and he should participate in selected mini-assessments. It is also important that the PPQA team does not just tick off the items of the practice implementation guidance sheets, but that they use the sheets as a guidance to “look for” and “listen for” the true story of the projects in their individual context.
The general usage of statements helps the team to objectively capture all information they receive, and they allow to comment on the artefacts the team has evaluated. The statement breakdown structure supports the approach of capturing information as long as the implementation of a practice is not clear. It also visualizes the fact that a final instance characterization can only be made if the supporting statements are sufficient (“roll-up”). This supports an open and objective appraisal that focuses on the goal of gaining confidence in the implementation.
Benefits: There are several very important benefits for continuously evaluating the organization and maintaining a database with the statement breakdown structures (thus the strengths, weaknesses and ratings).
As we already mentioned, the practice implementation guidance sheets allow the PPQA team to perform the assessments in the companies own language, while still a clear relationship to CMMI is maintained.
The continuous capturing of practice implementation indicators, significant strengths, weaknesses, and ratings fills the assessment database with the practice implementation indicators. The weaknesses are handled as non-compliances and tracked till closure. This way they evolve into a strength, thus another practice implementation indicator. In our approach the creation of practice implementation indicators is not a useless and cost-intensive effort, and the practice implementation indicators are a natural and useful output of the continuous work of the PPQA team.
Moreover, our assessment method ties the identification of non-compliances, the rating against CMMI, the identification of improvement opportunities and the generation of practice implementation indicators seamlessly together.
Step 3: Monitor your process improvement success
What to do: We continuously rate the CMMI conformance of the overall organization after each assessment based on all assessment ratings in the past 6 months (“rolling rating”). We also provide a measurement of the rate of fixing non-compliances. Both measurements give the organization an objective insight into its current CMMI conformance and its speed in fixing problems.
Output artefacts: The output of the continuous CMMI conformance monitoring are two measurements: an organizational rating and the rate of fixing non-compliances.
The organizational rating is based on the results of all assessments in the last 6 months. The organizational rating is based on the lowest rating of a practice in all assessments. Thus, if the practice PP SP 1.1 has been rated in the last 6 months in 8 assessments as FI, FI, LI, LI, PI, FI, FI, FI we rate it as “PI” on the organizational level5. This rating is more conservative rating than a SCAMPI A rating and provides the lower bound of the current CMMI conformance of the organization as a measurement. Note that the “rolling rating” is derived from the ratings of the mini-assessments without any further input, adjustment or judgement. This shall ensure that the results of mini-assessments are not tampered with.
As an additional measurement, we provide the rate of fixing non-compliances. This rate is the average time to fix non-compliances6. It provides a measurement for the capability of the organization of closing gaps.
A correct rating is based on the quality of the results of the miniassessments. It is therefore essential that the mini-assessments and their ratings provide a correct picture of the organization’s current maturity. (See section “Step 2: Perform continuous Mini-Assessments in your companies’ own language” on page 9.)
Benefits: The main benefit of the continuous mini-assessments and the rating measurements is that most of the SCAMPI A preparation work is done seamlessly with the “normal” PPQA work done day-today by the organization.
The rolling rating of the organization provides an understanding of the organization’s compliance toward CMMI and allows to judge whether an organization is ready to go into a SCAMPI class A appraisal or not. Because the rating is conservative and based on a broad set of projects, an organization will very likely pass a SCAMPI class A appraisal if the rating shown by the measurement is positive.
The mini-assessments and the “rolling rating” reduce the step of a readiness review for a SCAMPI A to the work of a day, where merely the completeness of the practice implementation indicators in the assessment database is checked. Basically, in our set-up of the miniassessments we have a continuous readiness-assessment build in.
Step 4: When you are ready, perform an efficient and no-surprise SCAMPI A appraisal
What to do: The last step is an efficient and no-surprise SCAMPI A appraisal. This appraisal is based on the practice implementation indicators gathered by continuously assessing the organization (see “Step 2: Perform continuous Mini-Assessments in your companies’ own language”). The appraisal is only performed if the organization is ready (see “Step 3: Monitor your process improvement success”).
Output artefacts: The appraisal produces all known outputs of a SCAMPI A appraisal, including a rating of the organization’s maturity level.
Benefits: In our approach the SCAMPI A appraisal does not require any extra work to capture the practice implementation indicators. Moreover, the appraisal is only performed if the organization is obviously ready. This avoids hectic improvement of gaps found while preparing for the appraisal – or the effort for non-successful appraisals.
Our approach saves time and effort spent for an appraisal. The visible progress reported by the PPQA team and the reduction of hectic improvements at the end also improves the credibility of the overall process improvement initiative.
Summary
A major prerequisite for our approach is the monitoring of each step by an authorized Lead Appraiser. He has the responsibility to verify the correctness of the traceability from an early stage on (step 1). He must also monitor and coach the work of the PPQA in order to ensure the generation of correct practice implementation indicators (step 2). Last but not least the Lead Appraiser will review the PPQA measurements and the “rolling rating” to ensure they provide a correct forecast of the organization’s rating (step 3).
Additionally, a process specialist is necessary who can support the development of the model of the components of the organization’s standard process, and who can guide establishing a maintainable set of process patterns.
Our experience shows that practice implementation indicators can be of great help if they are created as part of the normal PPQA work. The prerequisite for this is a proper process architecture that defines process elements that are mapped to CMMI and that are actually used in the projects. If this is done right, a SCAMPI A appraisal can be performed very effectively and with no surprises.
Glossary
Affirmation: An oral or written statement confirming or supporting the implementation of an element of the organization’s set of standard processes (see SPII – Standard Process Implementation Indicator) or of an implementation of a specific or generic practice of CMMI (see PII – Practice Implementation Indicator). [SCAMPI MDD and wibas]
Direct Artefact: The tangible output resulting directly from implementation of an element of the organization’s set of standard processes (see SPII – Standard Process Implementation Indicator) or from implementation of a specific or generic practice of CMMI (see PII – Practice Implementation Indicator). [SCAMPI MDD and wibas]
EPG – Engineering Process Group: The team responsible for maintaining the processes and which is the focal point for process improvement. [wibas]
Indirect Artefact: Artifacts that are a consequence of performing an element of the organization’s set of standard processes (see SPII – Standard Process Implementation Indicator) or of a specific or generic practice (see SPII – Standard Process Implementation Indicator). Indirect artefacts substantiate an implementation, but they are not the primary output. [SCAMPI MDD and wibas]
PII – Practice Implementation Indicator: An objective attribute or characteristic used as a “footprint” to verify the conduct of an activity or implementation of a CMMI model specific or generic practice. Types of practice implementation indicators include direct artifacts, indirect artifacts, and affirmations. [SCAMPI MDD]
PIID – PII Description: A structure or schema defined to provide a repository for the PII information. This is a notional description of the content, not a physical definition of the format. [SCAMPI MDD]
PIIDB – PII Database: Organizations may provide as input to the appraisal a PII database, with a mapping of model practices to corresponding processes and objective evidence that can be used to verify practice implementation. [SCAMPI MDD]
PIG – Practice Implementation Guidance: Whereas the PII are the actual direct and indirect artefacts used as evidence that a practice is performed, the practice implementation guidance sheets list the expected instances and affirmations. The PIGs map the artefact definitions, role definitions and activity definitions of the standard process to CMMI. [wibas]
PIGDB – PIG Database: The database that contains the practice implementation guidance sheets. [wibas]
PPQA – Process and Product Quality Assurance: In this article, the team which is responsible for providing staff and management with objective insight into processes and associated work products. [CMMI, wibas]
Process Description: A documented expression of a set of activities performed to achieve a given purpose that provides an operational definition of the major components of a process. The documentation specifies, in a complete, precise, and verifiable manner, the requirements, design, behavior, or other characteristics of a process. It also may include procedures for determining whether these provisions have been satisfied. Process descriptions may be found at the activity, project, or organizational level. [CMMI] In this article we extend the definition of “process” to cover not only activities, but also artefacts and roles. In our model of the elements of a standard process we use Artefact Descriptions, Role Descriptions, and Activity Descriptions. From these elements process flow patterns (Procedure Descriptions and Lifecycle Descriptions) are constructed. A process description is composed of all these process elements. [wibas]
SPII = Standard Process Implementation Indicator: An objective attribute or characteristic used as a “footprint” to verify the conduct/ implementation of a standard process element. Types of process implementation indicators include direct artifacts, indirect artifacts, and affirmations. [wibas]
Whereas a PII indicates that a CMMI practice is implemented, the SPII indicates that a standard process element is implemented. The goal is that SPIIs and PIIs are identical, that is, an objective evidence element (“footprint”) indicates that both CMMI and the standard process are implemented.
Literature
- Mary Beth Chrissis, Mike Konrad, Sandy Shrum: CMMI – Guidelines for Process Integration and Product Improvement, Addison-Wesley, 2003
- Members of the Assessment Method Integrated Team: Standard CMMI Appraisal Method for Process Improvement (SCAMPI), Version 1.1, Method Definition Document, Carnegie Mellon University, 2001
- Tim Kasse: Action Focused Assessment, Artech House, Inc., 2002
- CMMI Product Team: Appraisal Requirements for CMMI, Version 1.1, (ARC, V1.1), Carnegie Mellon University, 2001
- IBM: Developing Object Oriented Software, Prentice Hall, 1997
- Will Hayes et. al.: Handbook for Conducting SCAMPI B and SCAMPI C Appraisals, Carnegie Mellon University, 2004
