Victor
José Aguiar Teixeira de Melo Fran
State
University of Maringá, Brazil
E-mail: victor.jatmf@gmail.com
Gislaine
Camila Lapasini Leal
State
University of Maringá, Brazil
E-mail: gclleal@uem.br
Renato
Balancieri
State
University of Maringá, Brazil
E-mail: renatobalancieri@gmail.com
Ana
Cristina Rouiller
Universidade
Federal Rural de Pernambuco, Brazil
E-mail: anarouiller@gmail.com.br
Submission: 3/26/2019
Accept: 9/19/2019
ABSTRACT
With the dissemination of quality models and the need for measurement-based management for software organizations, a variety of approaches and models are emerging to support organizations on this journey. According to a Systematic Review of Literature that selected 30 of 692 papers returned from search sources, it was noticed the need for guidelines and guides regarding the measurement process in a specific sector: micro and small companies that work with software maintenance. In this work, an empirical study was carried out to understand how the measurement process aligned with quality models can be implemented in these organizations. The field study followed the implementation of an improvement program based on CMMI® level 2 in 7 Brazilian companies from Maringá, Paraná. Finally, it was found that the process for the definition of a measurement guide can be common to a specific group of companies, taking steps and guidelines for its execution, always aligned with the organization’s strategy, giving top management an objective view work and process, supporting decision making.
Keywords: Measurement guide; Software process improvement; Software maintenance; CMMI; Metrics
1.
INTRODUCTION
Software Process
Improvement (SPI) has received considerable attention in recent years, both in
academia and industry. A reflection of this is the popularization of maturity
models that have been disseminated and applied in software organizations due to
factors related to competitiveness, product quality, economy and efficiency
(BRASIL; FONTOURA; SILVA, 2013; TRAVASSOS; KALINOWSKI, 2014; O'CONNOR, 2019),
such models have absorbed areas for the measurement process. In Brazil, this
phenomenon can be observed through data on the number of official evaluations
in software maturity models such as the Capability Maturity Model Integration
for Development (CMMI®-DEV) (CMMI, 2010) with 71 current certifications and the
Reference Model for Software Process Improvement (MR-MPS-SW) (MPS.BR, 2011)
with 161 current certifications at various levels and constellations (people,
software and services).
Model-based SPI
encompasses the development of software from conception to final delivery,
through all stages of the software lifecycle. Despite the relevance of
maintenance in this life cycle, it did not receive the same degree of attention
as the other stages of development (APRIL; HAYES; ABRAN; DUMKE, 2005; EDBERG;
IVANOVA, 2011; PINO; et al., 2012; FERNÁNDEZ-SÁEZ;
CHAUDRON; GENERO, 2018).
Historically, software development has been far more representative than
maintenance in most organizations. However, this scenario is changing, given
the organizations' effort to make the most of their software development
investments, keeping the product operating as long as possible (EDBERG;
IVANOVA, 2011)
It is perceived that
maintenance is an essential part of the software life cycle and has been
gaining increasing attention among industry and researchers, according to the
Brazilian Association of Software Companies (ABES) (ABES, 2017), 55.4% of the
investments, in millions of dollars, for the Brazilian software and services
market, adds the stage of software maintenance and evolution. Other specific
characteristics of the maintenance context are: continuous and repetitive
nature, focus on individual response to change requests, and uncertainty of new
scope changes (APRIL; ABRAN, 2008, PORT; TABER, 2018). Thus, the improvement of the
process based on specific reference models for the development of a software
project is not totally adequate to the specifics of the software maintenance
and evolution context (KAJKO-MATTSSO, 2001; KAJKO-MATTSSON; SNYGG; HAMMARGREN,
2012; NIESSINK; VLIET, 2000; PINO et al., 2012) and has caused damages to organizations.
The problem may become even worse when observed in the micro and small
Brazilian companies sector. According to the Brazilian Software Association
(ABES), in a survey conducted in 24 Brazilian states, of the software
companies, 86% of them are classified as micro and small companies
(SUKARIE-NETO, 2018). This represents a significant portion of the sector.
Considering this
scenario, the main challenge faced by these organizations is the limitation of
resources: people, time and budget. These limitations lead to insufficient
processes, methodologies, guidelines, tools and documentation required for
software maintenance. Also in the context of some micro-software organizations,
a single product may be the basis of the organization's business model, and new
product development activity may be rare or nonexistent.
The Knowledge
Engineering Body of Knowledge (SWEBOK) (EDBERG; IVANOVA,
2011) presents some of the issues related to software maintenance, grouped
into four categories: i) Technical issues; ii) management issues; iii) Estimate
of Costs; and iv) measurement, the entities that can be subjected to
measurement are process, resource and product. If applied correctly, the
measurement has great potential to contribute to a better understanding of the
maintenance and evolution of software and to motivate necessary changes in
processes and methodologies (ROMBACH; ULERY, 1989).
However, measuring
quality, productivity, and other aspects of software maintenance within an
organization is a difficult task (DESHARNAIS; APRIL, 2010, OKIKE, 2018),
mainly because the nature of the work differs from software development by
design (ALORAN; EID; AL-SARAYREH, 2015), especially by the characterization of
its work continuous and on demand.
Therefore, the objective
of this work is to observe how the implementation of a measurement process
within a Software Process Improvement (SPI) program based on quality models in
micro and small companies, which also work with software maintenance, takes
place in practice.
2.
LITERATURE REVIEW
According to DeMarco
(DEMARCO, 1986), one cannot control what cannot be measured, which demonstrates
the need for measurement with respect to the Software Engineering context. Good
management based on indicators supposes the possibility of predicting the
future behavior of software products and processes, from the moment we have
reliable and appropriate information. In this way the measurement becomes an
important practice since one cannot predict what cannot be measured
(KITCHENHAM; PFLEEGER; FENTON, 1997). With this, several quality models and
guidelines for improvement were created based on the measurement process.
SPI can be implemented
ad hoc, but process improvement implemented systematically, based on models,
standards or approaches, has been shown to be more efficient and with better
results for organizations (MATA-LIMA et al., 2016; MUTAFELIJA; STROMBERG, 2003;
SALVIANO 2006).
Among the existing
quality models, the most widely disseminated in the context of micro and small
Brazilian companies that also work with the software maintenance stage are
Capability Maturity Model Integration (CMMI®) (GONÇALVES; OLIVEIRA; KOLSKI,
2017) and MR-MPS-SW.
These quality models
are intended to help companies evaluate current processes and provide guidelines
for driving improvements in them based on data analysis, corrective action and
decision making (CORNU et al., 2012; JAIN; SHARMA; AHUJA,
2019). For this work
the CMMI® model will be used to validate the field study due to its
international recognition. In CMMI®, the process area that represents and
contains measurement practices is called Measurement and Analysis.
2.1.
The CMMI® Measurement Process
The goal of the
Measurement and Analysis process area is to develop and sustain the measurement
capability that is used to support information management needs (CMMI, 2010).
This general purpose is divided into two specific objectives:
(1)
Align
measurement objectives and activities with identified information needs and
objectives;
(2)
Provide
measurement results that address the needs and objectives of identified
information. This process area, as well as the CMMI® model in all its
definitions, presents only what is charged, what the model expects from the
organization to consider the process area implemented, ie "what to
do" and not "how to make".
The Measurement and
Analysis process area, as well as the others, is composed of generic objectives
(SG) and specific objectives (SP):
·
SG1.
Align Measurement and Analysis Activities:
o
SP1.1.
Establish Measurement Objectives;
o
SP1.2.
Specify Measures;
o
SP1.3.
Specify Data Collection and Storage Procedures;
o
SP1.4.
Specify Analysis Procedures.
·
SG2.
Provide Measurement Results:
o
SP2.1.
Obtain Measurement Data;
o
SP2.2.
Analyze Measurement Data;
o
SP2.3.
Store Data and Results;
o
SP2.4.
Communicate Results.
In addition to the
practices of the Measurement and Analysis process areas, at the maturity level
2, we have the generic objective (GP2.8), which emphasizes that the processes
must be monitored and controlled, an activity that also involves measurement at
this level. Because it is a generic practice, it focuses on all process areas
of maturity level 2.
3.
RELATED WORK
The related works were
identified from a Systematic Review of Literature (SRL), which is a means of
identifying, evaluating and interpreting all available research relevant to a
particular research question, topic area or phenomenon of interest (KITCHENHAM,
2007).
From an SRL performed
by the authors, with 30 papers selected from 692 papers returned from automatic
search sources (IEEE Explore, ACM Digital Library, Science Direct, Scopus and
Inspec / Compendex) and manuals (14 sources from conferences and periodicals
relevant to the area of research) in the period 2007-2017, it was possible to understand
the state of the art and it was noticeable the effort of some researchers to
define quality models adherent to software maintenance, especially with regard
to the measurement process in practice (CHAPETTA; TRAVASSOS, 2016;
UNTERKALMSTEINER et al., 2014).
The work analyzed
shows, as of great relevance to the organization, the implementation of a
measurement process to evaluate the work performance and evolution of the
process improvement, always highlighting its organizational benefits focused on
improving the quality of product delivery and / or service. Several authors
(AKINGBEHIN, 2008; FERREIRA; BARCELLOS; SANTOS, 2017; DÍAZ-LEY; GARCÍA;
PIATTINI, 2010; KURTEL; OZEMRE, 2013; LIN; HUANG, 2009; SCHRETTNER et al.,
2012; SOUTHEKAL; LEVIN, 2011; UNTERKALMSTEINER et
al.) point out that there are not many options for defined standards and models
that guide the definition and maintenance of a measurement process in
organizations. All of them proposed a measurement-based process quality
framework, some of which are specific to software maintenance such as Modus
(MOREIRA, 2008), PPQM (GUCEGLIOGLU; DEMIRORS, 2011), MIS-SMME (DÍAZ-LEY; GARCÍA; PIATTINI, 2010), among others.
According to case
studies (BURGER; HUMMEL, 2012; DIKICI; TURETKEN; DEMIRORS, 2012; FERREIRA et
al., 2007; MONTEIRO; OLIVEIRA, 2010; PEIXOTO et al., 2010; YU et al., 2009), it
can be seen that the process measurement applied to the organizations can be
used as support in increasing performance and productivity, as well as helping
in actions for process improvement and maturity of these companies. Some
authors have focused directly on improving decision making and strategic
direction (BASILI et. al., 2007; RODRÍGUEZ.; KUVAJA; OIVO, 2014).
It is also agreed
between the authors of the works, the need for some quality model to evaluate
the proposed indicators, with CMMI® being the most cited model among them.
Still to aggregate the results, an analysis was performed on the of the
organizations observed in the studies, with only 3% referring to
micro-enterprises (given that this proves the need for research in this
sector), and more than 50% of the studies at least mentioned the size of the
organization, generalizing the indicators, the measurement process has to
adhere to the size of the organization, cannot be generalized (DÍAZ-LEY;
GARCÍA; PIATTINI, 2010; FERREIRA et al., 2007; MOURA; FRANÇA; ROUILLER, 2015;
PÉREZ; MENS; KAMSEU, 2013). The results obtained in this systematic review were
important for the knowledge of how the literature addresses the measurement
process in the context of SPI for software maintenance companies.
4.
RESEARCH METHOD
After SRL performed by
the authors, a field study was planned and conducted. A field study or study is
a case study of the real world without necessarily having direct intervention
by the researcher / observer responsible (KITCHENHAM et al. 2002).
When conducting a field
study it is important to keep in mind that this type of specific study seeks to
deepen a given reality, it is basically carried out by means of direct
observation of the activities of the group studied and, if necessary,
interviews with the purpose of capturing explanations and interpretations about
the studied context. This type of study differs from the case study, since the
case study can be defined as an exhaustive, deep and extensive study of one or
a few units, empirically verifiable, in a way that allows its comprehensive and
detailed knowledge (YIN, 2015).
Field research has the
purpose of observing facts and phenomena of the way they occur in reality and
also collecting data on the elements observed and then analyzing and
interpreting them, based on a solid and well-founded theoretical foundation, in
order to understand and explain the problem that is the object of study of the
research. The field study does not provide specific techniques for data
collection, and they may vary according to the needs of the work developed.
The field study
performed in this work is characterized as an exploratory and descriptive study
where the researcher analyzes, observes, records and correlates aspects,
phenomena or variables that involve facts, without manipulating them. The
investigation takes place without the interference of the researcher, where he
only "seeks to discover, with the possible precision, how phenomena occur,
their relation and connection with others, and also their nature and their
characteristics" (BERVIAN; CERVO; SILVA, 2002). Being also exploratory, it
aims to provide greater familiarity with the object of study. It is necessary
to define a research process that identifies the nature of the phenomenon and
points out the essential characteristics that one wants to study (KÖCHE, 1997).
Characteristics of the
descriptive research are spontaneity (the researcher only observes the
phenomenon), naturality (the facts are studied in their natural habitat) and a
high degree of generalization, which dispenses with great concern about the
representativeness of the selected sample. In addition, it is a qualitative
study, we sought to verify a phenomenon through observation and study (KIRK;
MILLER; MILLER,1986).
The field research
conducted for the execution of this work followed the steps that were adapted
from the method proposed by Neto (NETO, 2002):
(1)
definition
of the research objective;
(2)
choosing
the context that the study would run;
(3)
selection
of companies to be observed;
(4)
observation
of the context that was defined for the execution of the research;
(5)
data
collection through interviews and analysis of available documents; and
(6)
analysis
and conclusion of the aspects observed in field research.
5.
FIELD STUDY
5.1.
Contextualization
This field research was
conducted in the city of Maringá-PR, which is a regional hub in software
development, and has the presence of a Local Productive Arrangement of Software
(APL). In addition, Maringá has one of the highest percentages of companies
evaluated in CMMI® nationally, in relation to the local universe of companies,
being behind only of São Paulo and Recife, according to Software By Maringá and
Commercial and Business Association of Maringá (ACIM). These characteristics
make the proposed theme and the study applied in the region relevant.
The study was conducted
during the implementation of level 2 CMMI® quality model activity activity
which lasted about 11 months. In spite of the presence of the other process
areas present in the model, the observation stopped the activities related to
the Measurement and Analysis process, given the scope of this work.
The SPI implementation
project aligned with CMMI® was a partially subsidized initiative by SEBRAETEC
(SEBRAETEC), a national program of the Brazilian Micro and Small Business
Support Service (SEBRAE), which encourages and supports initiatives to improve
processes and evaluations in national and international quality models for
software companies (SEBRAE, 2017).
The improvement project
was carried out jointly with SWQuality Consultoria e Sistemas, a Brazilian
company recognized nationally and internationally for its successful work with
SPI in the most diverse models and business sectors, also accredited by the
CMMI Institute as an official evaluation institution of the model.
The purpose of this
observation is to understand how the measurement process aligned with quality
models can be implemented in micro or small companies that work with software
maintenance.
5.2.
Sample Selection
The method of selection
of the sample adopted is non-probabilistic and for convenience. The sample for
convenience refers to the acquisition of answers by companies that have availability
and are willing to collaborate (KITCHENHAM; PFLEEGER, 2008).
The companies were
selected using criteria defined according to the study objectives, where there
is no knowledge of the probability that a given population element will be
selected. To select the samples, companies were chosen that, at the time of the
research, accepted to be observed and were available for observation of the
researcher.
To reduce the chance of
choosing samples that did not contribute to the study objective, the following
criteria were taken into account:
· micro or small businesses;
· perform maintenance of your software
products;
· be implementing the software process
improvement aligned to some quality model that focuses on measurement;
· be implementing the SPI with the support
of some specialized external consulting; and
· have the objective of undergoing a
formal and official evaluation of the quality model in question.
Seven micro and small
Brazilian companies from the Maringá region of Paraná were selected. All
observed organizations had as main activity the maintenance and evolution of
their software products. Some of them used agile practices, and two of them
followed the cascade model of software development, they had 1 to 3 teams,
which were composed of 4 to 8 professionals who worked throughout the
maintenance cycle.
For all organizations
observed, the main product was an Integrated Enterprise Management System
(EMS), which varied the market niche that the company operated, ranging from
stalls to medical laboratories, also some of them had government software.
5.3.
Study Execution
The implementation of
the process improvement and definition of the measurement guide in the
organizations was given by biweekly visits where the consulting company trained
with formal orientations or from mentoring the employees of the companies and
left activities for them to execute. These activities would be reviewed on the
next visit and the cycle would be repeated until the end of the project.
This project was
divided in phases as it is possible to see below, the visits of the consultancy
in the companies were accompanied by the researcher / observer so that it could
carry out its analysis.
·
Phase 1 - Diagnosis: At this stage, the consulting team
carried out a diagnosis of the companies with the objective of identifying to
what degree the organization adhered to the processes required by CMMI®-DEV
level 2, to understand how the work was carried out in the day to day of the
company, identification of problems and alignment of expectations with those
involved about the improvement program. Based on the information collected, the
consultancy carried out high-level planning considering the goals established
for the project.
·
This
phase took place through separate job interviews: directors, managers,
architects, requirements analysts and developers were heard.
·
Phase 2 - Kick-off and Training. The improvement program was
presented to all employees of the company with the goal of leveling
expectations and informing the kick-off of the project. In this presentation,
doubts were clarified and guidance was given regarding the structuring of the
environment conducive to the accomplishment of the works.
·
Training
was also conducted in order to level the employees of the companies directly
involved in the CMMI®-DEV implementation project, in this training the model
was presented, as well as its process areas, and the general objective of each
of them, as well as also a high-level schedule of the implementation of these
processes, with milestones to review the progress and possible planning of the
other activities to be developed.
·
Phase 3 - Reformulation of the
Software Development Process: After mapping, the current software development and maintenance
processes were reformulated according to the improvement program, good
practices and new procedures were added using the SCRUM agile method (SCHWABER,
1997) and model guidelines as a reference.
·
A
close follow-up of the consulting team and the team of developers with training
and mentoring were carried out to better transpose the work method with gradual
changes. Adaptations in the SCRUM method were performed according to the need
and limitation of each organization (sprint size, distribution of roles and
responsibilities, etc.). Parallel to this phase, Phase 4 has already begun.
·
Phase 4 - Deployment of Management
Process Areas: At
this stage, emphasis was placed on the process areas related to the management
of the requirements, work and the monitoring and follow-up of the day to day.
Processes were defined to make these activities as fluid as possible. In the
monitoring and control stage, indicators proposed by SCRUM were initially used,
such as Team speed. The number of hours began to be followed to serve as a
basis in the calculation of the other indicators to be defined.
·
Phase 5 - Support Process Area
Deployment: At this
stage, the areas of Process Management Configuration, Product Quality Assurance
and Process and Measurement and Analysis were worked directly. For these
process areas, the work was structured and adapted to the reality of each of
the organizations. Strategies to meet the requirements of the model were
defined without hurting the agility of work in companies. These processes were
implemented, monitored and adapted whenever necessary.
·
Phase 6 - Institutionalization of
Processes: At this
stage the processes were institutionalized in all the companies' projects. This
stage began next to step 3 and followed gradually until the completion of the
implementation of the processes adhering to the proposed quality model.
Durante a
institucionalização, o uso dos processos foi monitorado por meio de indicadores
e possíveis desvios e melhorias foram identificados e implementados. Essa etapa
teve como principal objetivo observar como a empresa se comportava e executava
os processos definidos sem interferência direta da equipe de consultoria, que
intervia somente onde era necessário.
·
Phase 7 - Preparation for Evaluation. Finally, a simulated evaluation
was carried out, indicating whether companies could undergo an official
evaluation of the Standard CMMI Appraisal Method for Process Improvement SCAMPI
A (SCAMPI, 2011), or whether some adjustments would still be necessary. To
conclude, a workshop was held and organizations presented their results to each
other by adding value to organizations in the form of an exchange of
experiences.
5.4.
Discussion of the Field Study
During the execution of
the field survey, during each of the phases, notes were made to record the
observations that made analogy to the process of process improvement with the
measurement process. Whenever necessary, non-systematic questions were asked
both for the organization's employees and for the members of the consulting
firm.
Also during observation of the improvement project phases, each goal of
the CMMI® Level 2 Measurement and Analysis process area was related to the
steps to check if the work performed meets the needs of the models. This
relationship can be seen in Table 1.
Table 1:
Relationship between the phases of the implementation and the expected results
of the Measurement and Analysis in CMMI®
|
CMMI |
Phase 1 |
- |
Phase 2 |
S.P. 1.1 |
Phase 3 |
S.P. 1.2 |
Phase 4 |
S.P. 1.2 |
Phase 5 |
S.P. 1.1, S.P. 1.2 e S.P. 1.3, S.P. 1.4 |
Phase 6 |
S.P. 2.1, S.P. 2.2 e S.P. 2.3, S.P. 2.4 |
Phase 7 |
S.G. 1 e S.G. 2 |
Phase 1 did not correspond directly
to any of the practices, but was necessary for the start of the improvement
project. In the diagnostic carried out were collected some strategic objectives
of the managers and some information needs that later would serve as input for
the definition of the indicators, since, according to the existing approaches
to definition of measurement process, they should be aligned.
Phases 2, 3, and 4 are
more focused on process definitions and redefinitions, where the objectives of
measurement, measurement, data collection and storage, and analysis criteria
have been defined, validated and consolidated in Phase 5. In these phases the
process was also executed, but its focus was on the alignment of the
Measurement and Analysis activities.
Phase 6 had as main
objective the maturation of the execution of the previously made definitions,
such as obtaining, analyzing and storing the data and its communication to the
involved ones, being the focus in providing the measurement results. Finally,
Phase 7 encompasses all specific practices and achieves specific objectives.
The Measurement and
Analysis process, as well as the other CMMI®-DEV Level 2 processes, were
briefly addressed in employee leveling training, ensuring their commitment and
engagement in building new processes in organizations.
With the management of
the teams following the SCRUM method, a time-box was defined that, together
with the teams' timekeeping, allowed a periodicity for the collection of data
that would generate the indicators, initially guided by method metrics such as
team speed, based on the ability of story points with the technique of
estimative Planning Poker (GRENNING, 2002). By dividing management into short
and fixed time cycles, SCRUM assisted in standardizing the work.
In the implementation
of the management processes, the development management was carried out
following a previous planning, with organization and prioritization of the
demands, where the objectives and parameters and goals of monitoring were
defined by the responsible one. From these parameters, it was possible to
observe deviations in the defined goal, making it possible to take corrective
actions whenever necessary.
The Measurement and
Analysis process was worked with greater emphasis in the implementation of the
support process areas, having all its objectives worked directly, seeking
strategies to satisfy all the expected results of the reference model. Based on
the measurement objectives and needs already identified and the metrics created,
a Measurement Guide was established where this information was recorded and
related to the indicators and steps for the execution of the process were
described.
The Measurement Guide
also defined the collection and storage procedures to analyze this information
and how the entire process should be followed, with responsible and
well-defined steps. The strategy for the execution of the process was also
established, which includes collecting and analyzing the data, storing it and
communicating the results to those involved and interested. It should be noted
that this guide and its indicators should be reviewed periodically, since the
strategic needs of companies may vary over time.
In addition to a
document name, company name and document maintainer header, this guide has the
following sessions:
1. Objectives and Need for Information:
A relationship between each of the defined indicators, which strategic
objective it helped measure or respond to, and why that information was
necessary for the organization.
2. Indicator Specification: In this
section, the indicators were listed and for each of them the following
information was given:
a.
Name: Indicator name and acronym;
b.
Description: a brief description of the purpose
of the indicator;
c.
Target: value or expected value range to maintain the
indicator. It may also contain information about where you want to go.
d.
Periodicity of Analysis: informed of how long this
indicator should be analyzed and presented to those involved.
e.
Formula: What is the formula for calculating the
indicator?
f.
Unit of measure: which unit of measurement was
used.
g.
Collection Procedure: a step-by-step of how and where to
fetch the information needed to calculate the indicator.
h.
Presentation Form: how the indicator would be
displayed: histogram, pie chart, table, among others.
i.
Analysis Criterion: This field was based on the goal,
and deviation ranges were defined and classified as "OK",
"Alert", "Critical" and "Attention". For each of
the classifications, an action was defined and should be taken for the
normalization of the indicator.
j.
Storage: where the indicator will be stored after
collected for reuse and history.
k.
Responsible: Indicates the person or role
responsible for collecting and presenting the indicator.
l.
Target Audience: indicates the roles that should be
involved in presenting the indicator results.
m. Communication:
how and where the formal presentation of the indicators will be carried out.
3. Basic measures: each indicator is
composed of basic and derived measures. This session presents the definition,
description and form of collection of each of the basic measures present in the
formula related to the indicators defined in the guide.
6.
FINAL CONSIDERATIONS
The field study
presented in this study provided the interaction with the industry in several
stages of its development, which increases the quality and relevance of the
work (GONÇALVES; SILVA, 2017; RODRÍGUEZ; KUVAJA; OIVO,
2014), approaching companies of the university whose objectives are always
converging with respect research and production of new solutions.
Like the other process
areas, Measurement and Analysis was observed in practice in the organization.
All expected results have been reviewed and, if necessary, improved in the last
phase. The adherence of the Measurement and Analysis process was evaluated. All
expected results were checked to identify possible weaknesses and opportunities
for improvement.
Finally, it was
possible to observe how the Measurement process was implemented in micro and
small software companies that also work with maintenance. The definition of the
guide was a joint work between each company and the consultancy, and the
indicators could vary between organizations.
Because CMMI® level 2
does not require the process to be defined in detail, measurement was followed
in the guidelines found in the guide book section.
One of the most
interesting points observed was that the indicators to be measured were
constructed in a "personalized" way for each company, varying
according to the business, development process and strategic objectives.
At the end of the
implementation, a set of indicators was generated to support the managers of
the organizations and the 7 analyzed companies were submitted to official
evaluation by SCAMPI A (SCAMPI, 2011), which is the official method indicated
by CMMI Institute (current owner of CMMI®) to evaluate the model regarding the
maturity and capacity of organizations. The method consists of rules and
guidelines that guide the evaluation of the model, obtaining the level 2 certification
of maturity.
As a future work to
this study, we intend to find common points between the deployments in the
companies and their adaptations so that it is possible to propose a standard
base measurement guide for this type of organization (micro and small company
that operates with software maintenance). From this standard guide, the company
can evolve it according to its strategic objectives and will be independent in
the implementation of the process of Measurement and Analysis.
REFERENCES
ABES. Brazilian Software Market
(2018) Scenario and Trend (ABES).
http://central.abessoftware.com.br/Content/UploadedFiles/Arquivos/Dados%202011/ABES-Publicacao-Mercado-2017.pdf
AKINGBEHIN, K. (2008) Baseline-Based
Framework for Continuous Software Process Improvement (CSPI). In Advanced Software Engineering and Its
Applications. ASEA 2008. IEEE, 214–216.
ALORAN, M.; EID, H.; AL-SARAYREH, K.
T. (2015) A High-Quality Software after Maintenance Depend on Effectiveness
measures. In Proceedingsof the
International Conference on Intelligent Information Processing, Security and
Advanced Communication. ACM, 64.
APRIL, A.; ABRAN, A. (2008) Software Maintenance Management: Evalua-
tion and Continuous Improvement, John Wily &Sons. Inc., Hoboken, New
Jersey.
APRIL, A.; HAYES, J. H.; ABRAN, A.;
DUMKE, R. (2005) Software Maintenance Maturity Model (SMmm): the Software Maintenance Process Model. Journal
of Software: Evolution and Process, v. 17, n. 3, p. 197–223.
BASILI, V.; HEIDRICH, J.; LINDVALL,
M.; MUNCH, J.; REGARDIE, M.; TRENDOWICZ, A. (2007). GQMˆ+ Strategies–Aligning
Business Strategies with Software Measurement. In Empirical Software Engineering and Measurement. ESEM 2007. First
International Symposium on. IEEE, p. 488–490.
BERVIAN, P. A.; CERVO, A. L.; SILVA,
R. (2002) Metodologia científica.
São Paulo: Pretence Hall, p. 482–493.
MPS BR. (2011) BR–Melhoria de Processo do Software Brasileiro.
BRASIL, M. A. B.; FONTOURA, L. M.;
SILVA, L. A. L. (2013) Uma Proposta para Melhoria da Qualidade de Processos de
Software com base em MPS. BR. Simpósio
Brasileiro de Qualidade de Software (SBQS).
FERREIRA D. B.; BARCELLOS, M. P.;
SANTOS, G. (2017) A Software Measurement Pattern Language for Measurement
Planning at SPI. Simpósio Brasileiro de
Qualidade de Software (SBQS).
BURGER, S.; HUMMEL, O. (2012)
Applying maintainability oriented soft- ware metrics to cabinsoftware of a
commercial airliner. In Software
Maintenance and Reengineering (CSMR), 16th European Conference on. IEEE, p.
457–460.
CHAPETTA, W. A.; TRAVASSOS, G. H.
(2016) Software Productivity Measurement and Prediction Methods: what can we
tell about them? Simpósio Brasileiro de
Qualidade de Software.
TEAM PRODUCT CMMI. (2010) CMMI® for Development, Version 1.3,
Improving processes for developing better products and services. no.
CMU/SEI-2010-TR-033. Software Engineering Institute.
CORNU, C.; CHAPURLAT, V.; QUIOT, J.;
IRIGOIN, F. (2012) A maturity model for the deployment of Systems Engineering
processes. In Systems Conference (SysCon),
IEEE International. IEEE, p. 1–6.
DEMARCO, T. (1986) Controlling software projects: Management,
measurement, and estimates. Prentice Hall PTR.
DESHARNAIS, J.; APRIL, A. (2010)
Software maintenance productivity and maturity. In Proceedings of the 11th International Conference on Product Focused
Software. ACM, p. 121–125.
DÍAZ-LEY, M.; GARCÍA, F.; PIATTINI,
M. (2010) MIS-PyME software measurement capability maturity model–Supporting
the definition of software measurement programs and capability determination. Advances in Engineering Software v. 41,
n. 10, p. 1223–1237.
DIKICI, A.; TURETKEN, O.; DEMIRORS,
O. (2012) A case study on measuring process quality: Lessons learned. In Software Engineering and Advanced
Applications (SEAA), 38th EUROMICRO Conference on. IEEE, p. 294–297.
EDBERG, D.; IVANOVA, P. (2011)
Embracing or Constraining Change: An Exploration of Methodologies
forMaintaining Software. In System
Sciences (HICSS), 44th Hawaii International Conference on. IEEE, p. 1–10.
GONÇALVES, B. S.; SILVA, E. S.
(2017) O Financiamento Público em Ciência, Tecnologia e Inovação e o
Desenvolvimento Regional. In Congresso
Gestão Negócios TI–CONGENTI, v. 1.
GONÇALVES, T.; OLIVEIRA, K.; KOLSKI,
C. (2017) A study about HCI in practice of interactive system development using
CMMI-DEV. In 29 ème conférence francophone
sur l’Interaction Homme-Machine. 10–p.
GRENNING, J. (2002) Planning poker or how to avoid analysis
paralysis while release planning. Hawthorn Woods: Renaissance Software
Consulting, v. 3, p. 22–23.
GUCEGLIOGLU, A. S.; DEMIRORS, O.
(2011) The application of a new process quality measurement model for software
process improvement initiatives. In Quality Software (QSIC), 11th International Conference Software
Quality. IEEE, p. 112–120.
FERNÁNDEZ-SÁEZ,
A. M.; CHAUDRON, M. R. V.; GENERO, M. (2018) An industrial case study on the
use of UML in software maintenance and its perceived benefits and
hurdles. Empirical Software Engineering, v. 23, n. 6, p. 3281-3345,
2018.
FERREIRA, A. I. F.; SANTOS, G.;
CERQUEIRA, R.; MONTONI, M.; BARRETO, A.; BARRETO, A. O. S.; ROCHA, A. R. (2007)
Applying ISO 9001: 2000, MPS. BR and CMMI to achieve software process maturity:
BL informatica’s pathway. In Proceedings of the 29th international
conference on Software Engineering. IEEE Computer Society, p. 642–651.
JAIN,
P.; SHARMA, A.; AHUJA, L. (2019) A Customized Quality Model for Software
Quality Assurance in Agile Environment. International Journal of
Information Technology and Web Engineering (IJITWE), v. 14, n. 3, p. 64-77.
KAJKO-MATTSSON, M. (2001) Motivating
the corrective maintenance maturity mo- del (CM/sup 3/). In Engineering of Complex Computer Systems.
Proceedings. Seventh IEEE International Conference on. IEEE, p. 112–117.
KAJKO-MATTSSON, M.; SNYGG, J.;
HAMMARGREN, E. (2012) CM 3: Emer- gency problem management-A scenario-based
evaluation. In Information Science and
Digital Content Technology (ICIDT), 8th International Conference on, v. 2.
IEEE, p. 379–386.
KITCHENHAM, B. (2007) Guidelines for
performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE
Technical Report. EBSE. sn.
KIRK, J.; MILLER, M. L; MILLER, M.
L. (1986). Reliability and validity in
qualitative research. v. 1. Sage.
KITCHENHAM, B. A.; PFLEEGER, S. L.;
FENTON, N. (1997). Reply to: Comments on "Towards a Framework for Software
Measurements Validation". IEEE Transactions
on Software Engineering, v. 23, n. 3, p. 189.
KITCHENHAM, B. A.; PFLEEGER, S. L.
(2008) Personal opinion surveys. In Guide
to advanced empirical software engineering. Springer, p. 63–92.
KITCHENHAM, B. A.; PFLEEGER, S. L.;
PICKARD, L. M.; JONES, P. W.; HOAGLIN, D. C.; EMAM, K. E.; ROSENBERG, J. (2002)
Preliminary guidelines for empirical research in software engineering. IEEE Transactions on software engineering,
v. 28, n. 8, p. 721–734.
KÖCHE J. C. (1997) Fundamentos de metodologia científica.
rev. e ampl. Petrópolis, RJ: Vozes.
KURTEL, K.; OZEMRE, M. (2013)
Cohesive software measurement planning framework using ISO standards: a case
study from logistics service sector. Journal
of Software: Evolution and Process, v. 25, n. 7, p. 663–679.
LIN, C.; HUANG, Z. (2009) A flexible
metric-driven framework for software process. In INC, IMS and IDC. NCM’09. Fifth International Joint Conference
on. IEEE, p. 1198–1202.
MATA-LIMA, H.; MORGADO-DIAS, F.;
SILVA, G.; CARRATO, M.; ALCÂNTARA, K.; ALMEIDA, J. A. (2016) A Systematic
Framework for the Design and Implementation of a Quality Management Practice:
The Case of a Consulting Engineering Company. Environmental Quality Management, v. 25, n. 4, p. 49–61.
MONTEIRO, L. F. S.; OLIVEIRA, K. M.
(2010) Defining a catalog of indicators to support process performance
analysis. Journal of Software: Evolution
and Process.
MOREIRA, R. T.; LIMA, G. M.; MACHADO,
B. B.; MARINHO, W. T.; VASCONCELOS, A.; ROUILLER, A. C. (2008) Uma Abordagem
para Melhoria do Processo de Software baseada em Medição. VIII Simpósio Brasileiro de
Qualidade de Software.
MOURA, M. A.; FRANÇA, V. J. A. T.
M.; ROUILLER, A. C. (2015) Implantação do Processo de Medição Aderente ao
Modelo MR-MPS-SW com Foco em Estudo de Tempos em Empresas com Times SCRUM. Simpósio Brasileiro de Qualidade de
Software (SBQS).
MUTAFELIJA, B.; STROMBERG, H. (2003)
Systematic process improvement using ISO
9001: 2000 and CMMI. Artech House.
NETO, J. A. M. (2002) Metodologia científica na era da
informática. Saraiva.
NIESSINK, F.; VLIET, H. V. (2000)
Software maintenance from a service perspective. Journal of Software Maintenance, v. 12, n. 2, p. 103–120.
O'CONNOR, R. V. (2019) Software
Development Process Standards for Very Small Companies. In: Advanced Methodologies and Technologies in
Digital Marketing and Entrepreneurship. IGI Global, p. 681-694.
OKIKE,
E. U. (2018) FQWCOS: A Flexible Model for Measuring Customer Satisfaction on
Software Based Products and Service. Software Engineering, v. 6, n.
4, p. 110.
PEIXOTO, D. C. C.; BATISTA, V. A.;
RESENDE, R. F.; PÁDUA, C. I. P. S. (2010) A Case Study of Software Process
Improvement Implementation. In Software Engineering
and Knowledge Engineering, p. 716–721.
PÉREZ, J.; MENS, T.; KAMSEU, F.
(2013) A pilot study on software quality practices in Belgian industry. In Software Maintenance and Reengineering
(CSMR), 17th European Conference on. IEEE, p. 395–398.
PINO, F. J.; RUIZ, F.; GARCIA, F.;
PIATTINI, M. (2012) A software maintenance methodology for small organizations:
Agile_MANTEMA. Journal of Software:
Evolution and Process, v. 24, n. 8, p. 851–876.
PORT,
D.; TABER, B. (2018) An empirical study of process policies and metrics to
manage productivity and quality for maintenance of critical software systems at
the jet propulsion laboratory. In: Proceedings of the 12th ACM/IEEE
International Symposium on Empirical Software Engineering and Measurement.
ACM. p. 37.
RACHEVA, Z.; DANEVA, M.; BUGLIONE,
L. (2008) Complementing measurements and real options concepts to support
inter-iteration decision- making in agile projects. In Software Engineering and Advanced Applications. SEAA’08. 34th
Euromicro Conference. IEEE, p. 457–464.
RODRÍGUEZ, P.; KUVAJA, P.; OIVO, M.
(2014) Lessons learned on applying design science for bridging the
collaboration gap between industry and acade- mia in empirical software
engineering. In Proceedings of the 2nd
International Workshop on Conducting
Empirical Studies in Industry. ACM, p. 9–14.
ROMBACH, H. D.; ULERY, B. T. (1989) Improving software maintenance through
measurement. Proc. IEEE, v. 77, n. 4,
p. 581–595.
SALVIANO, C. F. (2006) Uma proposta orientada a perfis de
capacidade de processo para evolução da melhoria de processo de software.
SCAMPI, T. U. (2011) Standard CMMI Appraisal Method for Process
Improvement (SCAMPI) A, Version 1.3: Method Definition Document.
SCHRETTNER, L.; FÜLÖP, L. J., KISS,
A.; GYIMÓTHY, T. (2012) Software quality model and framework with applications
in industrial context. In Software
Maintenance and Reengineering (CSMR), 16th European Conference on. IEEE, p.
453–456.
SCHWABER, K. (1997) Scrum development process. In Business
object design and implementation. Springer, p. 117–134.
SEBRAE (2017) SEBRAETEC - Serviços em Inovação e Tecnologia.
http://www.sebrae.com.br/sites/PortalSebrae/
Programas/sebraetec-inovar-no-seu-negocio-pode-ser-facil.
SOUTHEKAL, P. H.; LEVIN, G. (2011)
Formulation and empiri- cal validation of a gqm based measurement framework. In
Empirical Software Engineering and
Measurement (ESEM), 2011 International Symposium on. IEEE, p. 404–413.
Evidências sobre o desem- penho das empresas que adotaram o modelo MPS-SW.
Campinas, Brazil: Softex.
STARON, M.; MEDING, W.; KARLSSON,
G.; NILSSON, C. (2011) Developing measurement systems: an industrial case
study. Journal of Software: Evolution
and Process, v. 23, n. 2, p. 89–107.
SUKARIE-NETO, J. (2018) Mercado Brasileiro de Software – Panorama e
Tendências – 2017 (ABES).
http://central.abessoftware.com.br/Content/UploadedFiles/Arquivos/Dados%202011/ABES-EstudoMercadoBrasileirodeSoftware2018_ResumidaIngles.pdf
TRAVASSOS, G. H.; KALINOWSKI, M. (2014)
iMPS.
UNTERKALMSTEINER, M.; GORSCHEK, T.;
ISLAM, A. K. M.; CHENG, C. K.; PERMADI, R. B.; FELDT, R. (2014) A conceptual
framework for SPI evaluation. Journal of
Software: Evolution and Process, v. 26, n. 2, p. 251–279.
WALLACE, L. G.; SHEETZ, S. D. (2014)
The adoption of software measures: A
technology acceptance model (TAM) perspective. Information & Management,
v. 51, n. 2, p. 249–259.
YIN R. K. (2015) Estudo de Caso: Planejamento e Métodos.
Bookman editora.
YU, B.; CONG, G.; NING, L.; JIANG,
H.; WANG, X. (2009) The Model of Software Process Measurement and Improvement
Driven by Project Performance. In Computer Network and Multimedia Technology,
2009. CNMT 2009. International Symposium
on. IEEE, p. 1–4.