THE MEASUREMENT PROCESS IN MICRO AND SMALL SOFTWARE MAINTENANCE COMPANIES: EMPIRICAL STUDY

 

Victor José Aguiar Teixeira de Melo Fran

State University of Maringá, Brazil

E-mail: victor.jatmf@gmail.com

 

Gislaine Camila Lapasini Leal

State University of Maringá, Brazil

E-mail: gclleal@uem.br

 

Renato Balancieri

State University of Maringá, Brazil

E-mail: renatobalancieri@gmail.com

 

Ana Cristina Rouiller

Universidade Federal Rural de Pernambuco, Brazil

E-mail: anarouiller@gmail.com.br

 

Submission: 3/26/2019

Accept: 9/19/2019

 

ABSTRACT

With the dissemination of quality models and the need for measurement-based management for software organizations, a variety of approaches and models are emerging to support organizations on this journey. According to a Systematic Review of Literature that selected 30 of 692 papers returned from search sources, it was noticed the need for guidelines and guides regarding the measurement process in a specific sector: micro and small companies that work with software maintenance. In this work, an empirical study was carried out to understand how the measurement process aligned with quality models can be implemented in these organizations. The field study followed the implementation of an improvement program based on CMMI® level 2 in 7 Brazilian companies from Maringá, Paraná. Finally, it was found that the process for the definition of a measurement guide can be common to a specific group of companies, taking steps and guidelines for its execution, always aligned with the organization’s strategy, giving top management an objective view work and process, supporting decision making.

Keywords: Measurement guide; Software process improvement; Software maintenance; CMMI; Metrics

1.       INTRODUCTION

            Software Process Improvement (SPI) has received considerable attention in recent years, both in academia and industry. A reflection of this is the popularization of maturity models that have been disseminated and applied in software organizations due to factors related to competitiveness, product quality, economy and efficiency (BRASIL; FONTOURA; SILVA, 2013; TRAVASSOS; KALINOWSKI, 2014; O'CONNOR, 2019), such models have absorbed areas for the measurement process. In Brazil, this phenomenon can be observed through data on the number of official evaluations in software maturity models such as the Capability Maturity Model Integration for Development (CMMI®-DEV) (CMMI, 2010) with 71 current certifications and the Reference Model for Software Process Improvement (MR-MPS-SW) (MPS.BR, 2011) with 161 current certifications at various levels and constellations (people, software and services).

            Model-based SPI encompasses the development of software from conception to final delivery, through all stages of the software lifecycle. Despite the relevance of maintenance in this life cycle, it did not receive the same degree of attention as the other stages of development (APRIL; HAYES; ABRAN; DUMKE, 2005; EDBERG; IVANOVA, 2011; PINO; et al., 2012; FERNÁNDEZ-SÁEZ; CHAUDRON; GENERO, 2018). Historically, software development has been far more representative than maintenance in most organizations. However, this scenario is changing, given the organizations' effort to make the most of their software development investments, keeping the product operating as long as possible (EDBERG; IVANOVA, 2011)

            It is perceived that maintenance is an essential part of the software life cycle and has been gaining increasing attention among industry and researchers, according to the Brazilian Association of Software Companies (ABES) (ABES, 2017), 55.4% of the investments, in millions of dollars, for the Brazilian software and services market, adds the stage of software maintenance and evolution. Other specific characteristics of the maintenance context are: continuous and repetitive nature, focus on individual response to change requests, and uncertainty of new scope changes (APRIL; ABRAN, 2008, PORT; TABER, 2018). Thus, the improvement of the process based on specific reference models for the development of a software project is not totally adequate to the specifics of the software maintenance and evolution context (KAJKO-MATTSSO, 2001; KAJKO-MATTSSON; SNYGG; HAMMARGREN, 2012; NIESSINK; VLIET, 2000; PINO et al., 2012) and has caused damages to organizations. The problem may become even worse when observed in the micro and small Brazilian companies sector. According to the Brazilian Software Association (ABES), in a survey conducted in 24 Brazilian states, of the software companies, 86% of them are classified as micro and small companies (SUKARIE-NETO, 2018). This represents a significant portion of the sector.

            Considering this scenario, the main challenge faced by these organizations is the limitation of resources: people, time and budget. These limitations lead to insufficient processes, methodologies, guidelines, tools and documentation required for software maintenance. Also in the context of some micro-software organizations, a single product may be the basis of the organization's business model, and new product development activity may be rare or nonexistent.

            The Knowledge Engineering Body of Knowledge (SWEBOK) (EDBERG; IVANOVA, 2011) presents some of the issues related to software maintenance, grouped into four categories: i) Technical issues; ii) management issues; iii) Estimate of Costs; and iv) measurement, the entities that can be subjected to measurement are process, resource and product. If applied correctly, the measurement has great potential to contribute to a better understanding of the maintenance and evolution of software and to motivate necessary changes in processes and methodologies (ROMBACH; ULERY, 1989).

            However, measuring quality, productivity, and other aspects of software maintenance within an organization is a difficult task (DESHARNAIS; APRIL, 2010, OKIKE, 2018), mainly because the nature of the work differs from software development by design (ALORAN; EID; AL-SARAYREH, 2015), especially by the characterization of its work continuous and on demand.

            Therefore, the objective of this work is to observe how the implementation of a measurement process within a Software Process Improvement (SPI) program based on quality models in micro and small companies, which also work with software maintenance, takes place in practice.

2.       LITERATURE REVIEW

            According to DeMarco (DEMARCO, 1986), one cannot control what cannot be measured, which demonstrates the need for measurement with respect to the Software Engineering context. Good management based on indicators supposes the possibility of predicting the future behavior of software products and processes, from the moment we have reliable and appropriate information. In this way the measurement becomes an important practice since one cannot predict what cannot be measured (KITCHENHAM; PFLEEGER; FENTON, 1997). With this, several quality models and guidelines for improvement were created based on the measurement process.

            SPI can be implemented ad hoc, but process improvement implemented systematically, based on models, standards or approaches, has been shown to be more efficient and with better results for organizations (MATA-LIMA et al., 2016; MUTAFELIJA; STROMBERG, 2003; SALVIANO 2006).

            Among the existing quality models, the most widely disseminated in the context of micro and small Brazilian companies that also work with the software maintenance stage are Capability Maturity Model Integration (CMMI®) (GONÇALVES; OLIVEIRA; KOLSKI, 2017) and MR-MPS-SW.

            These quality models are intended to help companies evaluate current processes and provide guidelines for driving improvements in them based on data analysis, corrective action and decision making (CORNU et al., 2012; JAIN; SHARMA; AHUJA, 2019). For this work the CMMI® model will be used to validate the field study due to its international recognition. In CMMI®, the process area that represents and contains measurement practices is called Measurement and Analysis.

2.1.          The CMMI® Measurement Process

            The goal of the Measurement and Analysis process area is to develop and sustain the measurement capability that is used to support information management needs (CMMI, 2010). This general purpose is divided into two specific objectives:

(1)            Align measurement objectives and activities with identified information needs and objectives;

(2)            Provide measurement results that address the needs and objectives of identified information. This process area, as well as the CMMI® model in all its definitions, presents only what is charged, what the model expects from the organization to consider the process area implemented, ie "what to do" and not "how to make".

            The Measurement and Analysis process area, as well as the others, is composed of generic objectives (SG) and specific objectives (SP):

·                 SG1. Align Measurement and Analysis Activities:

o       SP1.1. Establish Measurement Objectives;

o       SP1.2. Specify Measures;

o       SP1.3. Specify Data Collection and Storage Procedures;

o       SP1.4. Specify Analysis Procedures.

·                 SG2. Provide Measurement Results:

o       SP2.1. Obtain Measurement Data;

o       SP2.2. Analyze Measurement Data;

o       SP2.3. Store Data and Results;

o       SP2.4. Communicate Results.

            In addition to the practices of the Measurement and Analysis process areas, at the maturity level 2, we have the generic objective (GP2.8), which emphasizes that the processes must be monitored and controlled, an activity that also involves measurement at this level. Because it is a generic practice, it focuses on all process areas of maturity level 2.

3.       RELATED WORK

            The related works were identified from a Systematic Review of Literature (SRL), which is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, topic area or phenomenon of interest (KITCHENHAM, 2007).

            From an SRL performed by the authors, with 30 papers selected from 692 papers returned from automatic search sources (IEEE Explore, ACM Digital Library, Science Direct, Scopus and Inspec / Compendex) and manuals (14 sources from conferences and periodicals relevant to the area of research) in the period 2007-2017, it was possible to understand the state of the art and it was noticeable the effort of some researchers to define quality models adherent to software maintenance, especially with regard to the measurement process in practice (CHAPETTA; TRAVASSOS, 2016; UNTERKALMSTEINER et al., 2014).

            The work analyzed shows, as of great relevance to the organization, the implementation of a measurement process to evaluate the work performance and evolution of the process improvement, always highlighting its organizational benefits focused on improving the quality of product delivery and / or service. Several authors (AKINGBEHIN, 2008; FERREIRA; BARCELLOS; SANTOS, 2017; DÍAZ-LEY; GARCÍA; PIATTINI, 2010; KURTEL; OZEMRE, 2013; LIN; HUANG, 2009; SCHRETTNER et al., 2012; SOUTHEKAL; LEVIN, 2011; UNTERKALMSTEINER et al.) point out that there are not many options for defined standards and models that guide the definition and maintenance of a measurement process in organizations. All of them proposed a measurement-based process quality framework, some of which are specific to software maintenance such as Modus (MOREIRA, 2008), PPQM (GUCEGLIOGLU; DEMIRORS, 2011), MIS-SMME (DÍAZ-LEY; GARCÍA; PIATTINI, 2010), among others.

            According to case studies (BURGER; HUMMEL, 2012; DIKICI; TURETKEN; DEMIRORS, 2012; FERREIRA et al., 2007; MONTEIRO; OLIVEIRA, 2010; PEIXOTO et al., 2010; YU et al., 2009), it can be seen that the process measurement applied to the organizations can be used as support in increasing performance and productivity, as well as helping in actions for process improvement and maturity of these companies. Some authors have focused directly on improving decision making and strategic direction (BASILI et. al., 2007; RODRÍGUEZ.; KUVAJA; OIVO, 2014).

            It is also agreed between the authors of the works, the need for some quality model to evaluate the proposed indicators, with CMMI® being the most cited model among them. Still to aggregate the results, an analysis was performed on the of the organizations observed in the studies, with only 3% referring to micro-enterprises (given that this proves the need for research in this sector), and more than 50% of the studies at least mentioned the size of the organization, generalizing the indicators, the measurement process has to adhere to the size of the organization, cannot be generalized (DÍAZ-LEY; GARCÍA; PIATTINI, 2010; FERREIRA et al., 2007; MOURA; FRANÇA; ROUILLER, 2015; PÉREZ; MENS; KAMSEU, 2013). The results obtained in this systematic review were important for the knowledge of how the literature addresses the measurement process in the context of SPI for software maintenance companies.

4.       RESEARCH METHOD

            After SRL performed by the authors, a field study was planned and conducted. A field study or study is a case study of the real world without necessarily having direct intervention by the researcher / observer responsible (KITCHENHAM et al. 2002).

            When conducting a field study it is important to keep in mind that this type of specific study seeks to deepen a given reality, it is basically carried out by means of direct observation of the activities of the group studied and, if necessary, interviews with the purpose of capturing explanations and interpretations about the studied context. This type of study differs from the case study, since the case study can be defined as an exhaustive, deep and extensive study of one or a few units, empirically verifiable, in a way that allows its comprehensive and detailed knowledge (YIN, 2015).

            Field research has the purpose of observing facts and phenomena of the way they occur in reality and also collecting data on the elements observed and then analyzing and interpreting them, based on a solid and well-founded theoretical foundation, in order to understand and explain the problem that is the object of study of the research. The field study does not provide specific techniques for data collection, and they may vary according to the needs of the work developed.

            The field study performed in this work is characterized as an exploratory and descriptive study where the researcher analyzes, observes, records and correlates aspects, phenomena or variables that involve facts, without manipulating them. The investigation takes place without the interference of the researcher, where he only "seeks to discover, with the possible precision, how phenomena occur, their relation and connection with others, and also their nature and their characteristics" (BERVIAN; CERVO; SILVA, 2002). Being also exploratory, it aims to provide greater familiarity with the object of study. It is necessary to define a research process that identifies the nature of the phenomenon and points out the essential characteristics that one wants to study (KÖCHE, 1997).

            Characteristics of the descriptive research are spontaneity (the researcher only observes the phenomenon), naturality (the facts are studied in their natural habitat) and a high degree of generalization, which dispenses with great concern about the representativeness of the selected sample. In addition, it is a qualitative study, we sought to verify a phenomenon through observation and study (KIRK; MILLER; MILLER,1986).

            The field research conducted for the execution of this work followed the steps that were adapted from the method proposed by Neto (NETO, 2002):

(1)            definition of the research objective;

(2)            choosing the context that the study would run;

(3)            selection of companies to be observed;

(4)            observation of the context that was defined for the execution of the research;

(5)            data collection through interviews and analysis of available documents; and

(6)            analysis and conclusion of the aspects observed in field research.

5.       FIELD STUDY

5.1.          Contextualization

            This field research was conducted in the city of Maringá-PR, which is a regional hub in software development, and has the presence of a Local Productive Arrangement of Software (APL). In addition, Maringá has one of the highest percentages of companies evaluated in CMMI® nationally, in relation to the local universe of companies, being behind only of São Paulo and Recife, according to Software By Maringá and Commercial and Business Association of Maringá (ACIM). These characteristics make the proposed theme and the study applied in the region relevant.

            The study was conducted during the implementation of level 2 CMMI® quality model activity activity which lasted about 11 months. In spite of the presence of the other process areas present in the model, the observation stopped the activities related to the Measurement and Analysis process, given the scope of this work.

            The SPI implementation project aligned with CMMI® was a partially subsidized initiative by SEBRAETEC (SEBRAETEC), a national program of the Brazilian Micro and Small Business Support Service (SEBRAE), which encourages and supports initiatives to improve processes and evaluations in national and international quality models for software companies (SEBRAE, 2017).

            The improvement project was carried out jointly with SWQuality Consultoria e Sistemas, a Brazilian company recognized nationally and internationally for its successful work with SPI in the most diverse models and business sectors, also accredited by the CMMI Institute as an official evaluation institution of the model.

            The purpose of this observation is to understand how the measurement process aligned with quality models can be implemented in micro or small companies that work with software maintenance.

5.2.          Sample Selection

            The method of selection of the sample adopted is non-probabilistic and for convenience. The sample for convenience refers to the acquisition of answers by companies that have availability and are willing to collaborate (KITCHENHAM; PFLEEGER, 2008).

            The companies were selected using criteria defined according to the study objectives, where there is no knowledge of the probability that a given population element will be selected. To select the samples, companies were chosen that, at the time of the research, accepted to be observed and were available for observation of the researcher.

            To reduce the chance of choosing samples that did not contribute to the study objective, the following criteria were taken into account:

·       micro or small businesses;

·       perform maintenance of your software products;

·       be implementing the software process improvement aligned to some quality model that focuses on measurement;

·       be implementing the SPI with the support of some specialized external consulting; and

·       have the objective of undergoing a formal and official evaluation of the quality model in question.

            Seven micro and small Brazilian companies from the Maringá region of Paraná were selected. All observed organizations had as main activity the maintenance and evolution of their software products. Some of them used agile practices, and two of them followed the cascade model of software development, they had 1 to 3 teams, which were composed of 4 to 8 professionals who worked throughout the maintenance cycle.

            For all organizations observed, the main product was an Integrated Enterprise Management System (EMS), which varied the market niche that the company operated, ranging from stalls to medical laboratories, also some of them had government software.

5.3.          Study Execution

            The implementation of the process improvement and definition of the measurement guide in the organizations was given by biweekly visits where the consulting company trained with formal orientations or from mentoring the employees of the companies and left activities for them to execute. These activities would be reviewed on the next visit and the cycle would be repeated until the end of the project.

            This project was divided in phases as it is possible to see below, the visits of the consultancy in the companies were accompanied by the researcher / observer so that it could carry out its analysis.

·       Phase 1 - Diagnosis: At this stage, the consulting team carried out a diagnosis of the companies with the objective of identifying to what degree the organization adhered to the processes required by CMMI®-DEV level 2, to understand how the work was carried out in the day to day of the company, identification of problems and alignment of expectations with those involved about the improvement program. Based on the information collected, the consultancy carried out high-level planning considering the goals established for the project.

·       This phase took place through separate job interviews: directors, managers, architects, requirements analysts and developers were heard.

·       Phase 2 - Kick-off and Training. The improvement program was presented to all employees of the company with the goal of leveling expectations and informing the kick-off of the project. In this presentation, doubts were clarified and guidance was given regarding the structuring of the environment conducive to the accomplishment of the works.

·       Training was also conducted in order to level the employees of the companies directly involved in the CMMI®-DEV implementation project, in this training the model was presented, as well as its process areas, and the general objective of each of them, as well as also a high-level schedule of the implementation of these processes, with milestones to review the progress and possible planning of the other activities to be developed.

·       Phase 3 - Reformulation of the Software Development Process: After mapping, the current software development and maintenance processes were reformulated according to the improvement program, good practices and new procedures were added using the SCRUM agile method (SCHWABER, 1997) and model guidelines as a reference.

·       A close follow-up of the consulting team and the team of developers with training and mentoring were carried out to better transpose the work method with gradual changes. Adaptations in the SCRUM method were performed according to the need and limitation of each organization (sprint size, distribution of roles and responsibilities, etc.). Parallel to this phase, Phase 4 has already begun.

·       Phase 4 - Deployment of Management Process Areas: At this stage, emphasis was placed on the process areas related to the management of the requirements, work and the monitoring and follow-up of the day to day. Processes were defined to make these activities as fluid as possible. In the monitoring and control stage, indicators proposed by SCRUM were initially used, such as Team speed. The number of hours began to be followed to serve as a basis in the calculation of the other indicators to be defined.

·       Phase 5 - Support Process Area Deployment: At this stage, the areas of Process Management Configuration, Product Quality Assurance and Process and Measurement and Analysis were worked directly. For these process areas, the work was structured and adapted to the reality of each of the organizations. Strategies to meet the requirements of the model were defined without hurting the agility of work in companies. These processes were implemented, monitored and adapted whenever necessary.

·       Phase 6 - Institutionalization of Processes: At this stage the processes were institutionalized in all the companies' projects. This stage began next to step 3 and followed gradually until the completion of the implementation of the processes adhering to the proposed quality model.

Durante a institucionalização, o uso dos processos foi monitorado por meio de indicadores e possíveis desvios e melhorias foram identificados e implementados. Essa etapa teve como principal objetivo observar como a empresa se comportava e executava os processos definidos sem interferência direta da equipe de consultoria, que intervia somente onde era necessário.

·       Phase 7 - Preparation for Evaluation. Finally, a simulated evaluation was carried out, indicating whether companies could undergo an official evaluation of the Standard CMMI Appraisal Method for Process Improvement SCAMPI A (SCAMPI, 2011), or whether some adjustments would still be necessary. To conclude, a workshop was held and organizations presented their results to each other by adding value to organizations in the form of an exchange of experiences.

5.4.          Discussion of the Field Study

            During the execution of the field survey, during each of the phases, notes were made to record the observations that made analogy to the process of process improvement with the measurement process. Whenever necessary, non-systematic questions were asked both for the organization's employees and for the members of the consulting firm.

Also during observation of the improvement project phases, each goal of the CMMI® Level 2 Measurement and Analysis process area was related to the steps to check if the work performed meets the needs of the models. This relationship can be seen in Table 1.

Table 1: Relationship between the phases of the implementation and the expected results of the Measurement and Analysis in CMMI®

 

CMMI

Phase 1

-

Phase 2

S.P. 1.1

Phase 3

S.P. 1.2

Phase 4

S.P. 1.2

Phase 5

S.P. 1.1, S.P. 1.2 e S.P. 1.3, S.P. 1.4

Phase 6

S.P. 2.1, S.P. 2.2 e S.P. 2.3, S.P. 2.4

Phase 7

S.G. 1 e S.G. 2

            Phase 1 did not correspond directly to any of the practices, but was necessary for the start of the improvement project. In the diagnostic carried out were collected some strategic objectives of the managers and some information needs that later would serve as input for the definition of the indicators, since, according to the existing approaches to definition of measurement process, they should be aligned.

            Phases 2, 3, and 4 are more focused on process definitions and redefinitions, where the objectives of measurement, measurement, data collection and storage, and analysis criteria have been defined, validated and consolidated in Phase 5. In these phases the process was also executed, but its focus was on the alignment of the Measurement and Analysis activities.

            Phase 6 had as main objective the maturation of the execution of the previously made definitions, such as obtaining, analyzing and storing the data and its communication to the involved ones, being the focus in providing the measurement results. Finally, Phase 7 encompasses all specific practices and achieves specific objectives.

            The Measurement and Analysis process, as well as the other CMMI®-DEV Level 2 processes, were briefly addressed in employee leveling training, ensuring their commitment and engagement in building new processes in organizations.

            With the management of the teams following the SCRUM method, a time-box was defined that, together with the teams' timekeeping, allowed a periodicity for the collection of data that would generate the indicators, initially guided by method metrics such as team speed, based on the ability of story points with the technique of estimative Planning Poker (GRENNING, 2002). By dividing management into short and fixed time cycles, SCRUM assisted in standardizing the work.

            In the implementation of the management processes, the development management was carried out following a previous planning, with organization and prioritization of the demands, where the objectives and parameters and goals of monitoring were defined by the responsible one. From these parameters, it was possible to observe deviations in the defined goal, making it possible to take corrective actions whenever necessary.

            The Measurement and Analysis process was worked with greater emphasis in the implementation of the support process areas, having all its objectives worked directly, seeking strategies to satisfy all the expected results of the reference model. Based on the measurement objectives and needs already identified and the metrics created, a Measurement Guide was established where this information was recorded and related to the indicators and steps for the execution of the process were described.

            The Measurement Guide also defined the collection and storage procedures to analyze this information and how the entire process should be followed, with responsible and well-defined steps. The strategy for the execution of the process was also established, which includes collecting and analyzing the data, storing it and communicating the results to those involved and interested. It should be noted that this guide and its indicators should be reviewed periodically, since the strategic needs of companies may vary over time.

            In addition to a document name, company name and document maintainer header, this guide has the following sessions:

1.   Objectives and Need for Information: A relationship between each of the defined indicators, which strategic objective it helped measure or respond to, and why that information was necessary for the organization.

2.   Indicator Specification: In this section, the indicators were listed and for each of them the following information was given:

a.        Name: Indicator name and acronym;

b.        Description: a brief description of the purpose of the indicator;

c.        Target: value or expected value range to maintain the indicator. It may also contain information about where you want to go.

d.        Periodicity of Analysis: informed of how long this indicator should be analyzed and presented to those involved.

e.        Formula: What is the formula for calculating the indicator?

f.         Unit of measure: which unit of measurement was used.

g.        Collection Procedure: a step-by-step of how and where to fetch the information needed to calculate the indicator.

h.        Presentation Form: how the indicator would be displayed: histogram, pie chart, table, among others.

i.         Analysis Criterion: This field was based on the goal, and deviation ranges were defined and classified as "OK", "Alert", "Critical" and "Attention". For each of the classifications, an action was defined and should be taken for the normalization of the indicator.

j.         Storage: where the indicator will be stored after collected for reuse and history.

k.        Responsible: Indicates the person or role responsible for collecting and presenting the indicator.

l.         Target Audience: indicates the roles that should be involved in presenting the indicator results.

m.      Communication: how and where the formal presentation of the indicators will be carried out.

3.   Basic measures: each indicator is composed of basic and derived measures. This session presents the definition, description and form of collection of each of the basic measures present in the formula related to the indicators defined in the guide.

6.       FINAL CONSIDERATIONS

            The field study presented in this study provided the interaction with the industry in several stages of its development, which increases the quality and relevance of the work (GONÇALVES; SILVA, 2017; RODRÍGUEZ; KUVAJA; OIVO, 2014), approaching companies of the university whose objectives are always converging with respect research and production of new solutions.

            Like the other process areas, Measurement and Analysis was observed in practice in the organization. All expected results have been reviewed and, if necessary, improved in the last phase. The adherence of the Measurement and Analysis process was evaluated. All expected results were checked to identify possible weaknesses and opportunities for improvement.

            Finally, it was possible to observe how the Measurement process was implemented in micro and small software companies that also work with maintenance. The definition of the guide was a joint work between each company and the consultancy, and the indicators could vary between organizations.

            Because CMMI® level 2 does not require the process to be defined in detail, measurement was followed in the guidelines found in the guide book section.

            One of the most interesting points observed was that the indicators to be measured were constructed in a "personalized" way for each company, varying according to the business, development process and strategic objectives.

            At the end of the implementation, a set of indicators was generated to support the managers of the organizations and the 7 analyzed companies were submitted to official evaluation by SCAMPI A (SCAMPI, 2011), which is the official method indicated by CMMI Institute (current owner of CMMI®) to evaluate the model regarding the maturity and capacity of organizations. The method consists of rules and guidelines that guide the evaluation of the model, obtaining the level 2 certification of maturity.

            As a future work to this study, we intend to find common points between the deployments in the companies and their adaptations so that it is possible to propose a standard base measurement guide for this type of organization (micro and small company that operates with software maintenance). From this standard guide, the company can evolve it according to its strategic objectives and will be independent in the implementation of the process of Measurement and Analysis.

REFERENCES

ABES. Brazilian Software Market (2018) Scenario and Trend (ABES). http://central.abessoftware.com.br/Content/UploadedFiles/Arquivos/Dados%202011/ABES-Publicacao-Mercado-2017.pdf

AKINGBEHIN, K. (2008) Baseline-Based Framework for Continuous Software Process Improvement (CSPI). In Advanced Software Engineering and Its Applications. ASEA 2008. IEEE, 214–216.

ALORAN, M.; EID, H.; AL-SARAYREH, K. T. (2015) A High-Quality Software after Maintenance Depend on Effectiveness measures. In Proceedingsof the International Conference on Intelligent Information Processing, Security and Advanced Communication. ACM, 64.

APRIL, A.; ABRAN, A. (2008) Software Maintenance Management: Evalua- tion and Continuous Improvement, John Wily &Sons. Inc., Hoboken, New Jersey.

APRIL, A.; HAYES, J. H.; ABRAN, A.; DUMKE, R. (2005) Software Maintenance Maturity Model (SMmm): the Software Maintenance Process Model. Journal of Software: Evolution and Process, v. 17, n. 3, p. 197–223.

BASILI, V.; HEIDRICH, J.; LINDVALL, M.; MUNCH, J.; REGARDIE, M.; TRENDOWICZ, A. (2007). GQMˆ+ Strategies–Aligning Business Strategies with Software Measurement. In Empirical Software Engineering and Measurement. ESEM 2007. First International Symposium on. IEEE, p. 488–490.

BERVIAN, P. A.; CERVO, A. L.; SILVA, R. (2002) Metodologia científica. São Paulo: Pretence Hall, p. 482–493.

MPS BR. (2011) BR–Melhoria de Processo do Software Brasileiro.

BRASIL, M. A. B.; FONTOURA, L. M.; SILVA, L. A. L. (2013) Uma Proposta para Melhoria da Qualidade de Processos de Software com base em MPS. BR. Simpósio Brasileiro de Qualidade de Software (SBQS).

FERREIRA D. B.; BARCELLOS, M. P.; SANTOS, G. (2017) A Software Measurement Pattern Language for Measurement Planning at SPI. Simpósio Brasileiro de Qualidade de Software (SBQS).

BURGER, S.; HUMMEL, O. (2012) Applying maintainability oriented soft- ware metrics to cabinsoftware of a commercial airliner. In Software Maintenance and Reengineering (CSMR), 16th European Conference on. IEEE, p. 457–460.

CHAPETTA, W. A.; TRAVASSOS, G. H. (2016) Software Productivity Measurement and Prediction Methods: what can we tell about them? Simpósio Brasileiro de Qualidade de Software.

TEAM PRODUCT CMMI. (2010) CMMI® for Development, Version 1.3, Improving processes for developing better products and services. no. CMU/SEI-2010-TR-033. Software Engineering Institute.

CORNU, C.; CHAPURLAT, V.; QUIOT, J.; IRIGOIN, F. (2012) A maturity model for the deployment of Systems Engineering processes. In Systems Conference (SysCon), IEEE International. IEEE, p. 1–6.

DEMARCO, T. (1986) Controlling software projects: Management, measurement, and estimates. Prentice Hall PTR.

DESHARNAIS, J.; APRIL, A. (2010) Software maintenance productivity and maturity. In Proceedings of the 11th International Conference on Product Focused Software. ACM, p. 121–125.

DÍAZ-LEY, M.; GARCÍA, F.; PIATTINI, M. (2010) MIS-PyME software measurement capability maturity model–Supporting the definition of software measurement programs and capability determination. Advances in Engineering Software v. 41, n. 10, p. 1223–1237.

DIKICI, A.; TURETKEN, O.; DEMIRORS, O. (2012) A case study on measuring process quality: Lessons learned. In Software Engineering and Advanced Applications (SEAA), 38th EUROMICRO Conference on. IEEE, p. 294–297.

EDBERG, D.; IVANOVA, P. (2011) Embracing or Constraining Change: An Exploration of Methodologies forMaintaining Software. In System Sciences (HICSS), 44th Hawaii International Conference on. IEEE, p. 1–10.

GONÇALVES, B. S.; SILVA, E. S. (2017) O Financiamento Público em Ciência, Tecnologia e Inovação e o Desenvolvimento Regional. In Congresso Gestão Negócios TI–CONGENTI, v. 1.

GONÇALVES, T.; OLIVEIRA, K.; KOLSKI, C. (2017) A study about HCI in practice of interactive system development using CMMI-DEV. In 29 ème conférence francophone sur l’Interaction Homme-Machine. 10–p.

GRENNING, J. (2002) Planning poker or how to avoid analysis paralysis while release planning. Hawthorn Woods: Renaissance Software Consulting, v. 3, p. 22–23.

GUCEGLIOGLU, A. S.; DEMIRORS, O. (2011) The application of a new process quality measurement model for software process improvement initiatives. In Quality Software (QSIC), 11th International Conference Software Quality. IEEE, p. 112–120.

FERNÁNDEZ-SÁEZ, A. M.; CHAUDRON, M. R. V.; GENERO, M. (2018) An industrial case study on the use of UML in software maintenance and its perceived benefits and hurdles. Empirical Software Engineering, v. 23, n. 6, p. 3281-3345, 2018.

FERREIRA, A. I. F.; SANTOS, G.; CERQUEIRA, R.; MONTONI, M.; BARRETO, A.; BARRETO, A. O. S.; ROCHA, A. R. (2007) Applying ISO 9001: 2000, MPS. BR and CMMI to achieve software process maturity: BL informatica’s pathway. In Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, p. 642–651.

JAIN, P.; SHARMA, A.; AHUJA, L. (2019) A Customized Quality Model for Software Quality Assurance in Agile Environment. International Journal of Information Technology and Web Engineering (IJITWE), v. 14, n. 3, p. 64-77.

KAJKO-MATTSSON, M. (2001) Motivating the corrective maintenance maturity mo- del (CM/sup 3/). In Engineering of Complex Computer Systems. Proceedings. Seventh IEEE International Conference on. IEEE, p. 112–117.

KAJKO-MATTSSON, M.; SNYGG, J.; HAMMARGREN, E. (2012) CM 3: Emer- gency problem management-A scenario-based evaluation. In Information Science and Digital Content Technology (ICIDT), 8th International Conference on, v. 2. IEEE, p. 379–386.

KITCHENHAM, B. (2007) Guidelines for performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE Technical Report. EBSE. sn.

KIRK, J.; MILLER, M. L; MILLER, M. L. (1986). Reliability and validity in qualitative research. v. 1. Sage.

KITCHENHAM, B. A.; PFLEEGER, S. L.; FENTON, N. (1997). Reply to: Comments on "Towards a Framework for Software Measurements Validation". IEEE Transactions on Software Engineering, v. 23, n. 3, p. 189.

KITCHENHAM, B. A.; PFLEEGER, S. L. (2008) Personal opinion surveys. In Guide to advanced empirical software engineering. Springer, p. 63–92.

KITCHENHAM, B. A.; PFLEEGER, S. L.; PICKARD, L. M.; JONES, P. W.; HOAGLIN, D. C.; EMAM, K. E.; ROSENBERG, J. (2002) Preliminary guidelines for empirical research in software engineering. IEEE Transactions on software engineering, v. 28, n. 8, p. 721–734.

KÖCHE J. C. (1997) Fundamentos de metodologia científica. rev. e ampl. Petrópolis, RJ: Vozes.

KURTEL, K.; OZEMRE, M. (2013) Cohesive software measurement planning framework using ISO standards: a case study from logistics service sector. Journal of Software: Evolution and Process, v. 25, n. 7, p. 663–679.

LIN, C.; HUANG, Z. (2009) A flexible metric-driven framework for software process. In INC, IMS and IDC. NCM’09. Fifth International Joint Conference on. IEEE, p. 1198–1202.

MATA-LIMA, H.; MORGADO-DIAS, F.; SILVA, G.; CARRATO, M.; ALCÂNTARA, K.; ALMEIDA, J. A. (2016) A Systematic Framework for the Design and Implementation of a Quality Management Practice: The Case of a Consulting Engineering Company. Environmental Quality Management, v. 25, n. 4, p. 49–61.

MONTEIRO, L. F. S.; OLIVEIRA, K. M. (2010) Defining a catalog of indicators to support process performance analysis. Journal of Software: Evolution and Process.

MOREIRA, R. T.; LIMA, G. M.; MACHADO, B. B.; MARINHO, W. T.; VASCONCELOS, A.; ROUILLER, A. C. (2008) Uma Abordagem para Melhoria do Processo de Software baseada em Medição. VIII Simpósio Brasileiro de Qualidade de Software.

MOURA, M. A.; FRANÇA, V. J. A. T. M.; ROUILLER, A. C. (2015) Implantação do Processo de Medição Aderente ao Modelo MR-MPS-SW com Foco em Estudo de Tempos em Empresas com Times SCRUM. Simpósio Brasileiro de Qualidade de Software (SBQS).

MUTAFELIJA, B.; STROMBERG, H. (2003) Systematic process improvement using ISO 9001: 2000 and CMMI. Artech House.

NETO, J. A. M. (2002) Metodologia científica na era da informática. Saraiva.

NIESSINK, F.; VLIET, H. V. (2000) Software maintenance from a service perspective. Journal of Software Maintenance, v. 12, n. 2, p. 103–120.

O'CONNOR, R. V. (2019) Software Development Process Standards for Very Small Companies. In: Advanced Methodologies and Technologies in Digital Marketing and Entrepreneurship. IGI Global, p. 681-694.

OKIKE, E. U. (2018) FQWCOS: A Flexible Model for Measuring Customer Satisfaction on Software Based Products and Service. Software Engineering, v. 6, n. 4, p. 110.

PEIXOTO, D. C. C.; BATISTA, V. A.; RESENDE, R. F.; PÁDUA, C. I. P. S. (2010) A Case Study of Software Process Improvement Implementation. In Software Engineering and Knowledge Engineering, p. 716–721.

PÉREZ, J.; MENS, T.; KAMSEU, F. (2013) A pilot study on software quality practices in Belgian industry. In Software Maintenance and Reengineering (CSMR), 17th European Conference on. IEEE, p. 395–398.

PINO, F. J.; RUIZ, F.; GARCIA, F.; PIATTINI, M. (2012) A software maintenance methodology for small organizations: Agile_MANTEMA. Journal of Software: Evolution and Process, v. 24, n. 8, p. 851–876.

PORT, D.; TABER, B. (2018) An empirical study of process policies and metrics to manage productivity and quality for maintenance of critical software systems at the jet propulsion laboratory. In: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ACM. p. 37.

RACHEVA, Z.; DANEVA, M.; BUGLIONE, L. (2008) Complementing measurements and real options concepts to support inter-iteration decision- making in agile projects. In Software Engineering and Advanced Applications. SEAA’08. 34th Euromicro Conference. IEEE, p. 457–464.

RODRÍGUEZ, P.; KUVAJA, P.; OIVO, M. (2014) Lessons learned on applying design science for bridging the collaboration gap between industry and acade- mia in empirical software engineering. In Proceedings of the 2nd International Workshop on Conducting Empirical Studies in Industry. ACM, p. 9–14.

ROMBACH, H. D.; ULERY, B. T. (1989) Improving software maintenance through measurement. Proc. IEEE, v. 77, n. 4, p. 581–595.

SALVIANO, C. F. (2006) Uma proposta orientada a perfis de capacidade de processo para evolução da melhoria de processo de software.

SCAMPI, T. U. (2011) Standard CMMI Appraisal Method for Process Improvement (SCAMPI) A, Version 1.3: Method Definition Document.

SCHRETTNER, L.; FÜLÖP, L. J., KISS, A.; GYIMÓTHY, T. (2012) Software quality model and framework with applications in industrial context. In Software Maintenance and Reengineering (CSMR), 16th European Conference on. IEEE, p. 453–456.

SCHWABER, K. (1997) Scrum development process. In Business object design and implementation. Springer, p. 117–134.

SEBRAE (2017) SEBRAETEC - Serviços em Inovação e Tecnologia. http://www.sebrae.com.br/sites/PortalSebrae/ Programas/sebraetec-inovar-no-seu-negocio-pode-ser-facil.

SOUTHEKAL, P. H.; LEVIN, G. (2011) Formulation and empiri- cal validation of a gqm based measurement framework. In Empirical Software Engineering and Measurement (ESEM), 2011 International Symposium on. IEEE, p. 404–413. Evidências sobre o desem- penho das empresas que adotaram o modelo MPS-SW. Campinas, Brazil: Softex.

STARON, M.; MEDING, W.; KARLSSON, G.; NILSSON, C. (2011) Developing measurement systems: an industrial case study. Journal of Software: Evolution and Process, v. 23, n. 2, p. 89–107.

SUKARIE-NETO, J. (2018) Mercado Brasileiro de Software – Panorama e Tendências – 2017 (ABES). http://central.abessoftware.com.br/Content/UploadedFiles/Arquivos/Dados%202011/ABES-EstudoMercadoBrasileirodeSoftware2018_ResumidaIngles.pdf

TRAVASSOS, G. H.; KALINOWSKI, M. (2014) iMPS.

UNTERKALMSTEINER, M.; GORSCHEK, T.; ISLAM, A. K. M.; CHENG, C. K.; PERMADI, R. B.; FELDT, R. (2014) A conceptual framework for SPI evaluation. Journal of Software: Evolution and Process, v. 26, n. 2, p. 251–279.

WALLACE, L. G.; SHEETZ, S. D. (2014) The adoption of software measures: A technology acceptance model (TAM) perspective. Information & Management, v. 51, n. 2, p. 249–259.

YIN R. K. (2015) Estudo de Caso: Planejamento e Métodos. Bookman editora.

YU, B.; CONG, G.; NING, L.; JIANG, H.; WANG, X. (2009) The Model of Software Process Measurement and Improvement Driven by Project Performance. In Computer Network and Multimedia Technology, 2009. CNMT 2009. International Symposium on. IEEE, p. 1–4.