SYSTEMS SUCCESS MEASUREMENT: INSTRUMENT & FRAMEWORK A NEW PERSPECTIVE

Ibrahim Eskandar Ibrahim Fadhel

School of Human Development and Techno-communication, Universiti Malaysia Perlis, National Institute of Administrative Science (NIAS), Malaysia

E-mail: ibrahim.eskandar@outlook.my

Syed Zulkarnain Syed Idrus

School of Human Development and Techno-communication, University Malaysia Perlis, Malaysia

E-mail: syzul@yahoo.com

Mohd Syukri Yeoh Abdullah

Institute of Malay World & Civilization (ATMA), The National University of Malaysia, Malaysia

E-mail: syukri@ukm.edu.my

Amani Ali Elmetwaly Ali Ibrahim

School of Human Development and Techno-communication, University Malaysia Perlis School of Human Development and Techno-communication, University Malaysia Perlis, Malaysia

E-mail: amaniali@unimap.edu.my

Mazni Omar

College of Science and Arts, School of Computing University Utara Malaysia, Malaysia

E-mail: omar.mazni@gmail.com

Submission: 09/10/2018

Revision: 11/06/2018

Accept: 11/06/2018

ABSTRACT

There are major concerns regarding the previous research that can be said as having poor measurements, lack of theoretical grounding, and concern heavily on investment justifies. Some are also considered as having weak survey instruments, inappropriate data collection approaches, and lack of agreement on the dependent variables measurement that resulted in its incompatibility. Several related issues pertaining to high rates of systems’ failure that justify the heavy investment costs and affect the systems’ success of measurement. This matter had raised a high concern, especially for the researchers, practitioners, organization managers and systems administrators. Thus, the successful measurement of any systems is vital. The purpose of this study is to provide a framework and a high-quality validated instrument based on the literature review and experts’ validation. Subsequently, it can be used in the future studies to efficiently assist in the systems’ success measurement. The statistical test of 344 users with the smart PLS for the instrument shows an excellent result.

Keywords: Experts’ Validation; Systems Success; Systems Fail

1. INTRODUCTION

In the Arab world, especially Yemen having almost nonexistence studies regarding to information systems success (FADHEL et al., 2018a; FADHEL et al., 2018b). The users of the systems are expanding than ever before, this result in measurement complexity of systems success. Researchers nowadays are facing many challenges, which is the sophistication of systems. With the increasing number users, this phenomenon can make us lose sight of the key elements such as (relevance, accuracy, timeliness etc.) of quality that are playing a major role in the success of the systems. The growth of measurements of systems success is increasing, which leads to more complexity that needs future studies (DELONE; MCLEAN, 2016).

For the researchers, DeLone and McLean (2016) highlight several themes that can be studied in future. One of these highlights is developing an adaptive research processes and frameworks to measure systems success. There is still a lack of development approach for building web systems success quality framework, and the field of measuring the web-based systems is not yet mature (ZAHRAN et al., 2014). Similarly, success measure is necessary to help assess how the systems is performing and outline any issues that could possibly be causing hindrance for the users (DELONE; MCLEAN, 2016).

2. BACKGROUND

Various conceptual frameworks and empirically instruments that have been tested, suggested to-date, each of them highlighting different factors that could potentially underline how a good quality WIS can be built. However, majority of these frameworks such as DeLone and McLean (1992), WEBQUAL and W-QEM are more suitable for business purposes (KHRED, 2017; MEBRATE, 2010). Sadly, most of these frameworks and instruments are in limited domain (MWANGI, 2016) and have not been thoroughly tested or applied in connection to system website development and implementation. Most of the frameworks and instruments only provide general quality characteristics, which is not helpful at times and seems to be a waste of resource as well (MEBRATE, 2010).

No mutually agreed definition is available and/or any reliable measurement instruments (MARDIANA; TJAKRAATMADJA; APRIANINGSIH, 2015; MCNAB; LADD, 2014; MWANGI, 2016). Therefore, the basic concerns are still prevailing pertaining to the explanation of quality criterion that could be potentially deployed to examine systems quality and effectiveness (MWANGI, 2016).

Accordingly, mixed results have also been reported in terms of what can explain systems. Mostly they outlined in relation to systems and software’s includes ineffective measurements, limited theoretical grounding, reliance on financial performance, lack of data collection, and limited knowledge on prediction (MARDIANA et al., 2015; DELONE; MCLEAN, 2016). While many studies have investigated the relationship between information systems (IS) characteristics and IS use, the results have been inconsistent (FORSGREN et al., 2016). This severe mixed results and lack of empirical focus on systems outlines psychological, cognitive and passionate prospects, which could intervene the relationship between the instruments and predictors of success (SNEAD et al., 2015). Research in the relationship of information systems success (use, satisfaction and benefit) has produced mixed results (GOEKE; CROWNE; LAKER, 2018).

Systems in the organizations are still having confrontation of the lack of research targeted users’ satisfaction (LAUMER, 2016; POLITES; KARAHANNA, 2012). Business researchers have been targeted the satisfaction of the users in the context of business. Nevertheless, notwithstanding the fact that the higher education market is getting more discerning, there is noteworthy lack of research (WONG, 2016).

3. INSTRUMENT OF SUCCESS MEASUREMENT

The current instrument consists of 11 constructs: 8 independent variables (IVs) (information quality, systems quality, ease of use, security, usability, reliability, functionally and efficiency); and 3 dependent variables (DVs) (satisfaction, benefit and loyalty).

3.1. Information quality

In the views of Edlund and Lovquist (2012) as cited by referring to Petter et al. (2008) quality in connection to information refers to significance and value of the provided information generated by the IS exhibits. Therein, with regards to measuring, how much satisfied the end users are with the provide information and its quality becomes the major important factor. As a consequence, it is viewed as the most important factor to outline user satisfaction which is often not found in an appropriate manner.

Accordingly, Edlund and Lovquist (2012) have also asserted that IS quality in terms of the information provided also defines the end-user satisfaction and thus, guides as to what length, it is reaching up to its expectations. Notably, user may end up experiencing frustration, if they fail to achieve accurate and quality information from the provided IS. Edlund and Lövquist (2012) based on the assertions of Bharati and Berg (2005) and Petter et al. (2008) have also outlined that several incidents can be noted pertaining to arguments amongst prominent researchers in this domain regarding the significance of information and its quality in particular. Studies have outlined several factors that define quality of information, which includes but not limited to accuracy, precision, relevance of information, element of timeliness and completeness.

Information quality: refers to the prominent features of the piece of outcome from a system or entity. In detail, this denotes to completeness, understandability and accuracy. On the grounds of the critical appraisal of the literature, information quality is highly important for student`s satisfaction when it comes to the website of the university.

First IV Perceived Information Quality (PINFQ) measures the accuracy, content and understandability:

a) The information outputs of my university web system (including on-screen and printed outputs) are Complete.

b) The information outputs of my university web system (including on-screen and printed outputs) are concise and are easy to understand.

c) It is easy to find what I’m looking for when using my university web system.

d) The information outputs of my university web system (including on-screen and printed outputs) are accurate and is free from errors.

e) My university web system provides the precise information I need.

These questions adapted from (BYRD et al., 2006; CHEN; KAO, 2012; CHIU et al., 2016; DAVARPANAH; MOHAMED, 2013; EDLUND; LÖVQUIST, 2012; FADHEL, 2015; GORLA; SOMERS; WONG, 2010; MOHAMMADI, 2015; WANG; LIAO, 2008; ZAIED, 2012).

3.2. SYSTEM QUALITY

Edlund and Lövquist (2012) whilst quoting Bharati and Chaudhury (2004) has outlined system quality as the generic performance of the information system. Whilst referring to Delone and McLean (1992) and Edlund and Lovquist (2012) highlighted another explanation of system quality. According to them, system quality talks about attributes of an information system that ensures the generation of information that is valuable for making effective decisions.

In connection to the study by Petter et al., (2008) vital prospects pertaining to system quality talks about system flexibility; ease of learning, and ease of use. This refers to extent to which the usage and learning of a particular system requires no effort and hassle. This element is crucial in the prospect of information quality since such efforts vary in terms of how they are perceived by the users. Individuals viewing a particular system as requiring more effort and full of stress may result in avoiding the use of the system. Therefore, IS usability perceptions being essential in this regard (EDLUND; LÖVQUIST, 2012).

System quality denotes to noteworthy features of the system including adaptability, trust and sophistication. On the basis of the literature review system quality, it is also principally related with student’s satisfaction with the web systems.

The Second IV Perceived System Quality (PSYSQ) measures the adaptability and sophistication:

a) It is easy for me to become skillful by using my university’s web system.

b) In general, I find my university’s web system is easy to use.

c) My university’s web system is well integrated.

d) My university’s web system has a short time lag between input and output of data, as example ‘the registration process’.

e) My university’s web system has a short response time for on-line enquiry.

Those questions were adopted from (CHIU et al., 2016; FADHEL, 2015; GORLA; SOMERS; WONG, 2010; MOHAMMADI, 2015; ZAIED, 2012).

3.3. EASE OF USE

According to Ofori, Larbi-Siaw, Fianu, Gladjah and Boateng (2016), as per the explanations by Davis (1989), it mentioned that Perceived Ease of Use (PEOU) denotes to the extent to which an individual believes that the specific information system or technology would be effort-free in use. Davis (1989) has also asserted that technology and its usefulness relies upon how convenient it is for users to use. In a simpler term, the easier it is for the users, the important it would be for them to interact with the social media, web portals and other online platforms cited in (OFORI et al., 2016).

Jongchul and Sung-Joon (2014) and Park, Rhoads, Hou and Lee (2014) on the other hand stated that there is a causal connection between PEOU and PU. This connection has also been confirmed by several studies conducted in different occupation settings cited in (OFORI et al., 2016). Al-Azawei and Lundqvist (2015) whilst referring to Venkatesh and Davis (2000), outlined that perceived ease of use refers to the extent to which a user views the usage of a particular system would be convenient and free from all the hassles and efforts. Thus, technology acceptance model 2 (TAM2) as elaborated in PEOU is important in outlining perceived usefulness and users’ attitudes towards a technology.

Perceived ease of use: The term denotes to student perception about the website usefulness and ease in connection to physical efforts. Prominent literature has sketched a significant association of ease of use of students as users with university`s electronic web systems.

Third IV Perceived Ease of Use (PEOU) measures the systems easiness:

a) I find my university’s web system flexibility to interact with.

b) My interactions with my university’s web system during an online process were clear and understandable.

c) My university’s web system is convenient for me.

d) My university’s web system is laid out in a modern and fashionable.

These questions were adopted from (DEVARAJ et al., 2002; KHAWAJA; BOKHARI, 2010; LIU et al., 2010; MOHAMMADI, 2015; WOLFINBARGER; GILLY, 2003).

3.4. RELIABILITY

According to Dreheeb et al., (2016) and Selvakumar, (2016) reliability is also an important and essential prospect when it comes to software quality. Dohi and Nakagawa (2013) however, mentioned that the reliability is set of attributes that can potentially trigger individual capability to maintain its performance level in a given period of time. Therein, the system is required to keep hold of software faults to ensure reliability and minimize software crashes. The systems are typically capable of re-establishing their performance levels in order to carry on generating same results based on Papanikolaou and Mavromoustakos (2008).

Reliability in connection to IT refers to capability of a system to offer or provide designated functions and features in a particular time period (MBIWA, 2014). Accordingly, Shiratuddin (2015) had suggested that the degree to which, a product and/or component executes the outlined conditions as per the specifications. Some of the reliability prospects are concerned with acting upon elements necessary for promised timings (VAN IWAARDEN et al., 2004).

Reliability denotes to the extent to which the system features and prospects are robust to perform specific functions and provide designated services and outcomes. Reliability includes maturity, fault tolerance, recoverability and availability. Literature has supported a significant relationship between reliability on the use of student`s satisfaction when it comes to the web system usage.

Forth IV Perceived Reliability (PREL) measures the maturity, fault tolerance, recoverability, availability and reliability:

a) My university’s web system never stops unexpectedly.

b) When there is a problem in some part or parts in my university’s web system, I can still can browse and perform some of processes.

c) In case of interruption of faulty, my university’s web system will recovers in a timely manner.

d) In general, my university’s web system is available 24 hours.

e) I believe that my university’s web system is reliable.

Those questions were adopted from (AGHAZADEH et al., 2015; ALVES ET AL., 2015; CONSTANTIN, 2013; DEVARAJ; FAN; KOHLI, 2002; MEBRATE, 2010).

3.5. USABILITY

Al-Manasra, Khair, Zaid and Taher Qutaishat (2013) said usability is an important. It is also one of the most important factors to outline software quality (DREHEEB; BASIR; FABIL, 2016). Usability is a crucial component that relies upon how well a particular application and/or software.

According to Madan and Dubey (2012) usability outlines crucial attributes pertaining to the establishment of successful software applications cited in (DREHEEB; BASIR; FABIL, 2016). Likewise, Dreheeb, Basir, and Fabil (2016) have also asserted that e-learning success is essential, and it is only possible through responsive usability of the software and online features (ARDITO et al., 2006). Moreover, usability is the core premise for the evaluation of e-learning technologies and systems. It denotes to the considerable features and prospects of software that enables it to help users understand, learn and attract connotations under specified conditions (DREHEEB; BASIR; FABIL, 2016; CHO; HYUN, 2016).

Usability defines as the extent to which, product or systems can bring it is feasible and objective to provide specific objectives, and thus it facilitates in achieving effectiveness and efficiency in satisfaction in the context of the use of any specified system. Usability includes user interface aesthetics and protection from user's error. Past studies have also outlined a strong association of usability perceptions with student`s satisfaction with web-site systems.

Fifth IV Perceived Usability (PUSA) measures the user interface aesthetics and protection from users’ error:

a) The interface design of my university’s web system is attractive.

b) All interface elements are well combined and harmonious in my university’s web system.

c) My university’s web system protects me from making errors when interring data.

d) My university web system errors messages clearly indicate to me how to rectify the problem.

e) In my university’s web system, it is easy to recover from the error instantaneously.

Those questions were adopted from (ALVES et al., 2015; ASTANI; ELHINDI, 2008; MEBRATE, 2010; PADAYACHEE; KOTZE; VAN DER MERWE, 2010; WOLFINBARGER &; GILLY, 2003; SUWAWI, 2015).

3.6. FUNCTIONALITY

Functional prospects refer to the potential of a service or product meeting the implied needs under particular conditions (MBIWA, 2014; TANDON; KIRAN; SAH, 2017). Also referred as Suitability, functionality is the degree to which a particular product or systems offers processes and functionality that meets the desired expectations of the customers (SHIRATUDDIN, 2015).

Functionality defines as the extent to which, a specific product or service-based system offers features and prospects that are in line with the implied needs of users under designated conditions. Functionality includes navigation and search. Literature on the topic has also underlined functionality to be of high significance when it comes to student`s satisfaction, whilst using university’s web systems and related systems.

Sixth IV Perceived Functionality (PFUN) measures the navigation and search:

a) It is easy to go to the home page while I’m browsing any other page in my university’s web system.

b) While using my university’s web system, I can easily navigate backwards through previously visited pages.

c) My university’s web system provides varied search options (e.g. by faculty, courses, etc.).

d) Search hints are provided when wrong keywords search is used.

Those questions were adopted from (ALADWANI, 2002; KHAWAJA; BOKHARI, 2010; MEBRATE, 2010).

3.7. EFFICIENCY

Aghazadeh, Pirnejad, Aliev and Moradkhani (2015) stated that efficiency is another important component which refers to performance quality of the software. As per definition, efficiency in connection to systems can be referred as capability of the software towards offering responsive performance, whilst using highly reasonable amount of resources in any stated situation.

Users are generally expected to operate in a manner whereby, they have to use the minimum amount of resources with the highest possible e-learning experience. Accordingly, system response denotes to performance of the system or software in terms of time, graphics, page set up and loading in order to enhance the user satisfaction (PAPANIKOLAOU; MAVROMOUSTAKOS, 2008) as in (DREHEEB et al., 2016). Efficiency is also very important when it comes to performance of the software and to what length it is relatively using minimum resources compared to other alternative options in a given situation (MBIWA, 2014).

Efficiency defines as the potential of the software to offer desired functions in order to reach the desired objective needs from the software use. Efficiency includes time behavior and accessibility. Literature available on the topic has also confirmed its significance relationship with user`s satisfaction with the web system of the university.

Seventh IV Perceived Efficiency (PEFF) measures the time behavior and accessibility:

a) It is possible to find in my university’s web system what I want in a reasonable time.

b) My university’s web system enables me to get on to it quickly.

c) My university’s web system does not use advertises or unwanted plug-ins.

d) I can access my university’s web system from my favorite browser.

e) It is easy to get and browse any part on my university’s web system.

Those questions were adopted from (ALVES et al., 2015; KHAWAJA; BOKHARI, 2010; MEBRATE, 2010; ROCHA, 2012; ZEHIR et al., 2014).

3.8. SECURITY

When it comes to the web systems, their technical features may include a security. As per definition, Astani and Elhindi (2008) had suggested that the web system security relates with authentication for user and its potential in this regard. In detail, security refers to the capability of a portal to provide secure access virtual environment to users whereby, they can use data related to a given product or service without any scam (MBIWA, 2014).

According to Shiratuddin (2015) and Ludin and Cheng (2014) security also denotes to degree to which the system protects the information and data in such a manner that users are able to access it as per the level of authorization. Security effects the satisfaction of users significantly (CHIANG; HUANG; YANG, 2011).

Security defines as to caters to privacy in the mutual exchanges i-e financial as well as non-financial. The availability of secure inline systems builds students` confidence and reliability in the web portal and offers a friendly environment for completing transactions, which includes security and privacy. The extent to which system protects information and important data related to personnel involved in the transaction significantly enhances user satisfaction with the web system of the university.

Eighth IV Perceived Security (PSEC) measures the security privacy and trust:

a) I believe my university’s web system is secure.

b) Overall, I trust my university’s web system.

c) My university’s web system has adequate security features that make you feel secure while using.

d) I believe that the information offered by my university on the university’s web system is sincere and honest.

e) The output information of my university’s web system is secure.

These questions were adopted from (ALVES et al., 2015; JEON, 2009; MALIK et al., 2016; WEBB; WEBB, 2004; WOLFINBARGER; GILLY, 2003; ZAID, 2012; ZEHIR, 2014).

3.9. SATISFACTION

In the views of Vaezi et al., (2016) the literature on the topic has outlined a considerable gap in terms of what user desire and expect against what is offered to them and how it influences success of an information system. Overall, there are several explanations and instrument for measurement are currently available, but how and to what length these measures are vital may varies. Gradually, the scholars are moving towards developing one unified user satisfaction construct and measure, which would ultimately change the idea and view about the concept especially in the area of information systems research (VAEZI et al., 2016).

Studying user satisfaction and how to predict it is essential for organization. Such studies are significant to help enterprises comprehend with the idea and how they can strategize to keep their users satisfied with their services, facilities and performance prospects (VAEZI et al., 2016). Concerning the dimensions of success, satisfaction of users is essential due to the fact that the field of research pertaining to Information System (IS) is very limited.

Students’ satisfaction denotes to the measure of satisfaction of students with the major system features a student interacts with. This primarily includes online support systems, reports and access, university online systems and online course data banks. Review of the literature has suggested that satisfaction to student with system and online portals can be of significant value towards system benefit and enhancing loyalty with these web systems.

First DV Students’ Satisfaction (STSA) measures the students’ satisfaction

a) My university’s web system is of high quality.

b) My university’s web system has met my expectations.

c) My interaction with my university’s web system is very satisfying.

d) Overall, I am satisfied by using my university’s web system.

e) Overall, I’m happy with my university’s web system.

Those questions were adopted from (AL-AZAWEI; LUNDQVIST, 2015; CHIU et al., 2016; CONSTANTIN, 2013; EPPLER; ALGESHEIMER; DIMPFEL, 2003; FADHEL, 2015; JEON, 2009; KIRAN; DILJIT, 2011; LIAW; HUANG, 2013; MOHAMMADI, 2015).

3.10. BENEFIT

Wang and Liao (2008) showed the result that has considerable support for the DM model and encourage the study of perceived net benefit. Alshibly (2015) also forwarded support and recommendations towards net benefits and asserted that it ideally should be designed under a specific framework to help scholars and practitioners to effectively assess system benefits. DeLone and McLean (2016) have outlined that some of the most prominent measures for assessing IS success are designers, managers, users and so on. Therein, the net impacts are system outcomes, which are generally compared to the core purpose of the system.

For this reason, the Net Impacts construct will be the most contextual dependent and varied of the six D&M Model success dimensions (DELONE; MCLEAN, 2016). Several methods are available to examine the net effects at all four levels i-e organizational, individual, and societal and industry. It is recommended that the usage of individual measure would be more appropriate for the assessing information system success, rather than from other general prospects (DELONE; MCLEAN, 2016).

Benefit defines as one of the highly important prospects of systems success is the benefit measure, which denotes to the influence and outcomes of the systems from individuals to economies and societies at large. Scholars in the area have outlined a significant feature when it comes to systems and their benefits. The benefits refer to the extent to which a system is healthy and worthwhile for users, organizations, groups, business sectors and economies at large such as system facilitation in decision making, productivity enhancement, welfare or job effectiveness.

Second DV Benefit (BENE) measures the systems benefit

a) My university’s web system helps me to retrieve my information easier and quickly.

b) My university’s web system saves my time.

c) Overall, I obtained benefits from using my university’s web system.

d) My university’s web system is an important and valuable aid to me.

e) My university’s web system has a large, positive impact on me as a user.

Those questions were adopted from (CHIU et al., 2016; DERNBECHER, 2014; FADHEL, 2015; MCGILL; HOBBS; KLOBAS, 2003; WANG; LIAO, 2008; WIXOM; WATSON, 2001).

3.11. LOYALTY

There seems to be little empirical attention towards outlining what causes customer satisfaction, especially amongst the tertiary students and whether or not, these service features are capable of generating healthy benefits and outcomes such as customer satisfaction and loyalty towards institutions (BROWN; MAZZAROL, 2006). Brown and Mazzarol (2006) have also asserted that at present educational environment is more regarded as service business and students as customers. A study made by Senate (2001) outlined that the customer satisfaction and customer value, both have values towards academic institutions (BROWN; MAZZAROL, 2006).

Likewise, Cronin, Brady, and Hult (2000) have mentioned that behavioral intentions as the last items in the analysis. According to them, rising customer retention and lower customer defection is the core prospect through which an organization can generate more profits (ZEITHAML; BERRY; PARASURAMAN, 1996). In the views of Cronin, Brady and Hult (2000), positive behavioral intentions are important for enterprises and help them to get their customers to forward positive thoughts about the service and company products like positive words; recommendations; express loyalty; investing more in other company products and show willingness to purchase premium products.

Loyalty defines as a behavioral prospect that outlines acceptance and satisfaction with a certain product or service and leads towards repeat using, encourages referrals and recommendations. Loyal students in this context would be ones engaged in repeatedly using the online system of the university and actively recommending of the same to other students.

Third DV loyalty (LOYA) measures the students’ loyalty

a) I will be using more of my university’s web system in the future.

b) I will recommend my university’s web system to others.

c) I will say positive things about my university’s web system to others.

d) I like using my university’s web system.

e) I use my university’s web system frequently.

Those questions were adopted from (CONSTANTIN, 2013; EPPLER et al., 2003; JEON, 2009; KIRAN; DILJIT, 2011; MOHAMMADI, 2015; VALVI; WEST, 2013; ZEHIR et al., 2014).

4. PROBLEM AND OBJECTIVE

With the high number of the systems failure existed globally, they are due to the mix results and weak survey instruments (FADHEL et al., 2018a; FADHEL et al., 2018b). Thus, this study aims to produce a new prescription for the systems success measurement by providing a high reliability validated instrument and framework.

5. FLOW AND METHODOLOGY

Yemeni universities web systems are large integrated applications that considered as primary central applications for student information. It allows the administrator to manage and provide data to the staffs, students visitors, etc. Furthermore, it is giving permission to the students to register and deal within the related details of the study until they graduated (KHRED, 2017).

This study starts with the validation process for the instrument, which is a strong way for effectively instrument design. Based on the previous studies, to make the process of validation in the fields of information systems and software engineering (quality and testing), there must be at least three experts (academic experts in the field or in the related fields with a PhD as a minimum qualification or technical experts in the field with at least 3 years of experience).

Then, this study used all the required statistical tests for instrument approval as a valid tool to measure web-based system's success in the domain of universities. A pilot test was performed all necessary statistical tests to measure the instrument reliability has been done (Rho_A, Composite Reliability (CR), Average Variance Extracted (AVE) and Cronbach’s Alpha (α)). Confirmatory Factor Analysis (CFA) has also been done to see how were items load, are those items related to their constructs are not.

Finally, after real data collected the required statistical tests has been performed Construct Validity, Convergent Validity, Discriminant Validity and Multicollinearity (CFA, Rho_A, CR, AVE, α, Fornell, HTMT and VIF). After making sure that all tests are perfect the instrument and framework are proposed.

Research can be either be qualitative, quantitative or mixed methods. The best method depends on the research purpose as each research has its own merits and demerits (FADHEL, 2015). This study aims to comprehensively explain the phenomenon by using a quantitative & qualitative methods to achieve the maximum benefits, these approaches are considered as the best means that are suited under the current circumstances. Current research following the qualitative way in the process of the instrument validation only.

After the validation process ended a full quantitative way has been used in testing the instrument in the pilot test and in the real data test. Instrument has been tested in pre-test with 9 users, pilot test with 33 users then used in final data collection process with 344 students (users of the systems) from three different universities. Smart PLS used to perform the results as its categorized as one of the best tools used for predicating the results.

Based on the literature review of Smart PLS, we provide a summary of its benefits, that are it works well with structural equation that are comparatively new techniques to model series of cause-and-effect connections with latent variables. The PLS-SEM method is known to be a user-friendly tool for statistical model development in addition to predicting or making forecast cited in. Particularly, it was employed for the study because of the following reasons. To develop an instrument and a structural equation models was the first motives PLS-SEM was leveraged. However, it has been illustrated to be advanced in performing models in the fields of information systems and software engineering.

Additionally, it is more suitable for the actual world applications, as it is well more beneficial when modeling. PLS soft modelling technique includes (i.e. ability to flexibly develop and validate complex models) it can also be employed to estimate huge complex models. PLS-SEM application on huge complex models is the main reason the study adopts PLS for enhanced prediction. In many existing information systems and software engineering researches, data tends to possess normality issues. However, PLS does not necessarily needs the data to be normal for it to be analysed. Also, non-normal dataset is treated better with PLS.

To avoid problem of data normality path modelling technique was finely selected. Moreover, PLS used in behavioral and social sciences, SEM is a powerful statistical analysis tools that is able to test various relationships concurrently. Finally, PLS-SEM has a valid and semantically correlated outcome, while other existing techniques for data analysis often results in less unclear outcomes and mostly have separate analyses.

6. QUALITATIVE RESULTS

In this research, experts reviewing and consulting for this work, instrument and framework is one of the main steps. Noted from the literature most of researchers used 4-7 experts in their researches. Number of experts can be exceeding 20 experts, no matter if it is exceeding twenty validators but usually minimum numbers is preferred. Typically, in most of researches number of experts around six quoted in (OLSON, 2010) by his referring to (HOLBROOK et al., 2007; JANSEN; HAK, 2005; PRESSER; BLAIR, 1994; THEIS et al., 2002).

6.1. Expert number one

First of all, the researcher starts the process of validation by the expert Dr. Hmaad, Assistant Professor in Yemen, as he is specialized in information systems. The researcher than meets the validator face-to-face in three different occassions for a few hours ranged from 2-3 hours. After these meetings, the validator Dr. Hmaad provided his acceptance on the instrument and framework with some advices for the upcoming stages.

6.2. Expert number two

The researcher then contacted Dr. Shatena via email and telephone call, who works as a lecturer and Head of Technology and Supply Chain Management Studies Center in the Universit Teknologi Mara (UiTM). She specialised in business and technology systems management. Dr. Shatena provided some amendments and notes for the mistakes located in the in the instrument. Once the researcher had rectified the requested amendments and corrected the mistakes, she re-reviewed the instrument once again and provided her agreement with advices to use the PLS as an analysing tool.

6.3. Expert number three

The next stage the researcher had contacted Dr. Ali, who is specialised in information systems and works as Visiting Senior Lecturer in the Universiti Utara Malaysia. In the first contact with Dr. Ali via telephone, the researcher had asked for a pilot test, and told him that the pilot test will be performed after completing the process of validations from all experts. He advised, to perform the pilot test initially, since the the two prior experts had already accepted the work.

He then recommended to the researcher to perform another pilot test once the validation process has been completed. The researcher responded to Dr. Ali’s request and then met him in his office for two hours and half. Dr. Ali’s expert advice had provided his agreement on the work, with some adjustments in the instrument. The researcher finally applied the adjustments and gets the agreement on the instrument.

6.4. Expert number four

On first of November 2017, the researcher gets the validation agreement from Professor Azizah, she is specialized in software engineering and works as a lecturer in Universiti Utara Malaysia. Prof. Dr. Azizah in the second face-to-face meeting provided her strong agreement upon the work, framework and the questionnaire with an advice to remove one construct and some items.

6.5. Expert number five

After finishing with the four experts, the researcher then call and send an email to Professor Dr. Ahmed, he was the former Dean of Faculty of Engineering and Computer Science and currently works as lecturer in Yemen. Strong and critique advices have been provided by him to remove some constructs, with the agreement on the instrument and framework.

6.6. Expert number six

Continually, researcher referred to Miss Fawzia the Director of Systems and Information for more than ten years in Yemen. She agreed with the advices given by Professor Dr. Ahmed, and she then provided the same advices for removing constructs and some items to make the framework comprehensively related to systems quality. Finally, she gives the researcher her covenants with advice to contact Dr. Fathya on the quantitative specialized area.

6.7. Expert number seven

The researcher had applied to Miss Fawzia advices and directly communicated with Associated Professor Dr. Fathya, in Yemen. Dr. Fathya is specialized in quantitative science. She provided a novel advice regarding to the Likert scale and removing the constructs that are not related to quality of systems. She also, provided advices regrading to data analysis and how unnecessary constructs and items can be negatively affect. She said she sees the signs of success and she sent her agreement via email.

6.8. Expert number eight

After all these advices researcher applied all the notes and remove the unrelated constructs and items from the framework. The instrument went to the respondents for a new pilot test and the results was brilliant. Before went to the main data process collection. The researcher had communicated with Dr. Israr from India.

The final validator for this research is Dr. Israr, he was working as Assistant Professor in Yemeni universities and with university Jamia Millia Islamia in India. His specialization is in systems and computer science. Dr. Israr sent his agreement with advice of using PLS as the analysing tool. Finally, after applied all experts’ notes, pre-test and pilot test, main data collection process was performed confidently.

7. QUANTITATIVE RESULTS

Here the researcher provides the results based on the pilot test and real data collection.

7.1. Pilot test

Significance of a pilot test in research contemplate can never be overemphasized claiming it diminishes the pressure that the researcher could have experienced amid the final analysis of the research (CAVANA et al., 2001). Subsequently, it is extremely urgent to lead a pilot test to assist researcher in assembling a decent establishment for the significant examination (ADEBOLA, 2014).

The pith of the pilot contemplate is to assist the researcher to find pressing issues that may emerge from the questionnaires and by allowing researchers in readdressing and altering principal consideration in the questionnaire (ADEBOLA, 2014; PALLANT, 2007).

Pilot test results are illustrated in the form of the tables. The tables show that all tests are acceptable and the values are accepted and showed excellent reliability. The factors loading for all items are perfect and all items under its related construct.

Table 1: Pilot Result - Construct Reliability and Validity

Factors	Rho_A	Composite Reliability	Average Variance Extracted	Cronbach’s Alpha
Benefit	0.8191	0.8726	0.5784	0.8177
Ease of Use	0.7914	0.855	0.5976	0.7734
Efficiency	0.8076	0.8635	0.5597	0.8019
Functionality	0.8239	0.8769	0.6435	0.8097
Information Quality	0.854	0.895	0.6317	0.8520
Loyalty	0.8018	0.8633	0.5591	0.8011
Reliability	0.7938	0.8569	0.546	0.7905
Satisfaction	0.829	0.8749	0.5857	0.8189
Security	0.8113	0.8682	0.5693	0.8097
System Quality	0.838	0.8827	0.6023	0.8328
Usability	0.8256	0.8741	0.5825	0.8191

Source: The Researcher

Table 2: Pilot Result - Confirmatory Factor Analysis

Factors	Items	Loadings
Benefit	Benefit1	0.7713
	Benefit2	0.7653
	Benefit3	0.7964
	Benefit4	0.7162
	Benefit5	0.7511
Perceived Efficiency	EFF1	0.7444
	EFF2	0.7812
	EFF3	0.7823
	EFF4	0.7771
	EFF5	0.6468
Perceived Ease of Use	EU1	0.8162
	EU2	0.7113
	EU3	0.8532
	Eu4	0.7002
Perceived Functionality	FUN1	0.8772
	FUN2	0.8158
	FUN3	0.8534
	FUN4	0.6407
Perceived Information Quality	IQ1	0.676
	IQ2	0.8361
	IQ3	0.8372
	IQ4	0.8206
	IQ5	0.7927
Loyalty	Loy1	0.6816
	Loy2	0.6967
	Loy3	0.7908
	Loy4	0.7971
	Loy5	0.7648
Perceived Reliability	REL1	0.7315
	REL2	0.6623
	REL3	0.7302
	REL4	0.8083
	REL5	0.7547
Perceived System Quality	SQ1	0.7099
	SQ2	0.815
	SQ3	0.7908
	SQ4	0.6924
	SQ5	0.8595
Satisfaction	Satisf1	0.6681
	Satisf2	0.7209
	Satisf3	0.9047
	Satisf4	0.7517
	Satisf5	0.7609
Perceived Security	Sec1	0.7425
	Sec2	0.7372
	Sec3	0.784
	Sec4	0.6852
	Sec5	0.8172
Perceived Usability	USab1	0.6997
	USab2	0.8235
	USab3	0.8221
	USab4	0.7216
	USab5	0.7404

Source: The Researcher

7.2. Real data result

This phase of research shows the consequences of investigatory analysis of the research study using PLS principal component analysis. Every scaled construct for the propose study was concurrently adapted from existing researches. As reported in (UNIT, 2013) If the construct loadings greater or equal to 0.6 this construct is reliable without attention to size of sample (GUADAGNOLI; VELICER, 1988).

The statement of Guadagnoli and Velicer (1988) is advocated and supported by (FIELD, 2005). Cut-off should be used with items of 0.4 loading without care to size of sample (STEVENS, 1992). Loading of 0.32 is poor,0.45 is fair, 0.55 is good, 0.63 is very good and 0.71 is categorized as excellent loading (TABACHNICK; FIDELL, 2007). Construct items should be 0.6 and above to perform reliable analysis especially if sample size is small (MACCALLUM et al., 2001). Here in the tables real data test results of 344 users are illustrated. Tables showed all tests are acceptable and the values were perfect.

Table 3: Real Data Result - Confirmatory Factor Analysis

Factors	Items	Loadings
Benefit	Benefit1	0.7540
	Benefit2	0.6136
	Benefit3	0.7504
	Benefit4	0.7363
	Benefit5	0.7253
Perceived Efficiency	EFF1	0.7032
	EFF2	0.7287
	EFF3	0.6874
	EFF4	0.7525
	EFF5	0.6104
Perceived Ease of Use	EU1	0.8021
	EU2	0.6987
	EU3	0.7923
	Eu4	0.7079
Perceived Functionality	FUN1	0.7575
	FUN2	0.7347
	FUN3	0.7610
	FUN4	0.7742
Perceived Information Quality	IQ1	0.7138
	IQ2	0.7419
	IQ3	0.7276
	IQ4	0.7048
	IQ5	0.7037
Loyalty	Loy1	0.7045
	Loy2	0.7614
	Loy3	0.7911
	Loy4	0.7891
	Loy5	0.7700
Perceived Reliability	REL1	0.6747
	REL2	0.7059
	REL3	0.7055
	REL4	0.7213
	REL5	0.7035
Perceived System Quality	SQ1	0.6723
	SQ2	0.7661
	SQ3	0.686
	SQ4	0.7542
	SQ5	0.7354
Satisfaction	Satisf1	0.7094
	Satisf2	0.7601
	Satisf3	0.7438
	Satisf4	0.7350
	Satisf5	0.7256
Perceived Security	Sec1	0.7468
	Sec2	0.7982
	Sec3	0.7174
	Sec4	0.6505
	Sec5	0.6763
Perceived Usability	USab1	0.7582
	USab2	0.7960
	USab3	0.7839
	USab4	0.8075
	USab5	0.7002

Source: The Researcher

7.3. Construct Validity

Construct validity construct validity evaluates the degree gotten from employing a measure using fit of theories where test is planned (SEKARAN; BOUGIE, 2010). More so, it is worried about responding to inquiry: does the research instrument identify concept as theorized? In accomplishing the validity test, two kinds of validity tests were subjected to the scales of measurement: (convergent validity) and discriminant validity (DYBA, 2005). Two sub-classes of construct validity are convergent and discriminant validity (SEKARAN, 2003). Also, Hair et al., (2017) proposed average variance extracted (AVE) to evaluate convergent validity.

7.4. Convergent Validity

Convergent validity of this study was measured by methods for normal difference separated method or (average variance extracted technique AVE). AVE is the normal difference shared amongst variable and its measures. AVE variable ought to be greater than the fluctuation shared amongst variable with other variables in a specific model (COUCHMAN; FULOP, 2006).

Existing studying states that an AVE estimation of 0.5 or more prominent estimation is viewed as satisfactory (BARCLAY; HIGGINS; THOMPSON, 1995). AVE of 0.5 is advocated by (HAIR et al.., 2017). Composite reliability recommended value is 0.7 (HAIR et al., 2017). If the value of AVE is low than 0.5 researchers can still accept AVE values until 0.4 as long as composite reliability CR is > 0.6 In case of AVE = 0.4 and value of CR is > 0.6 no worry about the convergent validity of the factor ; LARCKER, 1981; HUANG et al., 2013). Cronbach’s Alpha as recommended by Julie Pallant (2013) should be higher than 0.7. The value of spearman's eliable rho_A should be > 0.6 (GARSON, 2009).

Table 4: Convergent Validity-Constructs Reliability and Validity

Factors	Cronbach's Alpha	rho_A	Composite Reliability	Average Variance Extracted (AVE)
Benefit	0.7660	0.7823	0.8409	0.5153
Loyalty	0.8216	0.8266	0.8749	0.5835
Perceived Ease of Use	0.7453	0.7643	0.8381	0.5652
Perceived Efficiency	0.7415	0.7532	0.8255	0.4874
Perceived Functionality	0.7524	0.7562	0.8429	0.5730
Perceived Information Quality	0.7665	0.7668	0.8421	0.5162
Perceived Reliability	0.7474	0.7498	0.8295	0.4933
Perceived Security	0.7660	0.7716	0.8424	0.5180
Perceived System Quality	0.7745	0.7869	0.8458	0.5239
Perceived Usability	0.8279	0.8309	0.8791	0.5931
Satisfaction	0.7871	0.7876	0.8545	0.5402

Source: The Researcher

Table above is shown results of Cronbach's Alpha, rho_A, CR and AVE. The united legitimacy is set for every one of the constructs. The delimitating factors of united legitimacy demonstrates that the Cronbach's Alpha is greater than 0.7, rho_A is higher than 0.6, CR is bigger than 0.7 and AVE all items are larger than 0.5 only two factors are little bit less than 0.5, perceived reliability is 0.49 and perceived efficiency is 0.49 there is no problem at all with these factors because as stated above AVE value can be accepted even its = 0.4 as long as CR > 0.6 So, all factors results are satisfactory. This was clear enough to affirm that items speak to different or distinct hidden or latent constructs, and thus developed their convergent validity.

7.4.1. Discriminant Validity

Discriminant validity, conversely identifies with whether measures that ought not to be connected are as a rule not related. In measuring the discriminant validity, the square root of the AVE for every factor is leveraged (FORNELL; LARCKER, 1981) as cited in (HAIR et al., 2017). Hair et al., (2017) based on Fornell and Larcker (1981) said the square roots of AVE coefficients are then demonstrated within the correlation matrix along its diagonal. It is observed that a squared AVE should be more prominent than an evaluated squared correlation resulting to a better confirmation of discriminant validity.

Table the assessment of discriminant legitimacy was demonstrated for variables utilized as part of the study. Table below shows square along the diagonal underlying roots of AVE for every one of the constructs. In any case, square foundations of developing higher AVE than off-diagonal items or coefficients in corresponding rows and columns, henceforth, developing a proof of discriminant legitimacy.By and large, the outcomes portrayed in tables below demonstrate that measures for all the eleven constructs are legitimate measures of their separate constructs in view of their factual noteworthiness and parameter scales, following (CHOW; CHAN, 2008).

To test discriminant validity one of the best accurate tests used is Heterotrait-Monotrait ratio (HTMT) (HENSELER; RINGLE SARSTEDT, 2014). The values should be smaller than 1 as noted in (ALARCÓN SÁNCHEZ DE OLAVIDE, 2015). Some researchers said values should be less than 0.85 (KLINE, 2011) other researchers said HTMT values should be lower than 0.90 (TEO SRIVASTAVA JIANG, 2008) as quoted in (ALARCÓN et al., 2015).

Table 5: Discriminant Validity of Factors Square Root of the AVE on the Diagonal (Fornell Test)

	Benefit	Loyalty	Perceived Ease of Use	Perceived Efficiency	Perceived Functionality	Perceived Information Quality	Perceived Reliability	Perceived Security	Perceived System Quality	Perceived Usability	Satisfaction
Benefit	0.7178
Loyalty	0.3817	0.7639
Perceived Ease of Use	0.5011	0.1831	0.7518
Perceived Efficiency	0.5745	0.3506	0.5576	0.6981
Perceived Functionality	0.2135	0.1903	0.1371	0.1758	0.757
Perceived Information Quality	0.5721	0.2884	0.6424	0.5721	0.2262	0.7185
Perceived Reliability	0.4741	0.2935	0.3484	0.5055	0.2185	0.4322	0.7023
Perceived Security	0.5636	0.3331	0.5357	0.5513	0.1884	0.6088	0.4702	0.7197
Perceived System Quality	0.46	0.2814	0.6168	0.4705	0.2167	0.4858	0.3388	0.4504	0.7238
Perceived Usability	0.5999	0.3281	0.5357	0.5689	0.1539	0.6243	0.405	0.6744	0.5451	0.7701
Satisfaction	0.6254	0.4706	0.4498	0.561	0.2745	0.4946	0.4657	0.6459	0.3928	0.5743	0.735

Table 6: Discriminant Validity of Factors (HTMT Test)

	Benefit	Loyalty	Perceived Ease of Use	Perceived Efficiency	Perceived Functionality	Perceived Information Quality	Perceived Reliability	Perceived Security	Perceived System Quality	Perceived Usability	Satisfaction
Benefit
Loyalty	0.4672
Perceived Ease of Use	0.6546	0.2313
Perceived Efficiency	0.7508	0.4344	0.7489
Perceived Functionality	0.278	0.2417	0.1881	0.2322
Perceived Information Quality	0.7347	0.3569	0.8247	0.7427	0.3026
Perceived Reliability	0.6036	0.369	0.4584	0.6843	0.2802	0.5602
Perceived Security	0.7258	0.4106	0.703	0.7091	0.2431	0.799	0.601
Perceived System Quality	0.5932	0.3436	0.8394	0.622	0.2812	0.6238	0.4457	0.5739
Perceived Usability	0.7465	0.3916	0.6783	0.7091	0.1868	0.7794	0.5086	0.8489	0.6553
Satisfaction	0.7829	0.5804	0.5755	0.7023	0.3537	0.629	0.5861	0.8243	0.4899	0.7066

7.4.2. Multicollinearity

A measure or the degree of correlation among independent variables is said to be the multicollinearity (HAIR et al., 2017). In this manner, multicollinearity test is the progression to confirm data validity before proceeding to regression analysis, checking multicollinearity should be possible through bivariate of the independent variables. Each indicators variance inflation factor (VIF) value should be less than 5 et al., 2011).

Table 7: Items VIF Values in Details

Items	VIF
Benefit1	1.4030
Benefit2	1.2594
Benefit3	1.5791
Benefit4	1.5318
Benefit5	1.5626
EFF1	1.2059
EFF2	1.4233
EFF3	1.4502
EFF4	1.6583
EFF5	1.2902
EU1	1.4399
EU2	1.3528
EU3	1.5335
Eu4	1.3636
FUN1	1.4555
FUN2	1.3929
FUN3	1.4966
FUN4	1.3989
IQ1	1.5661
IQ2	1.6667
IQ3	1.4706
IQ4	1.3941
IQ5	1.3045
Loy1	1.5270
Loy2	1.6198
Loy3	1.7687
Loy4	1.8319
Loy5	1.6594
REL1	1.3126
REL2	1.3914
REL3	1.6076
REL4	1.6027
REL5	1.2444
SQ1	1.3601
SQ2	1.5088
SQ3	1.4379
SQ4	1.4253
SQ5	1.4748
Satisf1	1.3872
Satisf2	1.5738
Satisf3	1.5567
Satisf4	1.5139
Satisf5	1.4631
Sec1	1.4734
Sec2	1.7553
Sec3	1.4622
Sec4	1.2726
Sec5	1.3616
USab1	1.6125
USab2	1.8450
USab3	1.8131
USab4	1.9735
USab5	1.4721

Source: The Researcher

8. THE PROPOSED MODEL

In below the proposed framework of this study.

Figure 1: Ibrahim’s Proposed Framework for Systems Success Measurement

Source: The Researcher

9. CONCLUSION

This study aims to make a new contribution to efficiently help in systems success measure to solve the problem of higher rate of systems fail. This study provides a strong validated high-quality instrument so, researchers can use it in the future in their studies. This instrument has been developed and tested carefully. Eight PhD experts redound for validate the current instrument to make it perfect as much as possible.

All statistical required tests have been performed to approve the quality, usability validity and reliability of the instrument. Results approved that the instrument is of high quality, usability validity and reliability. This study provided also, another perspective that can be used in the future of systems success measurement which is ISO 25010 standard. This theory of ISO 25010 with its factors (usability, security, efficiency, reliability and functionality) play a vital role in the measurement of system’s success.

The current framework and instrument have been tested in three different universities researchers are required to test the framework and instrument in their domains for the purpose of the generalizability and deep confirmation. The ISO 25010 is still a new one and it’s a general metric, researchers are in the open call to facilitate it in their studies in different domains. This model can be used in the fields of information systems, software engineering quality and can be used for field of software engineering testing.

Finally, in future studies, researchers in their works are required to validate it, whether this work is a PhD, Master or an article. The validation process is a very good process and gives the authors a strong support for the work, instrument and the framework. A number of the validators could be three to six which is quite fair and enough to do guarantee for the work.

REFERENCES

ADEBOLA, B. D. (2014) The determinant of information security practices towards organizational performance in the banking sector evidence from Nigeria. Universiti Utara Malaysia.

AGHAZADEH, S.; PIRNEJAD, H.; ALIEV, A.; MORADKHANI, A. (2015) Evaluating the effect of software quality characteristics on health care quality indicators. Journal of Health Management and Informatics, v. 2, n. 3, p. 67–73.

ALADWANI, A. M.; PALVIA, P. C. (2002) Developing and validating an instrument for measuring user-perceived web quality. Information.; Management, v. 39, n. 6, p. 467–476.

ALARCÓN, D.; SÁNCHEZ, J. A.; DE OLAVIDE, U. P. (2015) Assessing convergent and discriminant validity in the ADHD-R IV rating scale: User-written commands for Average Variance Extracted (AVE, p. Composite Reliability (CR, p. and Heterotrait-Monotrait ratio of correlations (HTMT) Composite Reliability (CR, p. and Heterotrait-Monotrait Ratio of Correlations (HTMT, p. p. 1–39. In Spanish STATA Meeting

ALSHIBLY, H. H. (2015) Investigating decision support system (DSS) success: A partial least squares structural equation modeling approach. Journal of Business Studies Quarterly, v. 6, n. 4, p. 56.

ALVES, J. M.; WANGENHEIM, C.; LACERDA, T.; SAVARIS, A.; WANGENHEIM, A. (2015) Adequate software quality evaluation model v1. 0. Instituto Nacional Para Convergência Digital–INCoD, Tech. Rep.

ARDITO, C.; COSTABILE, M. F.; DE MARSICO, M.; LANZILOTTI, R.; LEVIALDI, S.; ROSELLI, T.; ROSSANO, V. (2006) An approach to usability evaluation of e-learning applications. Universal access in the information society, v. 4, n. 3, p. 270–283.

ASTANI, M.; ELHINDI, M. (2008) An empirical study of university websites. Issues in Information Systems, v. 9, n. 2, p. 460–465.

AL-AZAWEI, A.; LUNDQVIST, K. (2015) Learner differences in perceived satisfaction of an online learning: An extension to the technology acceptance model in an Arabic sample. Electronic Journal of E-Learning, 13(5, p. 408–426.

AL-MANASRA, E.; KHAIR, M.; ZAID, S. A.; TAHERQUTAISHAT, F. (2013) Investigating the impact of website quality on consumers’ satisfaction in Jordanian telecommunication sector. Arab Economic and Business Journal, v. 8, n. 1–2, p. 31–37.

BARCLAY, D.; HIGGINS, C.; THOMPSON, R. (1995) The Partial Least Squares (pls) Approach to Casual Modeling: Personal Computer Adoption Ans Use as an Illustration.

BHARATI, P.; CHAUDHURY, A. (2004) An empirical investigation of decision-making satisfaction in web-based decision support systems. Decision Support Systems, v. 37, n. 2, p. 187–197.

BROWN, R. M.; MAZZAROL, T. (2006) Factors driving student satisfaction and loyalty in Australian universities: the importance of institutional image. In 20th Annual Australia.; New Zealand Academy of Management (ANZAM) Conference Paper, p. 1–12.

CAVANA, R. Y.; DELAHAYE, B. L.; SEKARAN, U. (2001) Applied business research: Qualitative and quantitative methods. John Wiley.; Sons Australia.

CHIANG, L.; HUANG, C.-Y.; YANG, W.-C. (2011) The effect of user’s satisfaction of web security on trust in eGovernment. In the Proceedings of the 11th European Conference on EGovernment: Faculty of Administration, University of Ljubljana, Ljubljana, Slovenia, p. 140.

CHIU, P.-S.; CHAO, I.-C.; KAO, C.-C.; PU, Y.-H.; HUANG, Y.-M. (2016) Implementation and evaluation of mobile e-books in a cloud bookcase using the information system success model. Library Hi Tech, v. 34, n. 2, p. 207–223.

CHOW, W. S.; CHAN, L. S. (2008) Social network and shared goals in organizational knowledge sharing.

CONSTANTIN, A. M. (2013) The antecedents of e-satisfaction and e-loyalty. Timisoara Journal of Economics, v. 5, n. 2, 18, p. 236–252.

COUCHMAN, P. K.; FULOP, L. (2006) Building trust in cross-sector R&D collaborations: Exploring the role of credible commitments. In 22nd EGOS Colloquium, Bergen, Norway.

CRONIN, J. J.; BRADY, M. K.; HULT, G. T. M. (2000) Assessing the effects of quality, value, and customer satisfaction on consumer behavioral intentions in service environments. Journal of Retailing, v. 76, n. 2, p. 193–218.

DAVARPANAH, A.; MOHAMED, N. (2013) Human resource information systems (HRIS) success factors in a public higher education institution context. In Research and Innovation in Information Systems (ICRIIS, p. 2013 International Conference on, p. 79–84, IEEE.

DAVIS, F. D. (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, p. 319–340.

DELONE, W. H.; MCLEAN, E. R. (2016) Information systems success measurement. foundations and trends® in Information Systems, v. 2, n. 1, p. 1–116.

DELONE, W. H.; MCLEAN, E. R. (1992) Information systems success: the quest for the dependent variable. Information Systems Research, v. 3, n. 1, p. 60–95.

DERNBECHER, S. (2014) Having the mind in the cloud: organizational mindfulness and the successful use of desktop as a service. In System Sciences (HICSS, p. 2014 47th Hawaii International Conference on, p. 2137–2146, IEEE.

DEVARAJ, S.; FAN, M.; KOHLI, R. (2002) Antecedents of B2C channel satisfaction and preference: validating e-commerce metrics. Information Systems Research, v. 13, n. 3, p. 316–333.

DOHI, T.; NAKAGAWA, N. (2013) Stochastic reliability and maintenance modelling. Springer.

DREHEEB, A. E.; BASIR, N.; FABIL, N. (2016) Impact of system quality on users’ satisfaction in continuation of the use of e-learning system. International Journal of E-Education, e-Business, e-Management and e-Learning, v. 6, n. 1, p. 13.

DYBA, T. (2005) An empirical investigation of the key factors for success in software process improvement. IEEE Transactions on Software Engineering, v. 31, n. 5, p. 410–424.

EPPLER, M. J.; ALGESHEIMER, R.; DIMPFEL, M. (2003) Quality Criteria of content-driven websites and their influence on customer satisfaction and loyalty: An Empirical Test of an Information Quality Framework. In IQ, p. 108–120.

FADHEL, I. E. I. (2015) An evaluation of information system success based on students’ perspective. Universiti Utara Malaysia.

FADHEL, I. E. I.; IDRUS, S. Z. S.; IBRAHIM, A. A. E. A.; OMAR, M.; BAHASHWAN, A. A. A.;.; AL-ANSI, A. A. M. (2018a) An integration between information systems engineering and software engineering theories towards engineering a novel framework of web-based systems success for institutions based on students’ perceptions. Journal of Physics: Conference Series, v. 1019, n. 1, p. 12081. Retrieved from http://stacks.iop.org/1742-6596/1019/i=1/a=012081

FADHEL, I. E. I.; IDRUS, S. Z. S.; IBRAHIM, A. A. E. A.; OMAR, M.; BAHESHWAN, F.; ALBZEIRAT, M. K.;.; ALBZEIRAT, S. K. (2018b) Measuring System Success In New Context By Adapting DM 2003 Framework With The External Factor Management Support. Journal of Physics: Conference Series, v. 1019, n. 1, p. 12003. Retrieved from http://stacks.iop.org/1742-6596/1019/i=1/a=012003

FIELD, A. (2005) Discovering statistics with SPSS. London: Sage.

FORSGREN, N.; DURCIKOVA, A.; CLAY, P. F.; WANG, X. (2016) The Integrated user satisfaction model: Assessing information quality and system quality as second-order constructs. In system administration. Communications of the Association for Information Systems, v. 38, n. 1, p. 39.

FORNELL, C.; LARCKER, D. F. (1981) Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, p. 39–50.

GORLA, N.; SOMERS, T. M.; WONG, B. (2010) Organizational impact of system quality, information quality, and service quality. The Journal of Strategic Information Systems, v. 19, n. 3, p. 207–228.

GUADAGNOLI, E.; VELICER, W. F. (1988) Relation to sample size to the stability of component patterns. Psychological Bulletin, v. 103, n. 2, p. 265.

HAIR JR, J. F.; HULT, G. T. M.; RINGLE, C.; SARSTEDT, M. (2017) A primer on partial least squares structural equation modeling (PLS-SEM) Thousand Oaks: Sage Publications.

HENSELER, J.; RINGLE, C. M.; SARSTEDT, M. (2014) A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, v. 43, n. 1, p. 115–135.

HOLBROOK, A. L.; KROSNICK, J. A.; MOORE, D.; TOURANGEAU, R. (2007) Response order effects in dichotomous categorical questions presented orally: The impact of question and respondent attributes. Public Opinion Quarterly, v. 71, n. 3, p. 325–348.

HUANG, C.-C.; WANG, Y.-M.; WU, T.-W.; WANG, P.-A. (2013) An empirical analysis of the antecedents and performance consequences of using the moodle platform. International Journal of Information and Education Technology, v. 3, n. 2, p. 217.

JANSEN, H.; HAK, T. (2005) The productivity of the three-step test-interview (TSTI) compared to an expert review of a self-administered questionnaire on alcohol consumption. Journal of Official Statistics, v. 21, n. 1, p. 103.

JEON, M. M. (2009) Impact of perceived website service quality on customer e-loyalty on a lodging website. Iowa State University.

JONGCHUL, O.; YOON, S.-J. (2014) Validation of haptic enabling technology acceptance model (HE-TAM): Integration of IDT and TAM. Telematics and Informatics, v. 31, n. 4, p. 585–596.

KHAWAJA, K. F.; BOKHARI, R. H. (2010) Exploring the factors associated with quality of website. Global Journal of Computer Science and Technology, v. 10, n. 14, p. 37–45.

KHRED, D. A. (2017) Systems development in our country.

KIRAN, K.; DILJIT, S. (2011) Antecedents of customer loyalty: Does service quality suffice? Malaysian Journal of Library.; Information Science, v. 16, n. 2, p. 95–113.

KLINE, R. B. (2011) Principles and practice of structural equation modeling, 3rd edn Guilford Press. New York.

LIAW, S.-S.; HUANG, H.-M. (2013) Perceived satisfaction, perceived usefulness and interactive learning environments as predictors to self-regulation in e-learning environments. Computers.; Education, v. 60, n. 1, p. 14–24.

LIU, I.-F.; CHEN, M. C.; SUN, Y. S.; WIBLE, D.; KUO, C.-H. (2010) Extending the TAM model to explore the factors that affect Intention to Use an Online Learning Community. Computers.; Education, v. 54, n. 2, p. 600–610.

LUDIN, I. H. B. H.; CHENG, B. L. (2014) Factors influencing customer satisfaction and e-loyalty: online shopping environment among the young adults. Management Dynamics in the Knowledge Economy, v. 2, n. 3, p. 462.

MACCALLUM, R. C WIDAMAN, K. F.; PREACHER, K. J.; HONG, S. (2001) Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, v. 36, n. 4, p. 611–637.

MADAN, A.; DUBEY, S. K. (2012) Usability evaluation methods: a literature review. International Journal of Engineering Science and Technology, v. 4, n. 2, p. 590–599.

MALIK, B. H.; SHUQIN, C.; MASTOI, A. G.; GUL, N.; GUL, H. (2016) Evaluating citizen e-satisfaction from e-government services: A Case of Pakistan. European Scientific Journal, ESJ, v. 12, n. 5.

MARDIANA, S.; TJAKRAATMADJA, J. H.; APRIANINGSIH, A. (2015) Validating the Conceptual Model for Predicting Intention to Use as Part of Information System Success Model: The Case of an Indonesian Government Agency. Procedia Computer Science, n. 72, p. 353–360.

MBIWA, P. N. (2014) Evaluating the quality of management information systems: A case of SACCOs in Nairobi, Kenya. University of Nairobi.

MCGILL, T. J.; HOBBS, V. J.; KLOBAS, J. E. (2003) User developed applications and information systems success: A test of DeLone and McLean’s model. Information Resources Management Journal, v. 16, n. 1, p. 24–45.

MEBRATE, T. (2010) A framework for evaluating academic website’s quality from students’ perspective. TU Delft, Delft University of Technology.

MOHAMMADI, H. (2015) Investigating users’ perspectives on e-learning: an integration of TAM and IS success model. Computers in Human Behavior, n. 45, p. 359–374.

OFORI, K. S.; LARBI-SIAW, O.; FIANU, E.; GLADJAH, R. E.; BOATENG, E. O. Y. (2016) Factors influencing the continuance use of mobile social media: The effect of privacy concerns. Journal of Cyber Security and Mobility, v. 4, n. 3, p. 105–124.

OLSON, K. (2010) An examination of questionnaire evaluation by expert reviewers. Field Methods, v. 22, n. 4, p. 295–318.

PADAYACHEE, I.; KOTZE, P.; VAN DER MERWE, A. (2010) ISO 9126 external systems quality characteristics, sub-characteristics and domain specific criteria for evaluating e-Learning systems. The Southern African Computer Lecturers’ Association, University of Pretoria, South Africa.

PALLANT, J. (2007) SPSS survival manual: A step guide to data analysis using SPSS for Windows (3rd Ed) Australia: Allan.; Unwin.

PALLANT, J. (2013) SPSS survival manual. McGraw-Hill Education (UK)

PAPANIKOLAOU, K.; MAVROMOUSTAKOS, S. (2008) Web2Train: A design model for corporate e-learning systems. In BIS (Workshops), p. 155–163.

PARK, N.; RHOADS, M.; HOU, J.; LEE, K. M. (2014) Understanding the acceptance of teleconferencing systems among employees: An extension of the technology acceptance model. Computers in Human Behavior, n. 39, p. 118–127.

PRESSER, S.; BLAIR, J. (1994) Survey pretesting: Do different methods produce different results? Sociological Methodology, p. 73–104.

ROCHA, Á. (2012) Framework for a global quality evaluation of a website. Online Information Review, v. 36, n. 3, p. 374–382.

SEKARAN, U. (2003) Research methods for business: A skill building approach. John Wiley.; Sons.

SEKARAN, U.; BOUGIE, R. (2010) Research methods for business: A skill building approach. Chichester: John Willey.; Sons Ltd. 2010.

SELVAKUMAR, J. J. (2016) Impact of service quality on customer satisfaction in public sector and private sector banks. Purushartha: A Journal of Management Ethics and Spirituality, v. 8, n. 1.

SENATE, A. (2001) Universities in crisis. Canberra: Commonwealth of Australia.

SHIRATUDDIN, N. (2015) Evaluation of e-Book applications using ISO 25010. In Technology Management and Emerging Technologies (ISTMET, p. 2015 International Symposium on, p. 114–118, IEEE.

STEVENS, J. (1992) Applied multivariate statistics for the social sciences (2nd edition) Hillsdale, NJ:Erlbaum.

TABACHNICK, B. G.; FIDELL, L. S. (2007) Using multivariate statistics. Allyn.; Bacon/Pearson Education.

TANDON, U.; KIRAN, R.; SAH, A. (2017a) Analyzing customer satisfaction: users perspective towards online shopping. Nankai Business Review International, v. 8, n. 3, p. 266–288.

THEIS, B.; FROOD, J.; NISHRI, D.; MARRETT, L. D. (2002) Evaluation of a risk factor survey with three assessment methods. Chronic Diseases and Injuries in Canada, v. 23, n. 1, p. 1.

TEO, T. S. H.; SRIVASTAVA, S. C.; JIANG, L. (2008) Trust and electronic government success: An empirical study. Journal of Management Information Systems, v. 25, n. 3, p. 99–132.

UNIT, M. C. AND B. S. (2013) What thresholds should I use for factor loading cut-offs? Retrieved from http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/thresholds

VAEZI, R.; MILLS, A.; CHIN, W.; ZAFAR, H. (2016) User satisfaction research in information systems: Historical roots and approaches. Communications of the Association for Information Systems, v. 38, n. 27, p. 501–532.

VALVI, A. C.; WEST, D. C. (2013) E-loyalty is not all about trust, price also matters: extending expectation-confirmation theory in bookselling websites. Journal of Electronic Commerce Research, v. 14, n. 1, p. 99.

VAN IWAARDEN, J.; VAN DER WIELE, T.; BALL, L.; MILLEN, R. (2004) Perceptions about the quality of web sites: a survey amongst students at Northeastern University and Erasmus University. Information.; Management, v. 41, n. 8, p. 947–959.

WANG, Y.-S.; LIAO, Y.-W. (2008) Assessing eGovernment systems success: A validation of the DeLone and McLean model of information systems success. Government Information Quarterly, v. 25, n. 4, p. 717–733.

WIXOM, B. H.; WATSON, H. J. (2001) An empirical investigation of the factors affecting data warehousing success. MIS Quarterly, p. 17–41.

WOLFINBARGER, M.; GILLY, M. C. (2003) eTailQ: dimensionalzing, measuring and predicting etail quality. Journal of Retailing, v. 79, n. 3, p. 183–198.

ZAIED, A. N. H. (2012) An integrated success model for evaluating information system in public sectors. Journal of Emerging Trends in Computing and Information Sciences, v. 3, n. 6, p. 814–825.

ZEHIR, C.; SEHITOGLU, Y.; NARCIKARA, E.; ZEHIR, S. (2014) ES-quality, perceived value and loyalty intentions relationships in internet retailers. Procedia-Social and Behavioral Sciences, n. 150, p. 1071–1079.

ZEITHAML, V. A.; BERRY, L. L.; PARASURAMAN, A. (1996) The behavioral consequences of service quality. The Journal of Marketing, p. 31–46.