Abstract
The use of rubrics to assess distance educa on courses in a brazilian university -------- Resumo: Distance educa on, mediated by Informa on and Communica on Technologies, is increasingly present in the Brazilian educa onal scenario due to legisla on incen ves. In this modality, assessment to ensure eff ec ve learning, considering the par culari es of distance educa on, represents a challenge to educa on managers. This research inves gates the competences developed in two en rely distance-lectured courses in the Informa on Management degree of the Federal University of Paraná. For this purpose, the bibliographic and documental methodology was used to construct two assessment rubrics in theCoRubric pla orm, a data collec on instrument for the research. Data are analyzed using sta s cal techniques. The results show the competences developed by students and those that need improvement, as well as the correla on between the characteris cs of the courses and the learning and how much they aff ect the result. It concludes that the assessment rubrics constructed from CoRubric are capable of genera ng informa on for decision making in educa on. 
O uso de rubricas para avaliação de disciplinas à distância em uma universidade brasileira ---------El uso de rubricas para evaluación de disciplinas a distancia en una universidad brasileña
Palavras-chave: Distance Educa on. Higher Educa on. Learning Assessment. Professional Competence.
Abstract: A Educação a Distância mediada pelas Tecnologias da Informação e Comunicação é cada vez mais presente no ensino brasileiro em virtude do incen vo dado pela legislação. A avaliação nessa modalidade representa um desafi o aos docentes a fi m de garan r a aprendizagem efi caz considerando-se as par cularidades da educação a distância. Esta pesquisa inves ga as competências desenvolvidas em duas disciplinas totalmente à distância do curso de Gestão da Informação da Universidade Federal do Paraná. Para atender ao obje vo, u liza metodologia bibliográfi ca e documental com vistas a construir duas rubricas de avaliação na plataforma
INTRODUCTION
Competence-oriented educa on has been a reality in the European Union since the Bologna Declara on was signed by European countries in 1998. The European Higher Educa on Area (EHEA) has modifi ed the dynamics of universi es with the aim of establishing a common educa onal system, equang diplomas and curricula in the con nent (ÁLVAREZ-PÉREZ; LÓPEZ-AGUILAR, 2018).
The Bologna Declara on foresees the development of general and specifi c competences based on a set of knowledge, skills, and a tudes (GARCÍA; RUBIO; PORTERO, 2018) . They are understood as needs arising from a job market that is focused on a society of informa on and knowledge (MARTINS, 2014) .
In Brazil, competence-oriented teaching is done mainly through distance educa on (EaD), conceptualized as "an educa onal modality in which the didac c-pedagogical media on in teaching-learning processes occurs through the use of informa on and communica on technologies" (ALVES; ALVES; VIANA, 2015, p. 16) .
The Brazilian legisla on that establishes competence-oriented educa on is Law No. 9394, from December 20, 1996, which establishes the Guidelines and Standards of Naonal Educa on and encourages the use of EaD in Higher Educa on Ins tu ons (HEIs). Allied to this law, there is the Administra ve Rule No. 1134, from October 10, 2016, which regulates the off er of en rely or par ally distance courses in HEIs, as long as a maximum of 20% be respected in this modality.
In support of these guidelines, the Ministry of Educa on (MEC) has published the Quality Guidelines for Distance Higher Educa on, which lists recommenda ons that the evalua ons in the Distance Educa on be predominantly qualita ve in order to evaluate the competences developed (BRASIL, 2007) .
In this sense, the idea of evalua on by competences in EaD arises and one of the instruments used for this purpose in European countries is the rubric. This type of assessment provides objec ve criteria based on evidence of learning to evaluate students and their advantage is in the transparency between teacher and students in checking for errors and correctness (CEBRIÁN DE LA SERNA; MOYA, 2014) .
The Federal University of Paraná (UFPR) has a partnership with the University ofMála-ga (UMA), which enables the sharing of educa onal technologies such as CoRubric 3 , Anotaciones de vídeo, and Webquest. CoRubric is a technological pla orm for the construc on and use of rubrics in a collabora ve way. The use of rubrics as an evalua on tool, as well as others of this kind, is provided in the Pedagogical Project of Informa on Management, one of the degrees off ered by the UFPR.
The Federal University of Paraná's Informa on Management degree was founded in 1998, branching from the Librarianship degree as the job market changed and the lack of professionals capable of dealing with informa on in diff erent media was brought to a en on. Its objec ve is to "(…) train professionals with skills, no ons, and techniques aimed at solving informa on problems (…) 
Revista EDaPECI 64
related to the produc on, collec on, organiza on, dissemina on and use of informa on" (MARCHIORI, 2002, p. 83-84) .
The degree presents three pillars: I. informa on science; II. management; and III. informa on technology. Because of its general approach, students may choose to take courses in other departments of the university, channeling towards a specifi c area of knowledge that is of their interest (MARCHIORI, 2002) .
Moreover, it was the fi rst degree under such name in Brazil and also pioneer in fl exibiliza on of distance educa on curricula, spli ng into two totally distance courses and the others varying between en rely (100%) or par ally (maximum 20%) at distance courses, as provided in the reformula on of the Pedagogical Project in 2013, which was supported by three researchers from the GTEA/ UMA (SILVA et al., 2013) .
In the discussions on the Pedagogical Project, the need to defi ne the professional profi le of the Informa on Manager emerged, so that the competences were eff ec vely developed throughout the degree, considering that the courses were par ally or en harmonsteven@example.net. However, a discrepancy was noted among the competences established in the literature, the Pedagogical Project and the recommenda ons for distance educa on. Therefore, the problem that involves this research is the need to integrate the profi le of the Informa on Manager and the desired competences in EaD, in order to train competent professionals to work in the increasingly compe ve labor market.
The objec ve of this study was to assess by means of two rubrics the competences developed in the following two courses, en rely lectured at distance, from the UFPR Informaon Management degree: I. Informa on and Society; and II. Informa on Technology Management. Herein, the theore cal framework, the methodology used and the conclusions are provided as follows.
THEORETICAL FRAMEWORK
Competency is understood as a set of knowledge, skills, and a tudes, capable of explaining a given performance (FLEURY; FLEURY, 2001; CAMARGO, 2013) . It may be defi ned based on two branches: construcvist and behaviorist. The former argues that competences are acquired due to necessity, while the la er states that they are individuals' inherent characteris cs (GARIBA JÚNIOR, 2005) .
Assuming the construc vist point of view, in which competence is developed in academic or professional circles, distance educaon allows a diff erent experience than on-site educa on owing to its unique characteris cs such as virtualiza on of learning, teaching, me and communica on fl exibility -associated with the use of Informa on and Communica on Technologies (ICTs).
In this sense, it is worth no ng that other competences are developed and need to be assessed in this modality of educa on. Assessment can be understood from three diff erent approaches: diagnos c, forma ve and summa ve. The diagnos c one iden fi es the profi le of the student, his/her competences as well as possible learning pa erns.
The forma ve approach verifi es mistakes and successes and aims at success by means of feedback. This assessment is conducted throughout the learning process. Finally, the summa ve verifi es the knowledge acquired and measures the level of learning achieved (GARIBA JÚNIOR, 2005) . The la er is the most frequent approach in teaching, using quizzes and tests to quan ta vely determine the learning rate (BITENCOURT; SEVERO; GALLON, 2013) .The competence assessment is formave since it iden fi es competences acquired and is not restricted to knowledge.
In addi on, it verifi es the skills and a tudes that the individual has and the ones that he/she needs to develop (PADILHA, 2013) . Thus, competence assessment seeks to educate good professionals and properly prepare them for the job market (VELOZO, 2014) . There are many instruments for competence assessment including cri cal incidents, performance measurement scales, behavioral observa on scales, self-evalua on, and ru-brics (CAMARGO, 2013) .Rubrics are "quanta ve and/or qualita ve scales, associated with pre-established criteria that measure the student's ac ons on the aspects of the task or ac vity to be assessed" (GORDILLO; RODRÍGUEZ, 2010, p. 142) .
The assessment through rubrics verifi es mistakes, successes and the range of the objec ves proposed (CEBRIÁN DE LA SERNA; MOYA, 2014).Among its advantages, feedback on students' performance and objec ve learning assessment can be highlighted. In distance educa on, rubrics are a viable alterna ve since they are related to tasks, papers and tests, inside or outside the Virtual Learning Environment (VLE).
METHODOLOGICAL PROCEDURES
This research followed a mixed approach (qualita ve and quan ta ve in nature), and it was characterized as applied research in rela on to its purpose; exploratory and ac ons research regarding its objec ves; and bibliographical and documentary research in terms of the methodological procedures followed (GIL, 2008) . Through the bibliographic research, the literature provided the competences rendered by the EaD and the profi le of the Informa on Manager. In the documentary research, the competences foreseen in the Pedagogical Project of the degree were iden fi ed.
Based on this informa on, two rubrics were built on the CoRubric pla orm, which served as a data collec on tool. The par cipants totaled 204 students from the UFPR Informa on Management degree. Among them, 202 were enrolled in the university unl 2017, and 2 were former students who had taken the courses addressed by this research from 2014.
CONSTRUCTION AND APPLICATION OF THE RUBRICS
Based on the comparison between the competences of distance educa on and the ones provided in the Pedagogical Project of the degree and in the literature of the area, two rubrics were constructed: I. learning assessment, and II. course assessment. The competences selected for the rubric "learning assessment" were based on the comparison regarding knowledge, skills, and attudes iden fi ed through bibliographic and documentary research.Skills and a tudes were priori zed, in case the instrument shall be reused in other courses since knowledge is par cular to each subject assessed (Table 1) . The competences selected for the "course assessment" rubric were based only on bibliographical research. They include aspects related to the professor's a tude and ini aves, as well as features of the didac c material (Table 2) . The use of rubrics to assess distance educa on courses in a brazilian university 67
Revista EDaPECI 66
The rubrics "course assessment" 4 and "learning assessment" 5 were constructed in the pla orm CoRubric. A former coordinator of the Informa on Management degree, a rubric specialist and a distance educa on specialist validated them. They were applied online to students directly through the pla orm from September 11, 2017, to September 29, 2017.
The access link was shared via email along with a tutorial. The same tutorial was published in a Google Forms document and shared via Facebook. About 40 students agreed to par cipate in the research. For the Informa on and Society course there were 27 students and for the Informa on Technology Management, 23 students. It should be emphasized that there were students who took both courses and not all of them answered both rubrics for each course.
DATA TREATMENT AND ANALYSIS
For the analysis of the data of this research, sta s cal methods were used. These are described in the following analysis protocol (Table 3) . The analyses were developed with the help of R 3.5.1, RStudio 1.1.456 and Microso ® Excel 2016 tools. When necessary, a 95% confidence level was adopted.
RESULTS AND DISCUSSION
In this sec on we present the results obtained with the applica on of the analysis protocol men oned for the two courses evaluated: Informa on and Society (IS) and Informa on Technology Management (ITM).
INFORMATION AND SOCIETY (IS)
From Pearson's correla on test, in which the learning mean of the par cipants was related to each one of the competences provided Revista EDaPECI 68 in the rubric "course assessment", the following characteris cs were iden fi ed as being related to learning: a) "answer to ques ons" (r = 0.5908) and R² of 0.3490; and b) "objec ve in rela on to competences" (r = 0.6277) and R² of 0.3940. These two correla ons are presented as moderate (TAYLOR, 1990) , establishing a rela onship that is directly propor onal to learning.
According to the results, the competence "answer to ques ons" is responsible for 34.90% of the variability found in learning and, in turn, the competence "objec ve in rela on to competences" contributes with 39.40% of the present learning varia on. In terms of eff ects, these two rela ons are not negligible, since they present values for Pearson's coefficient higher than 0.50, which indicates that the eff ect of these competences in rela on to learning in this model is high (COHEN, 1992) .
In search of a model capable of represenng the learning observed in the course, a mul ple regression was elaborated with the competences "answers to ques ons" and "objec ve in rela on to competences" as independent variables, and the variable "learning" as a dependent one. The results are shown in Table 4 . The results listed in Table 4 are non-significant (F (2.9) = 4.17, p-value < 0.052). In addi on, only the value obtained for intercept was reliable (p-value < 0.0127). In this way, a simple regression was chosen, elimina ng the competence "answer to ques ons" due to its p-value. The results of this simple regression are presented in Table 5 . The results listed in Table 5 are significant (F (1.10) = 6.502, p-value < 0.0280) and the coefficients obtained for the "intercept" (40.0251; p-valor < 0.0080) and "objective in relation to competences" (0.2356; p-value < 0.0288) are reliable. The equation representing these results is the one that follows:
This simple regression is able to explain 33.34% of the learning in the studied sample. Therefore, it is noted that the defi ni on of learning objec ves for students has the potenal of improving learning.
RUBRIC "LEARNING ASSESSMENT" (IN-FORMATION AND SOCIETY)
Regarding the classes, it was verifi ed that in rela on to learning for the course of IS, the
learning = 40.0251 + 0.2356 x abjec ve in rela an ta campet (1)
The use of rubrics to assess distance educa on courses in a brazilian university 69 highest mean belongs to the class of 2015 (n = 4) (65.67, with a standard devia on of 17.04). In rela on to the competences, the mode of the weight obtained by the level of competence was used. In CoRubric the competences have a rela ve quan ta ve value associated with each level, which is used to calculate the fi nal mean. Competences whose mode were higher than 50% were selected (Table 6 ). The competences that did not reach the ideal level were "teamwork", "use of ICTs", "eff ec ve communica on" and "informa on proac vity and problem-solving". Some competences were not superior to 50% due to a lack of consensus among the par cipants and, therefore, are not present in Table 6 ("con nuous learning", "cri cal and inves ga ve character" and "ethical conduct").
The "mathema cal, verbal and wri en logical reasoning" competence did not reach the ideal level, but the par cipants indicated that they had indeed developed verbal and wri en logical reasoning, which corresponds to the objec ve of the course. The other competences achieved the ideal level.
RUBRIC "COURSE ASSESSMENT" (INFOR-MATION AND SOCIETY)
Regarding the classes, and in rela on to the evalua on of the course, it was verifi ed that the highest mean belongs to the class of 2015 (n = 4) (66.67, with a standard devia on of 9.81). The highest value corresponds to 76.66, while the minimum iden fi ed was 56.66.
The class of 2014 (n = 7) had a mean of 59.28, with a standard devia on of 15.74. The maximum value found was 90.00, and the lowest one was 41.66. On the other hand, the class of 2017 (n = 3) reached the lowest mean among the groups (23.33, with a standard devia on of 23.33), presen ng as maxi-
Revista EDaPECI 70
mum and minimum values, 46.66 and 0.00, respec vely.
When the Shapiro-Wilk test was used to evaluate the distribu on of the data, it was found that they did not deviate dras cally from a normal distribu on: 2014 (W = 0.89912; pvalor < 0.3257); 2015 (W = 0.88207; p-valor < 0.3476); 2017 (W = 1; p-valor < 1). In addi on, the variances presented in the groups can be considered homogeneous (K² (2) = 1.5695, pvalue < 0.4562).
Thus, the ANOVA test was used to verify possible diff erences between the means of the classes studied. Based on the result of this test (F (2, 12) = 7.019; p-value < 0.0108), it is possible to affi rm that there are sta s cally signifi cant diff erences between the means of the classes with regard to the evalua on of the course.
In order to iden fy in which rela ons these diff erences reside, the Tukey test (Tukey HSD) was applied. The diff erences were found in the comparison of the class of 2017 with the class of 2015 (p-value < 0.0123) and 2014 (p-value < 0.0202). Between 2014 and 2015, it is not possible to state that there is a sta s cally signifi cant diff erence between the means (p-value < 0.7516). Competency analysis indicated that "assessment criteria", "reinforcement acvi es", "commented responses" and "self-assessment" did not reach ideal levels (Table 7) . Table 7 -Most indicated criteria per competence for the rubric "course assessment"
There was no consensus regarding "answer to ques ons", "resources availability", "content", "introductory module" and "peripheral elements" as most indicated criteria. Moreover, the sec ons that were improved are "resources planning", "competency goals", "addi onal bibliography and websites" and "language".
INFORMATION TECHNOLOGY MANAGE-MENT (ITM)
From the Pearson's correla on test, in which the par cipants' learning mean was related to each of the competences provided in the "course assessment" rubric, the following characteris cs were iden fi ed as being related to learning: a) "enhancement ac vi es" (r = 0.5064) and R² of 0.2564; b) "resources availability" (r = 0.5422) and R² of 0.2940; and c) "introductory module" (r = 0.5120) and R² of 0.2621.
Source: Elaborado by the authors (2019).
These three correla ons are presented as moderate (TAYLOR, 1990) , establishing a relaonship that is directly propor onal to learning. The competence "enhancement ac vies" is responsible for 25.64% of the variability found in learning while the "resources availability" competence contributes with 29.40% of the present learning varia on. Moreover, the competence "introductory module" corresponds to 26.21% of the learning varia on.
In rela on to the eff ects, these three correla ons are not negligible, since they present values for the Pearson's coeffi cient higher than 0.50, indica ng that the eff ect of these competences in rela on to the learning is high (COHEN, 1992; FIELD, 2009) .In search of an equa on capable of represen ng the learning observed in the course, a mul ple regression was elaborated with these three competences as independent variables and the "learning" variable as a dependent one. The results are shown in Table 8 .
The use of rubrics to assess distance educa on courses in a brazilian university 71 The results listed in Table 8 proved to be signifi cant (F (3.9) = 5.561, p-value < 0.019). However, a en on is drawn to the fact that the coeffi cient for the competence "Resources availability" is not reliable (p-value < 0.4060). Thus, a new regression was chosen, without the presence of this one. The results are presented in Table 9 .
Source: Elaborado by the authors (2019). Table 9 -Mul ple regression without the "material availability"
This regression explains 54.40% of the learning in the studied sample. From this result, we see that the enhancement ac vi es and the introductory module variables together are able to explain more than half of the learning in the course.
It is observed that the results listed in Table 9 are signifi cant (F (2.10) = 8.157; p-value < 0.007) and that the coeffi cients obtained for the "intercept" (45.0060; p-valor < 0.000), "enhancement ac vi es" (0.2356; p-value < 0.0118) and "introductory module" (0.2330;pvalue < 0.0113) are reliable. Equa on 2 represents these results. learning = 45.0060 + 0.2356 x anhancement ac vi es + 0.2330 x intraductary madule (2)
RUBRIC "LEARNING ASSESSMENT" (IN-FORMATION TECHNOLOGY MANAGEMENT)
Regarding the classes, the highest mean belongs to the class of 2013 (n = 5) (88.25, with a standard devia on of 10.83). The highest value corresponds to 100.00, while the minimum value iden fi ed was 75.21.
The class of 2014 (n = 7) had a mean of 78.06, with a standard devia on of 18.06. The maximum value found was 100.00, and the lowest value was 56.80. In turn, the class of 2015 (n = 3), presented the lowest mean (70.83 with standard devia on of 9.27). The maximum and minimum values were 80.83 and 62.50, respec vely.
When the Shapiro-Wilk test was used to evaluate the distribu on of the data, it was found that they did not deviate dras cally from a normal distribu on: 2013 (W = 0.91614; p--valor < 0.5054); 2014 (W = 0.86495; p-valor < 0.1676); 2015 (W = 0.97589; p-valor < 0.7023) . In addi on, it was iden fi ed that the variances presented in the classes can be considered homogeneous (K² (2) = 1.6133; p-value < 0.4464). Thus, the ANOVA test was used to verify possible diff erences between the means of the studied classes.
Revista EDaPECI 72
Based on the result of this test (F (1.13) = 3.045, p-value < 0.105), it is not possible to state that there are sta s cally signifi cant diff erences between the means of the studied classes regarding learning for this course. The skills that were iden fi ed by more than 50% of respondents were: "Informa on analysis and interpreta on", "systemic thinking" and "Informa on proac vity and problem-solving". The others reached a value close to the ideal one (Table 10) . Table 10 -Most indicated criteria per competence for the rubric "learning assessment"
The competences that were iden fi ed as in the ideal level by more than 50% of the par cipants were "informa on analysis and interpreta on", "systemic thinking" and "informa on proac vity and problem-solving". The other competences reached a value close to the ideal one.There were no consensus regarding the "cri cal and inves ga ve character", "mathema cal, verbal and wri en logical reasoning", "ethical conduct", "organiza on, discipline, responsibility and me management" and "autonomy" competences.
RUBRIC "COURSE ASSESSMENT" (INFOR-MATION TECHNOLOGY MANAGEMENT)
In regards to the classes, the highest mean belongs to the class of 2014 (n = 7) (85.59, with a standard devia on of 12.04), in terms of the course evalua on. The highest value corresponds to 100.00, while the minimum was 70.00. The class of 2013 (n = 5) had a mean of 84.00, with a standard devia on of 17.54. The maximum value found was 100.00, and the lowest was 56.66.
On the other hand, the class of 2015 (n = 3) reached the lowest mean among the groups (72.22, with a standard deviation of 12.61), having presented 86.66 and 63.33 as maximum and minimum values, respectively. When the Shapiro-Wilk test was used to evaluate the distribution of class data, it was found that they did not deviate drastically from a normal distribution: 2013 (W = 0.87514; p-valor < 0.2879); 2014 (W = 0.92027; p-valor < 0.4715); 2015 (W = 0.85465; p-valor < 0.253).
In addi on, it was iden fi ed that the variances presented in the classes can be considered homogeneous (K² (2) = 0.66532; p-value < 0.717). Thus, the ANOVA test was used to verify possible diff erences between the means of the studied classes. Based on the result of this test (F (1.13) = 0.953; p-value < 0.347), it is not possible to affi rm that there are sta s cally signifi cant diff erences between the means of the classes studied regarding the evalua on for this course.
In terms of the competences of the course, the only one not observed by more than 50% of the respondents was "self-assessment". The others reached the desired ideal level for EaD subjects (Table 11) . The use of rubrics to assess distance educa on courses in a brazilian university 73 Table 11 -Most indicated criteria per competence for the rubric "course assessment"
Regarding the competences of the course, the only one that was not indicated by more than 50% of the par cipants was "self--evalua on". The others reached the ideal level for distance educa on courses. No consensus was obtained among the par cipants for the "assessment criteria", "enhancement ac vi es", "resources availability", "commented answers" and "introductory module" competences.
FINAL CONSIDERATIONS
The objec ve of this study was to evaluate, by means of rubrics, the competences developed in two at a distance course from the Informa on Management degree of UFPR. For that, two rubrics were developed: one to evaluate the learning itself and another to evaluate the course, considering the didac c material and the teacher's a tude.
The CoRubric pla orm, on which they were built, is a tool capable of providing qualita ve and quan ta ve data, which allows an in-depth analysis of the learning taking place, genera ng insights for further studies on this topic.
With the sta s cal methods applied, it was possible to diagnose the situa on of each of the courses in rela on to the development of the competences that were expected in the EaD, as well as the skills and a tudes of the Informa on Manager according to the literature.
Although the data presented moderate correla on and regression results for the ex- plana on of learning from the characteris cs of the teaching material and the teacher's attude, some of them showed the poten al for future studies that involve competency-based evalua on by rubric.
One of the limita ons of this research was the diffi culty in ge ng students to par cipate because they had never used the CoRubric plaorm before, even with a suppor ng tutorial. In addi on, because there were two rubrics, the instrument was extensive. It is important to emphasize that this is a pilot study and that new applica ons of the method are necessary to reach conclusive results.
Nevertheless, the use of rubrics through the CoRubric pla orm to assess learning, whether in a face-to-face or in a virtual environment, is valid as a transparent and accessible way of competency-based assessment and, at the same me, provide data for the decision-making of teachers in rela on to their courses. 
REFERENCES


{'paper_id': '201428069', 'title': 'The use of rubrics to assess distance education courses in a brazilian university', 's2_url': 'https://api.semanticscholar.org/CorpusID:201428069', 'source': 'S2ORC'}