text:
Judul Design of an assessment system for collaborative problem solving in
STEM Education
Penulis Kuen-Yi Lin, Kuang-Chao Yu, Hsien-Sheng Hsiao, Yih-Hsien Chu,
Yu-Shan Chang, Yu-Hung Chien
Tahun Terbit 2015
Penerbit Journal Computer Education 2(3):301-322
DOI 10.1007/S40692-015-0038-x
Reviewer Lanuihsan, S.Pd, Gr
Tanggal Review 9 Juni 2016
Tujuan Penelitian ini mengembangkan sistem penilaian untuk mengevaluasi SMP
kemampuan pemecahan masalah kolaboratif siswa (CPS) dalam konteks
sains, pendidikan teknologi, teknik, dan matematika (STEM).
Subyek Penelitian Pelajar SMP di Taiwan sebanyak 222 orang
Usia 13-15 tahun
Metode Analisis statistik
Definisi Operasional Untuk menilai kemampuan pemecahan masalah kolaboratif siswa, tiga
Variabel Independen besar keterampilan pemecahan masalah kolaboratif diadopsi di sini, dan
terkait-silang dengan empat proses pemecahan masalah utama untuk
membentuk matriks keterampilan pemecahan masalah kolaboratif (lihat
OECD 2013). Tiga keterampilan dan empat proses tercantum dalam Tabel
1. Untuk memiliki keterampilan membangun dan mempertahankan
pemahaman bersama, siswa harus memiliki kemampuan untuk
mengidentifikasi pengetahuan timbal balik, untuk mengidentifikasi
perspektif dari agen lain, dan untuk membangun visi bersama dari masalah
negara (Dillenbourg dan Traum 2006; OECD 2013). Untuk keterampilan
mengambil tindakan yang tepat untuk memecahkan masalah, siswa harus
dapat memahami kendala masalah, membuat sasaran tim menuju solusi,
mengambil tindakan pada tugas, dan memantau hasil dalam kaitannya
dengan grup dan sasaran-sasaran masalah ( OECD 2013). Untuk
membangun dan mempertahankan organisasi kelompok, siswa harus dapat
memahami peran mereka sendiri dan peran dari agen lain, mengikuti aturan
keterlibatan untuk peran mereka, memantau organisasi kelompok, dan
memfasilitasi perubahan yang dibutuhkan untuk menangani hambatan
terhadap masalah (OECD 2013)
Cara & Alat Ukur Sebanyak 222 siswa SMP Taiwan berusia 13–15 tahun berpartisipasi dalam
Variabel pelajaran ini. Mereka berasal dari sembilan kelas di dua sekolah yang
berbeda, dan menghabiskan tiga minggu menyelesaikan delapan tugas
dalam sistem berbasis web CPS. Pada minggu pertama, para guru
menjelaskan pengoperasian sistem penilaian, dan para siswa diharapkan
untuk menyelesaikan dua tugas. Pada minggu kedua dan ketiga, siswa
harus menyelesaikan tiga tugas per minggu. Sehubungan dengan format
sistem CPS berbasis web, jumlah agen komputer yang berinteraksi dengan
siswa bervariasi. Siswa bekerja dengan satu agen (awalnya selama 4
modul) dan kemudian dua agen (selanjutnya, untuk 4 modul lainnya).
Hasil Penelitian Menurut data dan analisis yang disajikan di atas, sistem penilaian yang
dikembangkan dalam penelitian ini terdiri dari item dengan diskriminasi
yang sulit diterima dan memuaskan, dan validitas konten dan validitas
terkait kriteria juga dinilai baik. Secara keseluruhan, sistem penilaian untuk
pemecahan masalah kolaboratif dalam pendidikan STEM yang disajikan
oleh penelitian ini efektif dalam menilai keterampilan pemecahan masalah
kolaboratif siswa. Meskipun mayoritas interkorelasi yang digunakan
sebagai bukti konsistensi hasil di delapan tugas dalam keterampilan CPS
yang diberikan secara keseluruhan sangat signifikan secara statistik dan
juga dalam banyak kasus nilai korelasi yang cukup tinggi menjadi dianggap
Judul Design, implementation, and evaluation of an ePortfolio approach to
support Faculty development in vocational education
Penulis Annemarieke Hoekstra, Jocelyn R. Crocker
Tahun Terbit 2015
Penerbit Journal Studies in Educational Evaluation 562 (2015):xxx-xxx
DOI 10.1016/j.stueduc.2015.03.007 0191-491x
Reviewer Lanuihsan, S.Pd, Gr
Tanggal Review 9 Juni 2016
Tujuan Makalah ini menjelaskan desain, implementasi, dan evaluasi dari inisiatif
pengembangan fakultas yang disebut pendekatan ePortfolio, yang
dirancang untuk mendorong pembelajaran yang terjadi secara alami di
tempat kerja.
Subyek Penelitian Lokasi penelitian adalah institut kejuruan pasca-sekolah menengah Kanada
Barat yang menawarkan pelatihan pemagangan, program diploma dua
tahun, program gelar, dan sertifikat profesional di bidang perdagangan,
konstruksi bangunan dan desain, teknik dan ilmu terapan, kesehatan dan
keselamatan, TI dan elektronik, bisnis dan administrasi, serta perhotelan
dan seni kuliner. Lembaga ini memiliki sekitar 1100 fakultas dan 80.000
mahasiswa penuh waktu dan paruh waktu
Metode Metode survey dan wawancara
Definisi Operasional Berdasarkan perbandingan ini, kami membuat kategori di seluruh peserta
Variabel Independen serta hitungan berapa banyak peserta yang masuk dalam kategori ini.
Misalnya, adopsi tema pendekatan ePortfolio termasuk tiga kategori: (1)
tidak ada portofolio yang dibuat, (2) halaman portofolio dengan beberapa
dokumen yang diunggah, dan (3) ePortfolio yang lebih terperinci.
Cara & Alat Ukur Pendekatan ePortfolio diimplementasikan dalam dua pilot.
Variabel Setelah masing-masing percontohan, baik survei dan wawancara data
dikumpulkan untuk menentukan adopsi, persepsi fakultas, dan persepsi
fakultas tentang dampak pendekatan ePortfolio
Pendekatan ePortfolio diujicobakan dalam dua fase. Pertama-tama, para
dekan dari lima fakultas di lembaga diminta untuk mencari kursi-kursi
departemen yang bersedia untuk mencoba pendekatan baru dalam (salah
satu dari mereka) program-program yang lebih kecil. Yang kedua, para
dekan diminta untuk mencari kursi departemen dari dua hingga tiga
program yang lebih besar. Yang pertama pilot dilakukan dari September –
Desember 2011 dengan 32 peserta (termasuk 5 kursi dan 27 anggota
fakultas) dari lima program dalam perdagangan, ilmu kesehatan, bisnis, dan
media dan desain. Percobaan kedua dilakukan dari September – Desember
2012 dengan 75 peserta dari 12 program di 9 departemen 2 termasuk
program dalam perdagangan, teknik, ilmu terapan, dan kesehatan. Satu
program berpartisipasi dalam kedua pilot. Jumlah total departemen yang
berpartisipasi adalah 13: empat di pilot pertama, delapan di bagian kedua,
dan satu yang berpartisipasi dalam keduanya.
Evaluasi difokuskan pada dampak pendekatan ePortfolio dan
pelaksanaannya, termasuk rincian mengenai sesi orientasi yang diadakan,
survei evaluasi siswa baru, survei baru yang digunakan oleh kursi untuk
mengumpulkan umpan balik mengenai kinerja mereka, tingkat respons
siswa, dan alat umpan balik.
Software pengolah data menggunakan Mahara version 1.4.6
Hasil Penelitian Tabulasi silang yang serupa menunjukkan bahwa semua responden
yang mengumpulkan umpan balik pada aspek-aspek tertentu dari pekerjaan
mereka juga menunjukkan bahwa aspek-aspek tersebut sebenarnya
merupakan bagian dari pekerjaan mereka.
Tabel 2 menunjukkan bahwa mayoritas peserta (69%) mengumpulkan
lebih banyak dan / atau jenis umpan balik yang berbeda dalam pilot
daripada sebelum pilot. Dari lima komentar untuk pertanyaan ini, tiga
menunjukkan perbedaannya adalah pengumpulan umpan balik rekan.
Komentar lain menyatakan ''tidak ada perbedaan.' 'Hanya 31% yang
melaporkan bahwa mereka tidak mengumpulkan umpan balik yang lebih
banyak atau berbeda dari sebelumnya.
Wawancara mengkonfirmasi temuan survei, bahwa mayoritas (9 dari
12) orang yang diwawancarai mengumpulkan umpan balik. Tujuh dari 12
orang yang diwawancara menciptakan portofolio, meskipun hanya dua
yang menciptakan portofolio yang lebih rumit. Dari wawancara dengan
fakultas dan kursi, kami belajar bahwa setiap departemen mengadopsi
pendekatan ePortfolio ke tingkat yang berbeda selama uji coba pertama.
Dua kursi tidak mendiskusikan dan tidak berencana membahas portofolio
dengan anggota fakultas yang melapor kepada mereka. Tiga kursi
mengadakan diskusi dengan beberapa anggota fakultas dan berencana
untuk duduk dengan anggota fakultas yang tersisa
Judul Two dimensional efficiency measurements in vocational education
Penulis Peter Fieger, Renato Andrin Villano, John Rice and Ray Cooksey
Tahun Terbit 2015
Penerbit International Journal of Productivity and Performance Management Vol.66
No. 2. 2017 pp.196-215
Reviewer Lanuihsan, S.Pd, Gr
Tanggal Review 9 Juni 2016
Tujuan Tulisan ini berusaha memberikan analisis seperti itu melalui penerapan
pendekatan integrative untuk pengukuran efisiensi untuk mengeksplorasi
aktivitas dan efisiensi pengiriman (beban mengajar) dan efisiensi hasil
(hasil kerja
Subyek Penelitian sektor pendidikan dan pelatihan kejuruan (VET) Australia
Metode Dalam penelitian ini penulis menggunakan data yang berkaitan dengan
demografi kohort siswa, karakteristik kelembagaan dan data hasil
pendidikan, sementara menggunakan analisis stokastik, untuk
mengembangkan dua langkah dan model efisiensi yang berbeda.
Definisi Operasional Dalam studi ini, kami akan menetapkan dua jenis efisiensi yang berbeda di
Variabel Independen sektor TAFE Australia dan menggunakan analisis parametrik stokastik
stikastik (SFA) untuk menentukan efisiensi masing-masing lembaga.
Model empiris pertama dirancang untuk memperkirakan efisiensi dalam
transformasi sumber daya keuangan menjadi jam mengajar (dari sini
disebut "model beban mengajar"). Model kedua memperkirakan efisiensi
transformasi sumber daya pengajaran menjadi hasil pasca-studi, yaitu,
tingkat kerja lulusan TAFE (dari sini disebut "model hasil pekerjaan")
Cara & Alat Ukur Untuk memperkirakan efisiensi "jam pengajaran" dan efisiensi "hasil
Variabel pekerjaan" berdasarkan metodologi stochastic frontier yang dikembangkan
oleh Aigner et al. (1977). Kontribusi utama dari para penulis ini adalah
pengenalan pendekatan baru untuk spesifikasi istilah kesalahan, yaitu,
pemisahannya ke dalam istilah "noise" normal dan satu sisi istilah
inefisiensi. Fungsi produksi frontier stokastik merupakan perluasan fungsi
Cobb-Douglas (1928) klasik
Ini akan mencakup grafik dari kedua komponen efisiensi dan analisis
klaster untuk menentukan "kelompok efisiensi". Akhirnya, mereka akan
menggunakan analisis diskriminan kanonik dengan tujuan mengembangkan
tipologi lembaga yang efisien
Hasil Penelitian di kedua model yang telah diamati beberapa ketidakefisiensian yang jelas.
Inefisiensi ini terutama terkait dengan tingkat keterpencilan dan
karakteristik siswa, yang keduanya dapat dilihat sebagai eksogen bagi
TAFE itu sendiri. Sebagai contoh, lembaga TAFE yang paling tidak efisien
lebih mungkin ditemukan di lokasi terpencil, memiliki persentase laki-laki
yang lebih tinggi, dan proporsi yang lebih besar dari orang-orang dari latar
belakang yang tidak berbahasa Inggris.
inefisiensi ini didorong oleh kombinasi faktor-faktor yang saling terkait,
termasuk lokasi geografis, infrastruktur yang tersedia dan tidak adanya
jenjang profesi lulusan
J. Comput. Educ. (2015) 2(3):301–322
DOI 10.1007/s40692-015-0038-x
Abstract This study developed an assessment system for evaluating junior high
students' collaborative problem-solving skills (CPS) in the context of science,
technology, engineering, and mathematics (STEM) education. The main theoretical
basis for designing the CPS learning goals of this assessment system is the matrix of
collaborative problem solving proposed by the Organisation for Economic Co-
operation Development (OECD). In this assessment system, there are eight
assessment modules in STEM education developed by four groups of researchers
who participated in this study. The modules were delivered as a web-based learning
platform, where the computer was programmed to act as the agent of collaboration
(rather than human-to-human collaboration), thus making this exploratory study
more unique. The eight modules included problem tasks such as designing shelves,
using a microwave oven, construct a house, etc. In order to ensure the validity of the
assessment system, including the eight assessment modules in STEM education,
three specialists were invited to examine the eight modules' content validity.
Additionally, in order to ensure criterion-related validity of the effectiveness of the
modules, they were tested among 222 Taiwanese junior high students. Discrimi-
nation and difficulty indices were obtained for student CPS performance on each of
the eight modules. Furthermore, intercorrelations were obtained for the students'
performance on each of the eight assessment modules in relation to overall and each
of three collaborative problem-solving skills, i.e., (1) Establishing and maintaining
shared understandings, (2) Taking appropriate action to solve the problem, and (3)
Establishing and maintaining team organization. The following conclusions are
drawn: (1) there was an evidence that the assessment system items had acceptable
difficulty and satisfactory discrimination; (2) the intercorrelations of the students'
302 J. Comput. Educ. (2015) 2(3):301–322
performance on the eight tasks in relation to overall and each of the three collab-
orative problem-solving skill dimensions were sufficiently significant, in most cases,
to conclude that the eight tasks are consistently producing the same performance
outcomes for overall and each of the three skills; that is, this assessment system was
deemed effective for evaluating junior high students' collaborative problem-solving
skills in STEM education; and (3) with respect to some of the challenges in eval-
uating students' collaborative problem-solving skills, one of the most demanding for
us was designing a series of tasks and questions in evaluating students' performance
in establishing and maintaining team organization. Hence, we recommend that
further research on students' performance in establishing and maintaining team
organization should be the focus of future studies.
Background literature
J. Comput. Educ. (2015) 2(3):301–322 303
traditional role of subject matter will change, and as far as pedagogy, the most
important thing will be not how teachers ''teach'' their students, but how they
''communicate with'' their students and try to help them solve problems
Since the collaborative problem-solving skills are important to our students
(Darling-Hammond 2011; Greene 2011), how to explore students' performance in
collaborative problem-solving skills is an important issue. With respect to the
development of a system to assess collaborative problem-solving skills, two
important issues are worthy of further exploration as follows: (1) What are the
merits of using a computer agent (rather than a human agent)? Many scholars (e.g.,
Johnson and Valente 2008; Rosen and Tager 2013; VanLehn et al. 2007) used a
computer agent in training or evaluating, but further exploration is needed on the
best way for using a computer agent in assessing students' collaborative problem-
solving skills. (2) There is a need for a sound theoretical basis for the design of the
collaborative tasks. The OECD developed a matrix of collaborative problem solving
(OECD 2013) that is promising and we apply it here, but recent evidence is that it
has been difficult for researchers in designing collaborative tasks and assessing
students' performances using the matrix of collaborative problem solving. For
example, Kuo (2014) developed a collaborative problem-solving system, but the
tasks were simple, and there were difficulties in assessing the students' performance
based on skills derived from the matrix of collaborative problem solving. That is, a
more effective design of the collaborative tasks is needed for further exploration. In
addition to these cited challenges in assessing collaborative problem-solving skills,
Kuo and Wu (2013) also found that only 19 out of 66 computer-based assessments
took advantage of dynamic and interactive media for item presentations. Therefore,
in light of these issues and challenges, our goal was to explore more fully on how to
design and assess a good collaborative task, while using the computer as the agent
during collaboration, including a dynamic and interactive environment, to assess
students' collaborative problem-solving skills.
In order to construct a dynamic and interactive environment, this study included the
design of a series of collaborative problem-solving tasks that are related to junior
high school students' daily life experiences, instead of adopting traditional subject
matter-based problems. By incorporating tasks more grounded in daily life
experiences, we hope to increase the motivation and engagement of the students
during the CPS tasks. In order to solve these real-life problems collaboratively,
students have to use science, technology, engineering, and mathematics (STEM)
knowledge, as recommended in modern STEM education educational systems for
the twenty-first century (Bybee 2013). To sum up, this study develops and applies
an assessment system for evaluating students' collaborative problem-solving skills
in STEM education using a web-based delivery system and the computer as the
collaborative agent. The study employs the collaborative problem-solving frame-
work proposed by the Organisation for Economic Co-operation and Development
(2013) as a theoretical basis for the design of the CPS tasks and CPS skills included
304 J. Comput. Educ. (2015) 2(3):301–322
in the assessment system, and we have analyzed its effectiveness when applied
among 222 junior high students in Taiwan. The assessment system has eight
modules each based on a different situation in daily life; and a computer agent (or
two agents) that interact with students (simulating human kinds of verbal
interactions) in order to assess their collaborative problem-solving skills.
Research questions
Two research questions guided the study: (1) Is there evidence that the eight
modules have good validity in assessing junior high school students' collaborative
problem-solving skills? (2) What are some of the difficulties in developing an
assessment system for collaborative problem solving in STEM education?
Related work
J. Comput. Educ. (2015) 2(3):301–322 305
Computer agent
306 J. Comput. Educ. (2015) 2(3):301–322
learners and computer agents has been shown to be useful for developing students'
collaborative problem-solving skills, but more in-depth studies are needed.
Consequently, this study utilizes a computer agent method instead of a learner–
learner collaborative method. Moreover, this study incorporated a new assessment
system for collaborative problem solving in STEM education that encouraged
students to engage with practical problems during the CPS that are relevant to their
daily life; and this is an additional difference compared to other collaborative
problem-solving systems.
Theoretical framework
J. Comput. Educ. (2015) 2(3):301–322 307
scripted prompts from the computer agent is how we assess the effectiveness of the
CPS system in promoting students' learning of collaborative problem-solving skills.
Table 1 Matrix of collaborative problem solving including the specific CPS skills (A1–D3) that were
Skills process (1) Establishing and (2) Taking appropriate (3) Establishing and
maintaining shared action to solve the problem maintaining team
understanding organization
(A) Exploring (A1) Discovering (A2) Discovering the types (A3) Understanding roles
and perspectives and of collaborative interaction to solve the problem
Understanding abilities of team needed to solve the
members problem, along with the
(B) Representing (B1) Building a shared (B2) Identifying and (B3) Describing roles and
and representation and describing tasks to be team organization
Formulating negotiating the completed (communication
meaning of the protocols/rules of
problem engagement)
(C) Planning and (C1) Communicating (C2) Enacting plans (C3) Following rules of
Executing with team members engagement, (e.g.,
about the actions to prompting other team
be/being performed members to perform their
(D) Monitoring (D1) Monitoring and (D2) Monitoring results of (D3) Monitoring,
and Reflecting repairing the shared actions and evaluating providing feedback, and
understanding success in solving the adapting team
problem organization and roles
308 J. Comput. Educ. (2015) 2(3):301–322
System framework
The design of the assessment system is centered on eight assessment modules for
applying STEM content relevant in daily life for the participating Taiwanese
students (see Fig. 1). The modules include eight practical problem-solving tasks:
building shelves, using a microwave oven, defusing a bomb, interior design, two
J. Comput. Educ. (2015) 2(3):301–322 309
Fig. 1 Assessment modules including eight STEM tasks largely centered on practical type problems:
Building Shelves, Using a Microwave Oven, Defusing a Bomb, Interior Design, Two days' Travel in
Kaohsiung, Buying a Cell Phone, Building a House, and Designing a Desk Tidy
days' travel in Kaohsiung, buying a cell phone, building a house, and designing a
desk tidy. With respect to computer agents, students were required to work with the
agent(s) to solve problems collaboratively; they communicate with the computer
agents by responding using the keyboard to insert their responses, or selecting an
item onscreen (see Fig. 2). Once the students input what they want to say, the
computer agent searches the item bank and responds to the students (see Fig. 3). In
order to evaluate students' collaborative problem-solving skills, many possible
reactions to the student's input during the problem-solving process are included in
this system, these include one where wrong guidance is offered by the computer
agent, in order to evaluate students' CPS performance in relation to these
challenging situations.
The design elements of task characteristics, problem scenario, medium, and team
composition were considered in the process of developing the CPS system (OECD
2013). With regard to task characteristics, all the tasks in this study were related to
students' daily life, and the students and the computer agents had to play different
roles and take different responsibilities to solve the problems. With regard to the
problem scenario, as mentioned above, tasks that could plausibly appear in the daily
lives of these Taiwanese students were adopted. As for medium, students were
provided with sufficient information to make decisions by the computer agents as
they worked together, but students had to determine the correctness of the
information offered by the agent. Finally, for team composition, students worked
with either one or two computer agents and were required to play different roles in
the different scenarios.
310 J. Comput. Educ. (2015) 2(3):301–322
Fig. 2 Assessment system interface that the student uses to collaboratively interact with computer agents
assessment module
Communication module
Data input
Students Data analysis module Search
Reaction bank
(Return the data) Reaction module Reaction
Fig. 3 Assessment system diagram showing the information flow and interactions of student and
computer agents, including the role of the assessment module and its embedded agent where student input
data are analyzed, and appropriate items are retrieved from the Item bank to provide a reaction to the
students' input
J. Comput. Educ. (2015) 2(3):301–322 311
Fig. 4 Table of students' original performance scores on questions A1–D3 derived from the OECD
collaborative problem-solving matrix. For example, the original scores for the shelf construction task
(line 2), include the scores for sample questions C1 and C2 (1.00 and 0.60, respectively): These two
questions are: (C1) Do you think that a small and simple set of plastic shelves is a good choice for solving
this problem?; (C2) Do you know the installation process of building shelves?
312 J. Comput. Educ. (2015) 2(3):301–322
The higher the intercorrelations in each table, the more likely the eight scenarios
were consistent, yielding the same results for the particular identified skill, hence
supporting the validity of the design.
A total of 222 Taiwanese junior high students aged 13–15 years old participated in
this study. They came from nine classes in two different schools, and spent three
weeks finishing the eight tasks in the CPS web-based system. In the first week, the
teachers explained the operation of the assessment system, and the students were
expected to complete two tasks. In the second and third weeks, students had to finish
three tasks per week. With respect to the format of the web-based CPS system, the
number of computer agents that the student interacted with varied. The student
worked with one agent (initially during 4 modules) and later two agents
(subsequently, for 4 other modules).
Research Question 1 Is there evidence that the eight modules have good validity in
assessing junior high school students' collaborative problem-solving skills?
Average student performance using the assessment system was about 0.7 (out of a full
mark of 1.0; this is equal to 70 % correct in the test, see Table 2). Students tended to
have lower CPS performance if the problem scenario was related to students' hands-on
experiences (that is, making shelves or designing and tidying a desk). Based on the
Table 2 Means (M) and standard deviations (SD) for Students' performance on the assessment modules
Tasks Establishing and Taking Establishing and Collaborative
maintaining shared appropriate maintaining team problem solving
understanding action to solve organization
the problem
N = 222; the maximum possible score for each item was 1.0
J. Comput. Educ. (2015) 2(3):301–322 313
results, it is hard to find related studies for comparison to our finding. Because, many
prior studies focused on different dimensions to be assessed than the ones we used,
such as using the computer agent in designing reading, writing, and communicating
systems (McNamara et al. 2007; Johnson and Valente 2008), developing an inferential
problem-solving system (Azevedo et al. 2010; Biswas et al. 2010; Cai et al. 2011); and
so on. In our assessment system, the computer agent played an important role as a
technology specialist, and was designed to offer as much valuable information as
possible to the students, but the results were not as expected. Among possible reasons
for this discrepancy, the student's prior experiences with the particular kinds of hands-
on tasks we included in the CPS system may have been too limited. If the students had
very little prior experience with some of the practical kinds of tasks we included, it may
have been difficult for them to work effectively with the kind of information that the
computer agent was providing. That is, students' prior knowledge in the specific
domain could be an important factor in influencing their effective interaction with the
information provided by the computer agent in the collaborative STEM problem-
solving scenarios. Further issues with limitations that may have arisen due to the kind
of information and its organization used for the items by the agent in responding to the
student are also examined in the Discussion.
In addition to the content validity that was reported by the panel of expert reviewers,
the criterion-related validity was also examined through Pearson intercorrelations as
explained in the methods section on Evaluation. The use of this kind of evaluation
criterion is difficult to find in publications for the past few years. The only system of
a similar kind was developed by Kuo (2014). The core ideas of Kuo's system are
314 J. Comput. Educ. (2015) 2(3):301–322
Note CPS column entries are as follows: 1 represents ''Establishing and maintaining shared under-
standing,'' 2 represents ''Taking appropriate action to solve the problem,'' 3 represents ''Establishing and
maintaining team organization.''
similar to this study, but the difference is that Kuo's system is much simpler and
students can finish one test in a few steps (e.g., five steps). The eight assessment
modules used in our study were developed by four small groups (including one
professor, one graduate student, and one information technology engineer); thus, a
question can be raised about how consistent the eight modules were in yielding
similar evaluation results. Therefore, our rationale was to examine how consistently
the eight modules yielded similar results for overall and each of three CPS skills. If
the intercorrelations of the results for each of the eight modules were large within
any one CPS skill, this would support a conclusion that they were consistently
producing similar results. Accordingly, the results of the intercorrelations across the
eight task modules in relation to overall CPS performance are presented in Table 4.
The correlation values of the students' collaborative problem-solving skills, across
eight different problem scenarios, all reached a statistically significant threshold
J. Comput. Educ. (2015) 2(3):301–322 315
Table 4 Correlations of students' overall CPS performance across eight different problem scenarios
Tasks 1 2 3 4 5 6 7 8
* p \ .05, ** p \ .01
(p B 0.05); although the correlation values ranged considerably from 0.33 to 0.60.
That is, the eight tasks yielded sufficiently similar results for the assessment of the
students' CPS performance to be considered adequately consistent.
316 J. Comput. Educ. (2015) 2(3):301–322
Table 5 Correlations of ''Establishing and maintaining shared understandings'' across eight different
problem scenarios
Tasks 1 2 3 4 5 6 7 8
* p \ .05, ** p \ .01
Table 6 Correlations of students' ''Taking appropriate action to solve the problem'' across eight dif-
ferent problem scenarios
Tasks 1 2 3 4 5 6 7 8
* p \ .05, ** p \ .01
phone;'' (2) for task 1 ''Shelves'' and task 7 ''Construct a house,'' and (3) task 2
''Microwave oven and task 7 ''construct a house,'' we note that all values are less
than, or equal, to 0.2. Among the four tasks analyzed in these three comparisons the
number of computer agents varied. There are two computer agents in the tasks ''Buy
a cell phone'' and ''Construct a house.'' In these two module tasks, the two
computer agents will offer different choices to the students, and some of the choices
are designed to be wrong choices as explained in the foregoing methods section on
System framework. So the students may face more difficulties in taking appropriate
action to solve the problems in these two task modules, and this could be one of the
possible reasons for the low correlations of the outcomes for these two tasks.
J. Comput. Educ. (2015) 2(3):301–322 317
Table 7 Correlations of students' ''Establishing and maintaining team organization'' across eight dif-
ferent problem task scenarios
Tasks 1 2 3 4 5 6 7 8
* p \ .05, ** p \ .01
problem scenarios all reached a significant level, but the shelves and construct a
house are an exception. That is, there are six tasks at least that can assess students'
establishment and maintenance of team organization effectively. However, some
correlations were low, and these findings should therefore be taken cautiously and
re-checked by different studies and from different angles. If we check the
correlations of ''shelves and construct a house,'' ''shelves and travel in Kaohsiung,''
''shelves and buy a cell phone,'' ''defuse a bomb and construct a house,'' ''construct
a house and design tidy desk,'' we note that the values are below 0.2. The possible
reason for the low correlations with the shelves task in assessing students'
establishing and maintaining team organization is that we had a different design in
the shelves task. Students had to communicate with the computer agent by inputting
what they wanted to say using the keyboard, and the computer agent searched the
keyed-in input by accessing the database and responded to the students. In addition,
students also could indicate their response by selecting from among a set of choices.
According to the analysis of students' online record, some students did not want to
communicate with the computer agent by keying in their responses and preferred to
select the choice. Therefore, if the task is largely designed to be most effective using
the keyboard communication mode, but students preferentially elect to use the
simpler choice mode, this may be a source of differential performance in
establishing and maintaining team organization, hence contributing to lower
intercorrelations across tasks.
Research Question 2 What are some of the difficulties in developing an
assessment system for collaborative problem solving in STEM education?
This study was admittedly complex. Our goal was to provide some of the first
assessment-based evidence for a CPS web-based system that was theoretically based
(OECD matrix), that included computer agents (rather than human-to-human agents
during CPS), and incorporated problem tasks that were situated in every day kinds
of issues that junior high school students may encounter. Weaving all three of these
components into a novel system of this kind, that also used broad STEM content,
was challenging; especially given the lack of a similar ambitious study in the
318 J. Comput. Educ. (2015) 2(3):301–322
literature to help guide us. One of the issues concerns the challenge of creating a
computer agent system that is sufficiently responsive to the varied student inputs to
be reasonably natural and sufficiently contingent to encourage and reinforce student
engagement in a way that leads to consistent gains across the eight different task
modules. In this exploratory study, a team of experts designed a limited number of
potential items that the ''agent'' could retrieve in responding to student input.
Consequently, some lack of coherence is likely to occur; and it is difficult in every
case to develop an algorithm that would provide appropriate decision making by the
''agent'' to respond in both a logical and human-based natural way. More
sophisticated agent systems are clearly needed, and this study offers some insights
on the prospects for success as well as the challenges.
Another challenge emerged for the skill of ''Establishing and maintaining team
organization.'' This also was likely influenced partially by the lack of a more ''human-
like'' response of the computer agent as well as the limited number of options
programmed in the agent submodule in the server system (Fig. 3). Unless the student
has a sense that the collaborator is responding in a contingent and logical way, it is
difficult to see how a truly coordinated collaborative relationship can develop.
According to the data and analysis presented above, the assessment system developed
in this study consisted of the items with acceptable difficulty and satisfactory
discrimination, and the content validity and criterion-related validity were also judged
to be good. Overall, the assessment system for collaborative problem solving in
STEM education presented by this study was effective in assessing students'
collaborative problem-solving skills. Although the majority of intercorrelations used
as evidence of consistency of outcomes across the eight tasks within a given CPS skill
(Tables 4, 5, 6, and 7) on the whole were highly statistically significant and also in
many cases of sufficiently high correlation values to be considered strong, there were
a few that were low. These were highlighted in the Findings section and the reasons
for these discrepant findings remain somewhat enigmatic as to why the consistencies
were so low. Among these, the pairs of items that had low correlations: task 2
''Microwave oven'' and task 6 ''Buy a cell phone;'' task 1 ''Shelves'' and task 7
''Construct a house,'' and task 2 ''Microwave oven and task 7 ''construct a house,''
there are no immediate clues as to why these tasks of different demands were so
discrepant. These findings may offer an opportunity for further research to determine
what task attributes in relation to CPS skills are most likely to be challenging for
students at different grade levels and prior levels of knowledge.
As explained in the results for Research question 2, one skill that requires further
attention, ''Establishing and maintaining team organization,'' is especially chal-
lenging with respect to the design of a naturalistic computer agent. It proved hard to
design a problem scenario to assess this skill, mostly because it is difficult to always
design a response format for the computer agents that is sufficiently ''human-like;''
rather, they act according to a more limited set of item bank options developed by
our research team members. The complexities that emerge when the mode of
J. Comput. Educ. (2015) 2(3):301–322 319
interaction of the learner with the agent varies (such as keying in versus choosing
options) may also contribute to differences in performance assessment outcomes
across varied tasks. In Ding's (2009) study, three different collaboration patterns in
terms of joint and individual knowledge elaboration were shown to be effective, but
only one collaboration pattern was employed in the assessment system for
collaborative problem solving, because the students had to follow the script provided
in that study. In general, the limitations of a script-based computer agent are a major
problem for Ding's study, and likely for others given our current state of development
in the field. It was also a major issue in our study, when in addition to the complexities
of the OECD framework, we also included natural every day problems to assess
students' performance of collaborative problem-solving skills in STEM education.
That is, if students have different collaboration patterns (Ding 2009), and the
computer agent contributes yet another variation on the complexity of the patterns,
then the students may not achieve the level of sophistication that they may achieve in
a more natural human-to-human-based experience. Besides, if the task is designed as
a keyboard response mode, where the text may be complex and not easily parsed by
the software, it may be hard for the system to capture the students' real thoughts; this
may be especially difficult due to the limitation of our techniques in Chinese latent
semantic analysis. If we want to improve the system and reduce these mistakes, it
could be possible to offer the students multiple choices in responding, rather than a
single option provided by the computer agent (e.g., Kuo 2014), but it is difficult in this
case to prevent the students from merely choosing an answer instead of keying in an
answer that is more clearly what they want to say, in addition to the problem of less
consistency across different tasks, as discussed above.
In addition, there is another important difficulty that needs to be addressed. In
this study, we tried to assess 12 collaborative problem-solving skills in each
problem scenario; this large number of assessments became a drawback due to the
complexity of the problem scenarios. In some cases, there were just one or two
questions used to assess a given skill (see Table 1) and this may also be the reason
why the average difficulty of the eight assessment modules was deemed only
''acceptable'' according to Kuo's (1996) criteria. If in future work, we include fewer
of the 12 skills in one problem scenario to try to obtain a more robust item analysis
of the CPS responses, we face another limitation, i.e., it may be difficult to get a
comprehensive sense of students' real collaborative problem-solving skills.
Moreover, this study utilized a criterion-related validity method of examining the
intercorrelations of the results for each of the eight tasks in relation to overall and
each of three CPS dimensions as a way of gaining evidence of consistency and a form
of validity for the eight modules in the assessment CPS system. Even if a correlation
value reached the significant level (p B 0.05), we found in some instances that the
amount of variance accounted for was low; thus there is still a limitation that one or
more of the eight assessment modules could have by chance missed the core objective
of the collaborative problem-solving task that we intended to achieve.
To sum up, this study proposed a novel, and somewhat complex, assessment
system for collaborative problem solving in STEM education that can be used to
assess junior high students' collaborative problem-solving skills. Although we used
a computer agent, it is likely that with appropriate online data capture systems, it
320 J. Comput. Educ. (2015) 2(3):301–322
Acknowledgments This research was funded by the Ministry of Science and Technology of the
Republic of China under Contract numbers NSC 102-2511-S-003-059-MY2. The findings and
recommendations contained in this article of those of the authors and do not necessarily reflect those
of the Ministry of Science and Technology. We are extremely grateful to Professor O. Roger Anderson
and Ying-Tien Wu for their mentoring efforts, the reviewers for their helpful comments, and the teachers
and students who participated in this study.
Alvarez, C., Salavati, S., Nussbaum, M., & Milrad, M. (2013). Collboard: Fostering new media literacies
in the classroom through collaborative problem solving supported by digital pens and interactive
whiteboards. Computers & Education, 63(1), 368–379.
Azevedo, R., Moos, D., Johnson, A., & Chauncey, A. (2010). Measuring cognitive and metacognitive
regulatory processes used during hypermedia learning: Issues and challenges. Educational
Psychologist, 45, 210–223.
Barak, M., & Mesika, P. (2007). Teaching methods for inventive problem-solving in junior high school.
Thinking Skills and Creativity, 2(1), 19–29.
Biswas, G., Jeong, H., Kinnebrew, J., Sulcer, B., & Roscoe, R. (2010). Measuring self-regulated learning
skills through social interactions in a teachable agent environment. Research and Practice in
Technology-Enhanced Learning, 5(2), 123–152.
Blackwell, D., & Henkin, L. (1989). Mathematics: Report of the Project 2061 Phase I mathematics panel.
Washington, D. C.: American Association for the Advancement of Science.
Bybee, R. W. (2013). The case for STEM education: Challenge and opportunities. Arlington, CA:
National Science Teachers Association Press.
Cai, Z., Graesser, A. C., Forsyth, C., Burkett, C., Millis, K., Wallace, P., et al. (2011). Trialog in ARIES:
User input assessment in an intelligent tutoring system. In W. Chen & S. Li (Eds.), Proceedings of
the 3rd IEEE international conference on intelligent computing and intelligent systems (pp.
429–433). Guangzhou: IEEE Press.
Darling-Hammond, L. (2011). Policy frameworks for new assessments. In P. Griffin, B. McGaw, & E.
Care (Eds.), Assessment and teaching 21st century skills. Heidelberg: Springer.
Daugherty, M., & Wicklein, R. (1993). Mathematics, science, and technology teachers' perceptions of
technology education. Journal of Technology Education, 4(2), 28–43.
Davis, A., Fidler, D., & Gorbis, M. (2011). Future work skills 2020. Palo Alto, CA: Institute for the
Dillenbourg, P., & Traum, D. (2006). Sharing solutions: Persistence and grounding in multi-modal
collaborative problem solving. The Journal of the Learning Sciences, 15, 121–151.
Ding, N. (2009). Visualizing the sequential process of knowledge elaboration in computer-supported
collaborative problem solving. Computers & Education, 52, 509–519.
Dixon, R. A., & Brown, R. A. (2012). Transfer of learning: Connecting concepts during problem solving.
Journal of Technology Education, 24(1), 2–17.
Ebel, R. L., & Frisbie, D. A. (1986). Essentials of education measurement. Englewood Cliffs, NJ:
Prentice Hall.
Franklin, S., & Graesser, A. C. (1996). Is it an agent or just a program? A taxonomy for autonomous
agents. In Proceedings of the third international workshop on agent theories, architectures, and
languages. Springer, Berlin.
J. Comput. Educ. (2015) 2(3):301–322 321
Greene, R. W. (2011). Collaborative problem solving can transform school discipline. Phi Delta Kappan,
93(2), 25–28.
Harasim, L. (1999). A framework for online learning: The virtual-U. Computer, 32(9), 44–49.
Hong, J. C., Hwang, M. Y., & Tai, K. H. (2013). Applying the BaGua to revitalize the creative problem
solving process during a goal oriented contest. Thinking Skills and Creativity, 9, 123–128.
Isenhour, P. L., Carroll, J. M., Neale, D. C., Rosson, M. B., & Dunlap, D. R. (2000). The virtual school:
An integrated collaborative environment for the classroom. Educational Technology and Society,
3(3), 74–86.
Johnson, J. R. (1989). Technology: Report of the project 2061 phase I technology panel. Washington, D.
C.: American Association for the Advancement of Science.
Johnson, S. D., Dixon, R., Daugherty, J., & Lawanto, O. (2011). General versus specific intellectual
competencies: The question of learning transfer. In M. Barak & M. Hacker (Eds.), Fostering human
development through engineering and technology education (pp. 55–74). Rotterdam: Sense
Johnson, L. W., & Valente, A. (2008). Tactical language and culture training systems: Using artificial
intelligence to teach foreign languages and cultures. In M. Goker & K. Haigh (Eds.), Proceedings of
the twentieth conference on innovative application of artificial intelligence (pp. 1632–1639). Menlo
Park, CA: AAAI Press.
Kirschner, F., Paas, F., Kirschner, P. A., & Janssen, J. (2011). Differential effects of problem-solving
demands on individual and collaborative learning outcomes. Learning and Instruction, 21(4),
Kuo, S.-Y. (1996). Psychological and educational testing. Taipei: Jing-Hua Bookstore.
Kuo, B.-C. (2014). Teachers' collaborative problem solving teaching competency project. Retrieved on
March 7, 2015, from https://sites.google.com/site/cpswebsite2014/
Kuo, F.-R., Hwang, G.-J., Chen, S.-C., & Chen, S.-Y. (2012). A cognitive apprenticeship approach to
facilitating web-based collaborative problem solving. Educational Technology & Society, 15(4),
Kuo, C.-Y., & Wu, H.-K. (2013). Toward an integrated model for designing assessment systems: An
analysis of the current status of computer-based assessments in science. Computers & Education,
68, 388–403.
Looi, C.-K., Chen, W., & Ng, F.-K. (2010). Collaborative activities enabled by GroupScribbles (GS): an
exploratory study of learning effectiveness. Computers & Education, 54(1), 14–26.
Martin-Kniep, G., Feige, D., & Soodak, L. (1995). Curriculum integration: An expanded view of an
abused idea. Journal of Curriculum and Supervision, 10(3), 227–249.
McNamara, D. S., O'Reilly, T., Rowe, M., Boonthum, C., & Levinstein, I. B. (2007). iSTART: A web-
based tutor that teaches self-explanation and metacognitive reading strategies. In D. S. McNamara
(Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 397–421).
Mahwah, NJ: Erlbaum.
Nussbaum, M., Alvarez, C., McFarlane, A., Gomez, F., Claro, S., & Radovic, D. (2009). Technology as
small group face-to-face collaborative scaffolding. Computers & Education, 52(1), 147–153.
Organisation for Economic Co-operation and Development (2013). Draft collaborative problem solving
framework. Unpublished manuscript, OECD.
Raju, P. K., & Clayson, A. (2010). The future of STEM education: An analysis of two national reports.
Journal of STEM Education, 11(5&6), 25–28.
Roschelle, J., Rafanan, K., Estrella, G., Nussbaum, M., & Claro, S. (2010). From handheld collaborative
tool to effective classroom module: Embedding CSCL in a broader design framework. Computers &
Education, 55(3), 1018–1026.
Rosen, Y., & Tager, M. (2013). Computer-based assessment of collaborative problem-solving skills:
Human-to-agent versus human-to-human approach. Retrieved on June 2, 2014 from http://
Scifres, E. L., Gundersen, D. E., & Behara, R. S. (1998). An empirical investigation of electronic groups
in the classroom. Journal of Education for Business, 73, 247–250.
Sternberg, R. J. (2001). Teaching problem solving as a way of life. In A. L. Costa (Ed.), Developing
minds: A resource book for teaching thinking (3rd ed., pp. 451–454). Alexandria, VA: Association
for Supervision and Curriculum Development.
VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are
tutorial dialogues more effective than reading? Cognitive Science, 31, 3–62.
322 J. Comput. Educ. (2015) 2(3):301–322
Warwick, P., Mercer, N., Kershner, R., & Staarman, J. K. (2010). In the mind and in the technology: The
vicarious presence of the teacher in pupil's learning of science in collaborative group activity at the
interactive whiteboard. Computers & Education, 55(1), 350–362.
Wood, R., & Ashfield, J. (2007). The use of the interactive whiteboard for creative teaching and learning
in literacy and mathematics: A case study. British Journal of Educational Technology, 39(1), 84–96.
Kuen-Yi Lin is an Associate professor in technology education at National Taiwan Normal University,
Taiwan. His research interests deal largely with hands-on learning, STEM education, and technology
Kuang-Chao Yu is a Professor at National Taiwan Normal University, Taiwan. His research interests
deal largely with concept and knowledge construction related to technology and engineering education.
Hsien-Sheng Hsiao is a Professor and department head at National Taiwan Normal University, Taiwan.
His research interests deal largely with database management system, computer networks, e-learning, and
cloud computing.
Yih-Hsien Chu is an Associate Professor at National Taiwan Normal University, Taiwan. His research
interests deal largely with technology education, human resources education, human resources
development and manufacturing technology.
Yu-Shan Chang is a Professor at National Taiwan Normal University, Taiwan. His research interests
deal largely with technology education, technological creativity and multimedia design.
Yu-Hung Chien is an Associate Professor at National Taiwan Normal University, Taiwan. His research
interests deal largely with interior design, computer aided design and product design.
G Model
JSEE-562; No. of Pages 13
Article history: This article provides an account of the design, implementation, and evaluation of an ePortfolio approach
Received 15 May 2014 to faculty development and performance evaluation at a Canadian post-secondary vocational education
Received in revised form 18 February 2015 institute. The approach was piloted in two phases in 13 departments. Survey and interview data were
Accepted 26 March 2015
collected and analyzed to determine adoption, reception by faculty, and impact of the approach on
Available online xxx
faculty development. While adoption of the approach in the pilots was limited, participants who adopted
the approach reported collecting more and different feedback, developing increased awareness of areas
for improvement, and planning their professional learning more explicitly. Further studies are needed to
Faculty development
Professional development
determine what design elements of the portfolio optimally support professional development and
Professional learning performance evaluation.
Electronic portfolio ß 2015 Elsevier Ltd. All rights reserved.
Faculty evaluation
Vocational education
0191-491X/ß 2015 Elsevier Ltd. All rights reserved.
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
teaching practice if necessary; (3) formulate goals for future growth formal student feedback is a mandatory part of college or
based on their feedback and reflection; and (4) share their ePortfolio university workplaces, and many post-secondary teachers also
with their supervisor to collaboratively assess their performance seek informal feedback from their students. However, as Halonen
and determine goals for the next year. The paper also describes how and Ellenberg (2006) highlight, students might provide a one-sided
the ePortfolio approach was implemented in two consecutive pilots view of faculty performance or allow their feedback to be
and the survey and interview data collected in order to evaluate the influenced by bias. Feedback from additional sources such as
pilots. The research questions addressed in this paper are as follows: colleagues or faculty support staff might provide a more
comprehensive picture of professional performance (Millis,
(1) To what extent was the ePortfolio adopted in the participating 2006; Jimerson, 2014; Seijts et al., 1998; Wayman & Jimerson,
departments in the first and second pilot? 2014). Therefore, the taskforce established that feedback collection
(2) To what extent did faculty in the first and second pilot value the from multiple sources should be encouraged.
ePortfolio approach as a whole? Wayman and Jimerson (2014) also found that for feedback to be
(3) What was the impact of the ePortfolio approach on the relevant to teachers and students, it needs to be timely (see also
professional development of participating faculty members? Halonen & Ellenberg, 2006) so faculty members can discuss the
feedback with their students and adjust their teaching to
Literature review accommodate students' needs. Therefore, the third parameter
for the ePortfolio approach was that student feedback should be
We define faculty development in accordance with Taylor and sought 30–50% of the way through the semester with a quick
Rege-Colet (2009) who describe it as development across ''the full turnaround of the feedback.
spectrum of academic work'' (p. 143). Our view of faculty Educational development literature describes learning with and
development also aligns with Steinert (2000), who broadly defines from colleagues as a workplace learning strategy, using, for
it as any type of activity that is aimed at renewing or assisting instance, faculty learning communities (Cox & Richlin, 2004) or
faculty in their roles. Our definition is wider in scope than simply mentoring (Johnson, 2006). Faculty might also initiate informal
instructional development (Stes et al., 2010) and educational learning activities such as seeking feedback on course materials or
development (Taylor & Rege-Colet, 2009), which both refer to the discussing experiences during their break (Hoekstra et al., 2009;
development of the faculty member in the areas of teaching and Eraut, 2004). Based on these findings, the taskforce determined
student learning. Our definition of faculty development includes that faculty members would be encouraged to share their
professional learning in all aspects of the role of the faculty portfolios with colleagues.
member including, for example, teaching, research, and adminis-
trative tasks. Professional development plans and self-directed learning
Research on professional learning generally suggests that such
learning is deeply embedded in practice (Opfer & Pedder, 2011; Literature on workplace learning, teacher learning, and faculty
Schön, 1983) and informed by the way people within and outside development all stress that conditions in the workplace impact on
of the organization conduct and understand their work (Bound, the quality of professional learning at work (Amundsen & Wilson,
2011; Engeström, 2011; Lave & Wenger, 1991). Billett (2002) 2012; Clarke & Hollingsworth, 2002; Fuller & Unwin, 2011; Harris
proposes that learning at work occurs through participating in et al., 2001; Opfer & Pedder, 2011; Sambrook, 2005). It can thus be
work-related activities. These activities play a central role in our expected that enhancing faculty development requires faculty
definition of professional learning. Most conceptualizations of motivation to engage in learning activities; it also needs the
learning imply a relatively lasting change in behaviour or capacity workplace culture, structure, and systems to encourage and
for behaviour (Shuell, 1986). In the present study, professional nurture engagement in these activities (Fuller & Unwin, 2011).
learning is defined as engaging in activities that lead to improved One strategy to encourage more planning and regulation of
professional practice or the capacity to behave in improved ways. professional development adopted in several professions, includ-
Such learning may be intentional or unintentional and conscious or ing K-12 teaching, is the use of a personal development plan (PDP),
beyond the learner's awareness (Hoekstra, Beijaard, Brekelmans, & which aims to foster self-directed professional learning (Beausaert,
Korthagen, 2007; Eraut, 2004; Marsick & Watkins, 1990) and Segers, Gouarge, & Gijselaers, 2013). According to Roberson and
involve formal and/or informal learning activities (e.g., Hoekstra, Merriam (2005), self-directed learning refers to ''intentional and
Brekelmans, Beijaard, & Korthagen, 2009; Lohman & Woolf, 2001). self-planned learning, where the individual is responsible for and
in control of the learning'' (p. 270). Beausaert et al. (2013) reviewed
Reflective practice and feedback 54 studies and reported that attempts to implement PDPs have
varying degrees of success. In their own study, Beausaert et al.
Teacher educators, educational developers, and scholars who compared the number of learning activities reported by PDP users
study workplace learning all stress the importance of reflective with the number of activities of non-PDP users, concluding that,
practice as part of ongoing professional development (Hoekstra & retrospectively, PDPs encourage participation in learning activi-
Korthagen, 2011; Amundsen & Wilson, 2012; Brookfield, 1995; ties. Sharing their learning plans and assessment of their
Cross & Steadman, 1996; Van Woerkom & Croon, 2008; Zeichner & performance with the supervisor allows him or her to collaborate
Liu, 2010). While reflection can occur individually based on input with the faculty member and become an accountability partner for
from individual experience, reflection can be greatly enhanced the faculty member's own development (Higgerson, 2006; Murray,
through incorporating feedback from others into the individual's 1997). Based on this literature, the taskforce determined that the
own reflective process (Day, 1999; McGovern, 2006). The taskforce ePortfolio approach should encourage faculty members to set their
thus determined that the ePortfolio approach should include own learning goals and encourage department chairs to provide
student feedback as well as self-reflection on feedback (Hubball, sufficient modes of support and guidance for achieving these goals.
Pratt, & Collins, 2006; McGovern, 2006; Seijts, Taylor, & Latham,
1998). Performance evaluation and ePortfolios
Student feedback is a common component of faculty assess-
ment in universities (e.g., Pallett, 2006) and two-year colleges (e.g., The taskforce intended to design a process that would not only
Cross & Steadman, 1996). In many post-secondary institutions, foster faculty development but also support the assessment of
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
faculty performance in collaboration with the department chair. diploma programs, degree programs, and professional certificates
Peter Seldin's book ''Evaluating Faculty Performance'' (Seldin, in fields including trades, building construction and design,
2006) provides a strong case for the use of portfolios to assist in this engineering and applied sciences, health and safety, IT and
process (see also Freeman, Harkness, & Urbackzewski, 2009). electronics, business and administration, and hospitality and
Portfolios can include a collection of an individual's accomplish- culinary arts. The institute has approximately 1100 faculty and
ments, plans, reflections, observations, and work samples (Gran- 80,000 full-time and part-time students.
berg, 2010). One advantage of ePortfolios as opposed to the Historically, the institute's faculty members were required to
traditional format is they can also incorporate digital files such as create a performance management plan in which they outlined
simulations and videos (Butler, 2006). Smith and Tillema (2003) professional goals for the next year and annually discussed with
identify four types of portfolios including the dossier portfolio (a their supervisors the extent to which they achieved their goals.
record of achievements and work samples for promotional There is no promotion and tenure process at the institute; in other
purposes), the training portfolio (a required collection of efforts words, faculty members are directly hired into salaried positions
accumulated during a program), the reflective portfolio (a personal and remain faculty until retirement or job change. Professional
array of work providing evidence of growth and accomplishments development at the institute was most often conceptualized as
for promotion or admission), and the personal development portfolio attending conferences, courses, and workshops either on teaching
(a reflective account of professional growth over time). Portfolios or one's trade/discipline. The institute had a process for collecting
can be used as both formative assessment to support professional student feedback once per semester, but this feedback was not
learning and summative assessment for promotion or certification necessarily part of the performance management review cycle.
(Smith & Tillema, 2003). When the focus is on formative assessment,
portfolios can be used to motivate faculty to reflect on priorities, ePortfolio design
accomplishments, successes, and challenges and plan their learning
accordingly (Devanas, 2006). Portfolios can thus easily integrate In 2009, a taskforce was formed to create a comprehensive
aspects of the personal development plan, including activities such approach to faculty development and performance evaluation.
as goal setting and collecting feedback specific to those learning Previously, instructors who scored low on student feedback
goals. The 2013 study by Beauseart et al. suggests that including such surveys had been sent to the faculty support office for coaching,
activities in a portfolio approach would likely contribute to leading to the impression that faculty development was remedial
enhanced participation in learning activities. (see also Cross & Steadman, 1996) rather than a necessity for all
While portfolios are widely used in teacher education to faculty to stay current and up to date. In addition, there was a need
support the development of student teachers (e.g., Granberg, 2010) to review and update the existing student feedback questionnaires
and used increasingly with post-secondary students of various and processes. The approach the taskforce was asked to design was
disciplines including nursing (e.g., Jasper & Fulton, 2005), medicine not intended to replace existing human resources processes
(e.g., Carraccio & Englander, 2004) and business (e.g., Flanigan, regarding personnel decisions; rather, it was intended to encour-
2012), the use of portfolios in support of faculty development and age faculty to collect comprehensive feedback and share it with
evaluation has received much less attention (Butler, 2006; their chair so as to collaboratively and continually work toward
McColgan & Blackwood, 2009). With respect to universities, improving their practice.
ePortfolios have been found to support and enhance professional Based on literature on leading by engagement (Axelrod, 2010),
development of faculty within the university promotion and the taskforce held a series of town hall meetings and focus groups
tenure process (Blair, 2001; Freeman et al., 2009). However, little is with faculty and other staff to engage them in the design and
known about the effect or use of portfolios in general and implementation stage. During these sessions, members of the
ePortfolios specifically in support of faculty development. Con- academic staff union indicated that the role of faculty members
ceptually, ePortfolios have the potential to enhance and promote had significantly expanded beyond teaching and that faculty
self-directed learning (Goliath, 2009) and self-regulated learning development and performance evaluation should cover this
(Lamont, 2007). Based on this literature, the taskforce determined expanding array of faculty responsibilities; these views aligned
that the best way to satisfy the parameters was through the closely with the findings of the literature review (e.g., O'Meara &
creation and sharing of professional ePortfolios. While not Braskamp, 2005). The taskforce then used focus groups with
intended for purposes of tenure and promotion, the professional institutional leadership, faculty members, and executive members
ePortfolio approach designed by the taskforce would cut across of the academic staff union to collaboratively identify eight aspects
multiple types of portfolios as identified by Smith and Tillema of the role of a faculty member: teaching, formal leadership, formal
(2003). Faculty would be invited to include a list of professional administrative duties, professionalism, formal curriculum devel-
development sessions attended (as in a training portfolio), opment, applied research, scholarship and professional develop-
evidence of feedback collected, and documented reflections on ment, and corporate citizenship. See page 4 of Online supplement
their feedback (as in a reflective portfolio). The compilation of this 3 for a description of each of these aspects. During these focus
evidence would serve to inform reflections on personal profes- groups, faculty members also indicated that feedback collected
sional growth over time (as in a personal development portfolio). only from students might not be a reliable method to assess their
performance. Taskforce members agreed with this point of view,
Methods based on the literature described earlier in this paper. Stakeholders
and taskforce members collaboratively decided to identify the
The ePortfolio approach was implemented in two pilots. following sources of evidence of performance: self, students, peers,
Following each pilot, both survey and interview data were and ''others,'' the latter including externals and faculty support
collected to determine adoption, faculty perception, and faculty personnel. In further faculty engagement sessions, a matrix was
perception of the impact of the ePortfolio approach. created with the eight aspects as rows and the four sources as
columns. Participants in the sessions collaboratively determined
Study context which sources would be able to provide feedback on which aspects
of the faculty role.
The site of study is a Western Canadian post-secondary Faculty support staff then proceeded with developing feedback
vocational institute that offers apprenticeship training, two-year tools for those cells in the matrix that were identified as good
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
sources for feedback on certain roles. These tools were presented mance assessment and coaching. Specific attention was paid to the
together as a feedback toolkit and made available through the idea that performance evaluation should happen on an ongoing
institute's intranet. The toolkit included informal student feedback basis with the purpose of supporting professional growth. It was
tools, a standardized student survey, self-reflection resources, also highlighted that human resource processes around disciplin-
peer-review tools including classroom observation templates, and ary measures were outside the scope of the ePortfolio approach.
a standardized leadership survey. See Online supplement 1 for an A follow-up session in a computer lab introduced participants
overview of the feedback tools in the toolkit. Thus, based on the to the online ePortfolio software Mahara. A mandatory part of both
literature and faculty engagement sessions, the taskforce designed pilots included the automatic collection and reporting of student
the ePortfolio approach as a faculty development and performance feedback through a newly developed online electronic student
evaluation initiative that encouraged faculty to (1) collect feedback survey. In addition, supports were offered through a website,
from multiple sources on multiple aspects of their role; (2) compile online feedback toolkit (see Online supplement 1), reminder
this feedback in their ePortfolio, reflect on it, and make changes to emails, and additional workshops in which participants could
their teaching practice, if necessary; (3) formulate goals for future practice the use of the ePortfolio tool. These additional workshops
growth based on their feedback and reflection; and (4) share their were offered to departments who requested extra help.
ePortfolio with their supervisor to collaboratively assess their Based on the findings from the first pilot, a number of changes
performance and determine goals for the next year. were made to the implementation of the second pilot. The student
and leadership surveys were refined, more diverse feedback tools
The ePortfolio software were developed, and it was decided to provide a customized
orientation and workshop to each participating department
Mahara (mahara.org) is an open-source software program separately. Between the first and second pilot, the institute
chosen as the web-based ePortfolio software tool because it is initiated a drastic academic change that involved major curriculum
intuitive in use and works within any browser. Mahara permits development for all credit programs, a new credit system, the
password protection and allows the faculty member to customize introduction of common courses, and a profound revision of the
different versions of the portfolio for different audiences, such as academic administration software used by the institute. This
peer groups or supervisors. The faculty member can have full change caused a great increase in workload of faculty.
control over who sees which page(s) and can set limits for the
duration intended audiences can access those pages. Mahara Instruments
version 1.4.6 was slightly customized for use within the institute
by creating a login page with the institute's colours and logo. The evaluation focused on the impact of the ePortfolio approach
and its implementation, including details regarding the orientation
Pilot implementation sessions held, the new student evaluation survey, the new survey
used by chairs to collect feedback on their performance, student
The ePortfolio approach was piloted in two phases. In the first, response rates, and the feedback tools. This paper focuses on those
the deans of all five faculties in the institute were requested to seek questions pertaining to adoption of the approach, how faculty
department chairs willing to try out the new approach in (one of experienced the ePortfolio as a whole, as well as the impact of the
their1) smaller programs. In the second, the deans were requested approach on their professional learning activities. Data regarding
to seek department chairs of two to three larger programs. The first the orientation sessions and the composition and usefulness of
pilot was carried out from September–December 2011 with 32 individual feedback tools are not included in this paper.
participants (including 5 chairs and 27 faculty members) from five In collaboration with the taskforce, the researchers created a
programs in trades, health sciences, business, and media and survey and interview guide aimed at collecting feedback on the
design. The second pilot was carried out from September– orientation to the ePortfolio, the feedback tools developed, and the
December 2012 with 75 participants from 12 programs in ePortfolio approach as a whole. An interview guide was developed
9 departments2 including programs in trades, engineering, applied to collect more detailed feedback on these elements and obtain a
sciences, and health. One program participated in both pilots. The more in-depth understanding of the impact of the ePortfolio
total number of participating departments was 13: four in the first approach on faculty learning. Based on the responses from the first
pilot, eight in the second, and one that participated in both. pilot, we also evaluated our survey questions. The survey used for
In the first pilot, all participants, including the department the first pilot had many open-ended questions. The answers to
chairs, attended the same orientation to the approach during two those questions were not always easy to interpret, and not all
half-days in September of interactive introductory sessions participants answered. For instance, one question read as follows:
organized by the taskforce. The approach was just as new to the How do you feel about the whole experience of collecting feedback,
department chairs as to the instructors. During the orientation, an reflecting on it, and documenting it? One participant answered
open conversation between a faculty member and a chair with ''Informative.'' Another wrote: ''We are here because we are
regarding feedback of the faculty member was modeled. In the good at our trade. This generally means we are not all great
second pilot, departments received an orientation within their writers.'' The first answer is so general that it's hard to interpret
own program. In the orientation of both pilots, participants were what exactly this participant found informative; the second
introduced to the purpose of the ePortfolio approach: to support response most likely points to a reaction to the encouragement
faculty development and performance evaluation. They were to include written reflections. This question was amended to
presented with an overview of the philosophy behind the include more closed-ended questions, such as (1) I found the
approach, including the expanding role of faculty, eight aspects ePortfolio approach helpful for my professional development. (2)
of the faculty role, the reasons why feedback should be collected The ePortfolio approach is a useful way to set learning goals. While
from multiple sources, the value of reflective practice, and the role these amended questions yielded better data for evaluating the
the department chair could play in supporting formative perfor- second pilot, these changes prevented us from comparing the pilot
1 survey data with the pilot 2 survey data. The second pilot survey
Some department chairs oversee more than one program.
also included more closed-ended questions pertaining to concerns
One participating department included instructors who teach in three separate the taskforce had heard about anecdotally, such as a lack of trust in
programs. the security of the Online data. The interview questions were the
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
same as for the first pilot and are listed in Online supplement 2. The perceived value of ePortfolio approach, (3) impact of adoption of
survey questions are presented in Online supplement 3. ePortfolio approach. Interviews from the first pilot were coded by
two researchers, in most cases reaching a minimum of 85%
Data collection agreement, and discrepancies were discussed. Interviews from the
second pilot were analyzed by one researcher, using the same
The first pilot was carried out from September–December 2011 themes. For the analysis of interview data from both pilots, a
with 32 participants from five programs, including programs in matrix was created with each column representing a theme and
trades, health sciences, business, and media and design. Before the each row, a participant. Interview excerpts were copied into the
start of data collection, research ethics approval was obtained from cells or, in the case of very large excerpts, summarized in the cells.
the institute's research ethics board. This board verifies that This tool allowed the researchers to compare interview excerpts
proposed recruitment and data collection procedures are in relating to one theme across the participants. Based on these
accordance with In Canadian Tri-Council Framework for Research comparisons, we created categories across participants as well as a
involving Human Subjects. Data collection took place in early count of how many participants would fall into this category. For
2012. To invite faculty to the survey and to select interview instance, the theme adoption of ePortfolio approach included three
participants, the chairs of the participating programs were categories: (1) no portfolio created, (2) portfolio page with some
approached to provide a list of their participating faculty members. documents uploaded, and (3) more elaborate ePortfolio. See Online
From this list, 1–2 faculty members per participating program supplement 4 for a list of coding categories and the number of
were selected to be interviewed using a random number generator. interviewees categories apply to. Because only two interviewees in
In programs with more than six faculty members, two participants the second pilot created more than one page in their portfolio, we
were selected to be interviewed. In the first pilot, we interviewed limited the analysis of the second pilot interviews to the first two
seven randomly selected participants: one from each of the three research questions on adoption and perception of value of the
smaller programs (<6 faculty members) and two from the two ePortfolio approach. In the next section, we answer the research
larger programs (6 members). We also interviewed each of the questions by presenting survey and interview data. We describe
five department chairs. We thus had 12 interview participants in the categories developed in our interview analysis, how many
pilot 1: seven faculty members and five department chairs. Of all participants this category applied to, and examples of interview
32 pilot 1 participants who were invited to anonymously respond excerpts.
to an online survey, 26 responded. The decision to collect
anonymous feedback was made in an attempt to increase the Findings
chances of receiving honest feedback. Because a number of
participating programs were small (3 faculty members) the survey This section presents the findings in the order of our research
did not ask participants which program they were part of since that questions: (1) To what extent was the ePortfolio adopted in the
information might identify them. programs participating in the first and second pilot? (2) To what
The second pilot was carried out from September–December extent did faculty in the first and second pilot value the ePortfolio
2012 with 75 participants from 12 programs in 9 departments3 approach as a whole? (3) What was the impact of the ePortfolio
including programs in trades, engineering, applied sciences, and approach on the professional development of participating faculty
health. One program participated in both pilots. Data collection members?
took place in early 2013. We used the same sampling methods as in
pilot 1 to select interviewees. Three of the selected faculty The extent of ePortfolio adoption in pilot 1
members declined to participate. Research ethics guidelines
stipulate that those approached to participate in the study may Using survey and interview data, the extent of the adoption of
''opt out'' of the data collection without having to provide a reason. the ePortfolio approach was assessed in terms of whether or not
We were therefore unable to establish their reasons for declining. the participants reported collecting feedback and/or compiling
Using a random number generator, alternative participants were that feedback into their ePortfolio, reflecting on it, sharing it with
selected from the remaining faculty members in those depart- their supervisor, and making changes if necessary. Most of the first
ments. Unfortunately, in one small program, the freedom to opt out pilot participants reported collecting some type of feedback.
meant that we could not interview anyone. As a result, 14 Table 1 provides the frequency overviews related to two survey
participants from 11 programs from the second pilot were questions: (1) Is this aspect4 part of your job? (2) Who did you
interviewed. Time constraints prevented us from interviewing collect feedback from regarding this aspect of your job? These
the department chairs. All 75 pilot 2 participants were invited to results show that the majority of survey respondents (22 out of 26)
fill out the online pilot 2 survey. A reminder was sent two weeks were teaching during the pilot; cross-tabulation confirms that 20
after the survey was distributed, resulting in a total of of these 22 respondents collected student feedback on their
42 responses. teaching. Examination of the raw data file indicates that one survey
respondent chose not to fill out any of the questions listed in
Data analysis Table 1. Similar cross-tabulations show that all respondents who
collected feedback on certain aspects of their job also indicated
The responses to closed survey questions were summarized that those aspects were in fact part of their job. Table 1
into frequency overviews. The responses to open survey questions demonstrates that quite a few respondents collected feedback
were categorized and summarized. This paper only reports the from peers on job aspects that applied to them. For instance, 13 out
results from the survey questions relating to the adoption of of 23 respondents to whom Scholarship/Professional Development
the ePortfolio approach and faculty members' perception of the applied sought peer feedback.
value and impact of the approach. Table 2 shows that the majority of participants (69%) collected
Interviews were transcribed verbatim. Interview excerpts from more and/or different types of feedback in the pilot than they did
the first pilot were organized according to the research questions before the pilot. Of the five comments to this question, three
of this paper using the themes (1) adoption of ePortfolio, (2)
These aspects were based on the eight aspects of the role of a faculty member as
One department offered three separate programs. identified in the stakeholder consultation, which is described in the Method section.
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
Table 1
Aspects part of respondents' job and who they collected feedback from regarding that aspect.
Teaching 79% (22) 11% (3) 21% (6) 71% (20) 14% (4) 4% (1) 0% (0) 7% (2)
Professionalism 86% (24) 4% (1) 50% (14) 21% (6) 39% (11) 7% (2) 4% (1) 4% (1)
Corporate citizenship 68% (19) 18% (5) 14% (4) 4% (1) 11% (3) 4% (1) 29% (8) 14% (4)
Scholarship/professional development 82% (23) 7% (2) 46% (13) 0% (0) 18% (5) 4% (1) 29% (8) 0% (0)
Applied research 14% (4) 64% (18) 4% (1) 0% (0) 0% (0) 0% (0) 7% (2) 46% (13)
Formal curriculum development 64% (18) 21% (6) 29% (8) 7% (2) 25% (7) 0% (0) 18% (5) 11% (3)
Formal administrative duties 36% (10) 46% (13) 18% (5) 4% (1) 11% (3) 0% (0) 7% (2) 25% (7)
Formal leadership 36% (10) 46% (13) 29% (8) 7% (2) 14% (4) 0% (0) 4% (1) 32% (9)
Respondents were able to select more than one response regarding data source for each aspect of their job.
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
Table 3
Responses to the question: How much time did you spend on the following aspects of the ePortfolio approach?
Learning to use Mahara 20% (5) 72% (18) 4% (1) 4% (1) 0% (0)
Collecting and reflecting on feedback 16% (4) 48% (12) 28% (7) 8% (2) 0% (0)
Documenting your growth and development 35% (9) 42% (11) 8% (2) 15% (4) 0% (0)
(including writing reflections, uploading in Mahara, etc.)
participants found the approach helpful. The participant who your plans for the department.'' Four participants thought that the
indicated not understanding the ePortfolio approach did not ePortfolio approach was a good opportunity to reflect on
respond to the question regarding its helpfulness. performance for the purpose of identifying any gaps or areas for
The interviews provided more insight into the reasons for the future growth. Three participants felt the ePortfolio approach was
poor adoption of the approach in pilot 2. Fourteen participants helpful in making supervisors aware of the work their faculty
were interviewed; all reported placing a high value on receiving members are involved in. For example, Jamie expressed:
and learning from student feedback, yet only five of the 14 created
And that's where I do see the point where the supervisor might
an ePortfolio. Of the nine participants who did not build an
see that, Oh, wow. You've done this. Have you done this? Have
ePortfolio, two missed the orientation, and one stated lack of
you done this? You know they had forgotten, or they did not
interest due to imminent retirement. The other six indicated
know because they're busy managing their department.
limited time and did not see the creation of an ePortfolio as a
priority. Three interviewees put a few documents into Mahara, Three participants indicated it was nice to get good perfor-
such as an ePortfolio page and some teaching materials. The mance reaffirmed. Addy: ''It reassured me that I was doing some
remaining two interviewees built a more comprehensive ePortfo- things right.'' After receiving feedback using the department
lio, including some documentation of achievements and projects survey for chairs, chair MacKenzie expressed that the ePortfolio
they were working on. None of the interviewees included student process was a ''good opportunity for the leaders to ask for
feedback in their ePortfolio. These findings suggest that adoption assistance'' from their own supervisors.
of the approach in pilot 2 was less than during pilot 1. A sub-theme of perceived value of ePortfolio approach was the
view of Mahara, the online software used by the participants to
Faculty perception of value of the ePortfolio approach in pilot 1 create their ePortfolios. One participant did not access or use
Mahara at all and therefore did not express any views on the
In response to the question of whether participants found the software. Two participants felt that Mahara was difficult to learn.
ePortfolio approach helpful for their professional development Neither of these participants used Mahara to create an ePortfolio
(see Table 7), 47% found the approach helpful or very helpful; 46% because neither felt comfortable with the software. Addy explains:
found the approach not so helpful; 8% found the approach not
helpful at all. Back to Mahara, I guess I'm not a social media person. I don't use
Nineteen out of the 26 respondents provided an answer to the Facebook. I don't use MyPage or any of that, so I really wasn't
open-ended question: What did you find helpful and/or not helpful invested in it. If I was applying for a different job, I could see it,
about the ePortfolio approach? Seven indicated they found it right. I wasn't comfortable enough to use it for, like, a
helpful that the approach encouraged reflective practice. [performance review] presentation to my boss.
Responses in this category included ''making reflection and action Four participants did not see much added value in Mahara. Two
more deliberate and focused was helpful,'' and ''allows you to participants did not like the process of uploading paper
reflect on what you are doing.'' An additional three respondents documentation to Mahara's web site. Alex explains: ''I think it
indicated they appreciated how the ePortfolio allowed them to would be just much easier just to have a folder and have a file in
collate relevant documents where the supervisor could see them; your hard drive just to deposit stuff.'' Other participants felt that
it was ''helpful that everything could be in one place.'' Four of the Mahara was too much work to learn. For example, Jamie
19 comments indicated that participants found the approach expressed: ''It did more than I needed it to do, and it was one
unhelpful, redundant, or cumbersome though none provided a more piece of software that I need to learn.'' Despite these
rationale. The remaining five responses commented on the use of concerns, three of these participants created an ePortfolio using the
Mahara and the need for more training or practice. The interview software. Finally, five participants thought Mahara was easy to
data provided more in-depth insight into the faculty experiences. learn. For example, Tracy stated: ''I think the tool was easy.''
From the 12 participants interviewed, three did not see any
benefit or added value in the ePortfolio approach, either because Faculty perception of value of ePortfolio approach in pilot 2
they were not interested in learning due to imminent retirement or
because the approach was similar to the learning processes they The participants in the second pilot saw less value in the
usually engaged in. Nine participants saw added value in the ePortfolio approach, and they raised a number of concerns that
ePortfolio approach. Five of those nine thought the ePortfolio was a were absent in the first pilot. Table 8 shows that approximately a
good opportunity to store feedback, reflections, and achievements quarter of the participants found the ePortfolio approach helpful
in one place. Leslie liked the ePortfolio because ''It is good to have a for professional development, setting learning goals, assessing
place to put all that kind of stuff down, your goals and even just performance, and receiving coaching and support from their
supervisor. The majority of respondents were undecided or did not
Table 4 find the approach useful or helpful at all.
Responses to the question: Did you include your feedback, reflections, and evidence Ten participants took the opportunity to provide written
of accomplishments in Mahara? comments to the questions in Table 8. They indicated that the
Response Yes Most of it Some of it No ePortfolio was introduced at the wrong time (five comments) and
that they were uncomfortable with posting information online
Yes 8% (2) 19% (5) 46% (12) 27% (7)
(one comment). Three commented that the ePortfolio is not a good
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
Table 5
Time spent on ePortfolio approach in pilot 2.
How much time did you spend on the ePortfolio approach 27% (10) 22% (8) 35% (13) 5% (2) 11% (4)
as a whole, including setting learning goals, collecting
feedback, reflecting on feedback, and documenting these
in Mahara?
Table 6
Cross-tabulation of participants' perception of their understanding of the purpose and principles underlying the ePortfolio approach, and the question: ''Did you use Mahara
as an ePortfolio tool?''.
I'm sure I understand the purpose 24% (10) 24% (10) 48% (20) 11% (2) 21% (4) 16% (3) 16% (3) 32% (6) 5% (1)
and principles entirely
I'm not sure I entirely understand 29% (12) 21% (9) 50% (21) 10% (2) 5% (1) 35% (7) 15% (3) 20% (4) 15% (3)
the purpose and principles
I don't understand the purpose 0% (0) 2% (1) 2% (1) (0) (0) (0) (0) (0) (0)
and principles of the ePortfolio
Total 52% (22) 48% (20) 100% (42) 10% (4) 13% (5) 26% (10) 15% (6) 26% (10) 10% (4)
N/A = Not Applicable, SA = Strongly Agree, A = Agree, U = Undecided, D = Disagree, SD = Strongly Disagree.
use of time because only the format and not the content was expressed mostly concerns with the approach. Francis explains
discussed with the chair (one comment) or because they prefer to how s/he normally goes about improving practice and how an
use a different tool for professional development (two comments). ePortfolio would help in this process:
The remaining comment questioned whether the ePortfolio
I'm usually quite cognizant of getting feedback and reflecting.
approach would actually add to faculty accountability or not:
[. . .]. So I generally do it quite often, and I do it verbally quite a
''We can choose what our supervisor sees, so we can just pick the
bit. [. . .] Sometimes I have the students kind of just write on a
best survey [results].''
little sticky note what worked, what didn't. [. . .] I kind of run
Of the 14 interview participants, only one interviewee, Dallas,
some exercises by some colleagues before I try them in the class
expressed seeing a clear benefit to the approach. S/he believed in
to see if it would work or not, [. . .] And then I create lesson plans
the ePortfolio's abilities to create an overview of accomplishments
where I have little [. . .] At the bottom I just have a little area for
and support reflection:
comments[. . .] And the purpose is eventually to be able to [. . .]
I think, you know, a few years down the road it will be a nice kind of put that into some type of portfolio as part of the
timeline. Oh, I was working on this, [. . .] I've worked more into – reflection. Hasn't happened yet, yeah.
I've been helping with some research here and this here, you
While s/he values reflection, Francis did not create an ePortfolio
know. So I think it will be an excellent tool for me to reflect on
what I've been doing.
So I – like I said, the student feedback came in. I looked at it, I
Dallas wished for more time to work on the ePortfolio and was
addressed it personally, and I addressed it in the classroom.
eager to continue using the approach. The other four interview
[. . .]. I didn't do anything with that feedback with the ePortfolio,
participants who had created at least one ePortfolio page
unfortunately. I kind of had a bit of an – I guess a reservation
Table 7
Responses to the question about helpfulness of the ePortfolio approach.
To what extent did you find the ePortfolio approach helpful 12% (3) 35% (9) 46% (12) 8% (2)
for your own professional development?
Table 8
Responses to the question: Please indicate to what extent you agree with the following statements.
I like the ePortfolio approach 10% (4) 18% (7) 33% (13) 8% (3) 26% (10) 5% (2)
I found the ePortfolio approach helpful for my professional 10% (4) 13% (5) 26% (10) 15% (6) 26% (10) 10% (4)
The ePortfolio approach is a useful way to set learning goals 3% (1) 21% (8) 28% (11) 13% (5) 28% (11) 8% (3)
The ePortfolio approach is a useful way to assess performance 3% (1) 18% (7) 23% (9) 25% (10) 25% (10) 8% (3)
The ePortfolio is a useful way to receive coaching and support 3% (1) 20% (8) 30% (12) 15% (6) 25% (10) 8% (3)
from my supervisor
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
about the ePortfolio about the fact that it's online. I'm not really And the feedback I got was very positive. Nothing that surprised
comfortable with putting information online. me because I talked to my students all the time, and I know
where they're at. So I didn't personally find that feedback to be
The interviewees who did not create an ePortfolio were not
particularly of value to me.
asked what they found helpful about the approach. One of the
interviewees volunteered his expectation that it the approach Of the 12 pilot 1 participants interviewed, two indicated they
would be useful to keep a record. did not adopt the approach, experienced no impact, and did not
A total of six of the interviewees reported concern over putting intend to use the approach. A second group of two participants
their information online. Robin, one of these six, also expressed reported that, among other things, the approach helped make their
discomfort with having to learn a new software tool. Quinn would chair aware of all the work they did. For instance, Jamie reported:
rather the information be stored on a server hosted at the institute:
And there's a reason why I'm putting all this stuff in. Because I'm
''And then the other thing is who's going to see my information in
pretty sure our supervisor [the department chair] has a pretty
the long-term? How secure is it? Will my students have – be able to
good idea of what we're doing, but sometimes he/she might not
hack into it or access it or whatever?'' Harper wondered whether
remember just all the work that all of us staff do.
the information might be accessed by upper management who
might use it for salary decisions. While the taskforce had not Six participants reported their participation in the ePortfolio
wanted to be prescriptive in what participants should include in approach has helped them become aware of areas of improvement
their ePortfolio, three interviewees expressed they were unsure to their practice. Chair MacKenzie reported, for instance, ''It helped
what an ePortfolio might look like and expressed a desire for me pinpoint things that I need to work on for sure, yeah.'' Addy,
direction. Kelly indicated: ''For sure, templates would help and, also a chair, described an improvement in practice as a result of
you know, without knowing the full details on ePortfolios, but feedback received:
templates and what's expected throughout the whole and what
So I may talk to one associate chair and not the other. So one
you expect the finished product to look like.'' Robin expressed
hand doesn't know what the other hand is doing. So with that in
questions about ways to provide evidence of interpersonal skills:
mind, then I try to be more inclusive now [that I received that
And is it a true evaluation of my skills, I guess, is the other thing. feedback] because I really wasn't that conscious of it.
I just kind of feel – I think I bring something unique to the
Three participants reported that the ePortfolio approach made
classroom. I think I'm really personable with my students. I care
them think more explicitly about actively planning their profes-
a lot about them. I personally talk to them a lot. I end up being a
sional development. For instance, Jean indicated: ''I think the one
counsellor most days. But, again, I don't know how that – how
thing it did for me was, it made me think, ''Do I document enough?''
those two mesh.
Leslie commented:
Interviewer: So how can you show what your strengths are if But, yeah, it does help me to outline my goals. Definitely it
they're not in an actual project that you can post? helped me to identify new ideas and kind of have a place to put
them down and things that I want to do with the program to,
Robin: Right. Exactly. you know – in terms of organization of the second year, clinical
experience, different ways that we can [keep in touch with
Perceived impact of ePortfolio approach in pilot 1 students]– because our students are so far away.
Finally, Jordan indicated:
After pilot 1, when asked whether they used the feedback they
collected per aspect to improve their performance, more than 75% Now I've got things that I'm thinking, I could do that to get
of respondents to whom this aspect applied answered affirmative- better feedback, or I could do this. So it helped me think about
ly, with the exception of corporate citizenship and applied research what I'm going to do in the future for my professional
(see Table 9). development, definitely. It also identifies what I could probably
Although respondents who replied ''no'' did not provide an use more training on. So it kind of helps me plan.
explanation, at least one participant, Jean, indicated the feedback
was all positive and therefore not used to improve work: Perceived impact of ePortfolio approach in pilot 2
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
Table 10
Responses to the question: Please indicate to what extent you agree with the following statements.
The feedback I received in the summary student 7% (2) 57% (16) 21% (6) 14% (4) 0% (0) 0% (0)
feedback report influenced the way I teach
The feedback collected through the chair survey 0% (0) 57% (4) 29% (2) 0% (0) 14% (1) 0% (0)
influenced the way I supervise
was to foster naturally occurring professional learning activities of portfolio was a key contextual factor affecting the acceptance and
participating faculty members. uptake of portfolios by professionals. Goliath (2009) studied
The ePortfolio approach was piloted in two rounds. The adoption of an ePortfolio by physicians. Similar to our findings, she
research questions addressed in this paper were: (1) To what described that individual factors such as comfort level with
extent was the ePortfolio adopted in the programs participating in technology influenced participants' adoption of the ePortfolio
the first and second pilot? (2) To what extent did faculty in the first technology. Another reason for limited adoption that was
and second pilot value the ePortfolio approach as a whole? (3) mentioned by many participants in the second pilot was a lack
What was the impact of the ePortfolio approach on the of time. During the second pilot, an institute-wide curriculum
professional development of participating faculty members? This redevelopment process was implemented. This has, at least in part,
section discusses our main findings, lessons learned, and implica- accounted for the fact that adoption was greater in the first pilot
tions for research. than in the second.
Aside from a lack of time or interest, the participants provided
Added value of feedback other reasons for why they chose not to build an ePortfolio. A
number of them found it cumbersome to learn the software,
In terms of the adoption of the ePortfolio approach, the majority whereas others expressed concerns about the privacy and security
of participants in both pilots indicated that they collected of Mahara, a lack of trust in the organization where online
feedback, reflected on that feedback, and used it to improve their information may be accessed and used for purposes other than
practice. They most likely would have collected some type of formative assessment, and a lack of clarity about what kind of
feedback if they had not been participating in the pilot; however, information could be included in an ePortfolio. These concerns
the results suggest that due to the pilot, they collected more and/or could have been alleviated with additional preparation and
different types of feedback than they did previously. The support sessions. The need for advance preparation is echoed by
participating department chairs reported finding value in gather- Blair (2001), who identified separate workshops on using teaching
ing feedback by surveying the faculty in their department portfolios for professional development and creating an electronic
regarding their responsibilities as a chair, a practice that previously portfolio as being key to the success of an ePortfolio as a faculty
had not been common throughout the institute. All participants development tool. Smith and Tillema (2003) also pointed to the
interviewed explained that they greatly valued learning from importance of clarity of expectations: ''The trust in the portfolio
feedback, and many described ways in which they collected and was strengthened when there were explicit criteria for the content
discussed feedback with students in order to improve the students' of the portfolio, how to go about it and of how the portfolio was
learning experience. This concurs with similar findings by Smith going to affect their professional status'' (p. 644). This finding also
and Tillema, who concluded: confirms the need for clear performance expectations (e.g.,
Higgerson, 2006).
Another aspect which attracted much attention in the face-to-
Despite the challenges with the tool, participants reported that
face interviews was the importance of feedback during the
creating an ePortfolio was an opportunity for them to:
portfolio collection process. In three out of the four settings
(dossier, training and reflective portfolios) external feedback is
(1) Identify issues and ask assistance from supervisor in dealing
seen as a positive and essential part of the process. (p. 646)
with these issues.
By encouraging faculty to collect more and different types of (2) Collect more and different types of feedback than before.
feedback, the ePortfolio approach did indeed foster a naturally (3) Store learning goals, achievements and other information
occurring professional learning process of faculty. relevant to their development in one place.
(4) Store feedback and accomplishments over time, which allows
Creating an ePortfolio one to look back and reflect on performance and also allows for
the formulation of goals for future improvement.
In both pilots, less than half of the participants who responded (5) Show the supervisor all the different tasks one is involved in.
to the survey actually documented their feedback and reflections
in the online ePortfolio tool Mahara, though the adoption of the The above benefits reported by our participants are consistent
ePortfolio approach was greater in the first pilot than in the second. with Granberg's (2010) findings, which established that student
Our participants' weariness of the portfolio software and the fact teachers expressed benefits of using ePortfolios as: portfolios for
that it exists online conflicts with Freeman et al. (2009) who report learning (opportunities 1 & 2 discussed above), portfolios as an
the majority of their study participants had positive perceptions of archive (3 & 4), and portfolios for assessment (5).
their ePortfolio technology. This difference in reception could be
explained by the fact that the participants in the pilot Freeman Impact of the ePortfolio approach
et al. conducted all volunteered to use the ePortfolio software. Our
participants, on the other hand, were included in the pilot because One perceived impact of the approach involved an increased
the department chairs agreed for the faculty in an entire program awareness by participants of areas of practice that could use
to participate in the pilot. This concurs with Smith and Tillema's improvement. Participants also thought more explicitly about
(2003) findings that the mandatory or voluntary nature of a planning their professional development. These findings suggest
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
that if adopted fully, the ePortfolio approach has the potential to Fourthly, a four-month period might be too short to really try
enhance self-directed learning. This concurs with the findings of out the ePortfolio approach. Since the approach assumes a yearly
Goliath (2009) who found that all nine physicians in her study cycle, and given that in the first cycle participants and department
scored higher on the readiness scale for self-directed learning as a chairs are still learning the process, a two-year pilot might be more
result of creating an ePortfolio. It also concurs with the impact useful and could include an initial orientation and a number of
Smith and Tillema (2003) found in that the personal development follow up sessions. Regular formative feedback from the pilot
type of portfolio gave ''participants space and opportunity for self- participants on the approach itself, as well as the support sessions,
directed learning, which was valued as contributing to professional could allow the faculty development support staff and the
growth'' (p. 645). More research is needed to study whether these department chairs to address concerns as they arise.
impacts can be sustained over time.
Design and implementation, lessons learned
The study of the impact of the ePortfolio approach was limited
Many faculty development initiatives serve a limited number of to self-reported outcomes described in interviews. The findings
faculty members over a limited amount of time (e.g., Stes et al., regarding the impact of the approach are limited because few
2010). Because of its potential to enhance reflective practice and participants actually adopted the approach in the way it was
self-directed learning (e.g., Goliath, 2009), an initiative like the intended. The interview guide and survey used were designed to
ePortfolio approach could be considered as a way to systematically directly inform practice and required data collection, analysis, and
support the professional learning of all faculty members and over a reporting to be completed within three months. Studying the
prolonged period of time, possibly throughout their entire career at participants' ePortfolios themselves could have increased our
the institute. However, upon the evaluation of the two pilots of the insight into the impact of the approach on professional learning,
ePortfolio approach, questions remain about whether the ePortfo- but this was considered to create an artificial environment where
lio would be suitable for institute-wide implementation and, if so, participants might feel the need to create an ePortfolio just for the
how it should be implemented. researchers. In addition, it might have created undue stress in
First of all, based on the findings, the question arises whether participants as they might be afraid the researchers would evaluate
the ePortfolio approach should be maintained in its current form. their performance rather than the impact of the ePortfolio on their
Participants indicated learning the ePortfolio software, Mahara, is professional learning. Other limitations of our methods include the
cumbersome. In addition, they were wary of posting information fact that the pilot 2 survey included improved survey questions,
online. Sharing portfolio documentation with their department which allowed the taskforce to gain more detailed insight into
chair does not require the use of ePortfolio software, nor does it participants' experiences, but unfortunately prevented the
require online storage. The literature suggests that composing an researchers from comparing survey data across pilots. Finally,
ePortfolio allows for enhanced reflective practice (e.g., Lamont, our recruitment method for the survey in both pilots, and the
2007), but the question is whether the extent to which this interviews in the second pilot, may have contributed to sample
reflection is enhanced by the use of the software warrants the costs bias. For the second pilot, we know that non-response was most
involved in using that software. These costs include online storage likely due to lack of time. However, we do not have conclusive
fees for third party storage provider, cost of technical support, cost information about this, as research ethics procedures stipulate that
of development and delivery of faculty workshops, as well as the survey and interview participation is voluntary, and no reason has
time spent by faculty to learn to use the software. Our pilot data did to be provided for not responding.
not suggest that the software added to the quality of reflection but
that it was the act of compiling documents that encouraged Implications for research
reflection on and formulating goals for professional development.
A simplified, less costly version of an ePortfolio could potentially While the use of portfolios for student assessment and student
serve the same purpose. This simplified version could, for instance, learning is taking hold in several disciplines, its use in supporting
include a template with questions for reflection and performance ongoing professional learning of post-secondary educators so far
assessment to be filled out by faculty, along with a number of has been limited. Yet, the literature suggests it could be a valuable
appendices including feedback, reflections, teaching materials, and approach to supporting faculty learning throughout their careers.
documentation of achievements and other records of performance. Further studies are needed to determine the organizational
Faculty could compile and store these documents on their own conditions facilitating the implementation of the ePortfolio
computers and show them to their chair during a performance approach, including issues of power and human resources
assessment meeting. processes. In addition, studies are needed to determine the design
Secondly, the findings of both pilots point to lack of time. Wider requirements for the ePortfolio approach that optimally facilitates
implementation of the ePortfolio approach would need to be self-reflection and self-directed learning, requires minimal learn-
carefully timed and not coincide with other drastic institute-wide ing on the part of the faculty, and facilitates performance
innovations. assessment. Carefully designed studies investigating the impact
Thirdly, our findings also highlight how the supervisors and of a well-designed ePortfolio tool accompanied by ePortfolio
their understanding of the approach played a role in determining assessment processes should be able to shed light on the impacts of
the need of their faculty for support and in setting expectations for such an approach on faculty development and ultimately on
their department members. Because of the important role they students' learning experience.
play, department chairs might need more guidance and prepara-
tion before the approach is introduced in their departments. In Conclusion
general, department chairs might play a larger role in fostering
faculty development: for instance, by encouraging transfer of Our study suggests that an ePortfolio approach to faculty
learning in courses and workshops into the workplace. (See, for development and performance evaluation has the potential to
instance, Lancaster, Di Milia, & Cameron (2012). Chairs would need support collection of various types of feedback, increase awareness
to set clear expectations for both the content of the ePortfolio as of areas of improvement, and encourage more explicit thought
well as how this content will be assessed. about planning and monitoring of professional development.
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
When adopted by a whole organization, the approach could Flanigan, E. J. (2012). ePortfolios and technology: Customized for careers. International
Journal of Information and Communication Technology Education, 8(4), 29–37.
integrate professional development plans, student feedback Freeman, L., Harkness, M., & Urbaczewski, A. (2009). International Journal of Global
practices and other feedback on performance, as well as Management Studies, 1(2), 31–46.
performance review processes. With careful implementation, a Fuller, A., & Unwin, L. (2011). Workplace learning and the organization. In M. Malloch,
L. Cairns, K. Evans, & B. N. O'Connor (Eds.), The SAGE handbook of workplace learning
well-designed ePortfolio process potentially constitutes an effec- (pp. 46–59). Los Angeles, CA: SAGE.
tive learner-centred approach to faculty development and perfor- Granberg, C. (2010). E-portfolios in teacher education 2002–2009: The social construc-
mance evaluation. tion of discourse, design and dissemination. European Journal of Teacher Education,
33(3), 309–322.
Goliath, C. L. (2009). Diffusion of an e-Portfolio to assist in the self-directed learning of
Acknowledgements physicians: An exploratory study. University of Akron (Dissertation).
Halonen, J. S., & Ellenberg, G. B. (2006). Teaching evaluation follies: Misperception and
misbehavior in student evaluation of teachers. In P. Seldin (Ed.), Evaluating faculty
A heartfelt thanks to the pilot participants who provided their performance: A practical guide to assessing teaching, research, and service (pp. 150–
thoughts and feedback in our interviews and surveys. Many thanks 165). San Francisco, CA: Jossey-Bass.
to the taskforce members for their collaboration with the research Harris, R., Simons, M., Hill, D., Smith, E., Pearce, R., et al. (2001). The changing role of staff
development for teachers and trainers in vocational education and training. Leabrook,
team. Thanks to the students in the Captioning and Court Reporting Australia: The National Centre for Vocational Education Research (NCVER) Ltd.
program for their assistance with interview transcription. We are Retrieved from hhttp://www.ncver.edu.au/publications/595.htmli.
grateful for the thoughtful input and editing assistance from Susan Higgerson, M. L. (2006). Building a climate for faculty evaluation that improves
teaching. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to
Chaudoir, Jeffrey Kuntz, and Jennifer Semchuk in the final stages of assessing teaching, research, and service (pp. 35–49). San Francisco, CA: Jossey-Bass.
writing. And finally, thanks to the blind reviewers for their helpful Hoekstra, A., Beijaard, D., Brekelmans, J. M. G., & Korthagen, F. A. J. (2007). Experienced
comments. teachers' informal learning from classroom teaching. Teachers and Teaching: Theory
and Practice, 13(2), 189–206. http://dx.doi.org/10.1080/13540600601152546
Hoekstra, A., Brekelmans, M., Beijaard, D., & Korthagen, F. (2009). Experienced teachers'
Appendix A. Supplementary data informal learning: Learning activities and changes in behaviour and cognition.
Teaching and Teacher Education, 25(5), 663–673. http://dx.doi.org/10.1016/
Supplementary data associated with this article can be found, in the j.tate.2008.12.007
Hoekstra, A., & Korthagen, F. (2011). Teacher learning in a context of educational
online version, at http://dx.doi.org/10.1016/j.stueduc.2015.03.007. change: Informal learning versus systematically supported learning. Journal of
Teacher Education, 62(1), 76–92. http://dx.doi.org/10.1177/0022487110382917
Hubball, H. T., Pratt, D. D., & Collins, J. B. (2006). Enhancing reflective teaching practices:
References Implications for faculty development. The Canadian Journal of Higher Education,
35(3), 57–81.
Amundsen, C., & Wilson, M. (2012). Are we asking the right questions? A conceptual Jasper, M., & Fulton, J. (2005). Marking criteria for assessing practice-based portfolios at
review of the educational development literature in higher education. Review of masters' level. Nurse Education Today, 25(5), 377–389.
Educational Research, 82(1), 90–126. http://dx.doi.org/10.3102/0034654312438409 Jimerson, J. B. (2014). Thinking about data: Exploring the development of mental
Axelrod, R. H. (2010). Terms of engagement: New ways of leading and changing organiza- models for ''data use'' among teachers and school leaders. Studies in Educational
tions (2nd ed.). San Francisco, CA: Berrett-Koehler Publishers. Evaluation, 42, 5–14. http://dx.doi.org/10.1016/j.stueduc.2013.10.01
Beausaert, S., Segers, M., Fouarge, D., & Gijselaers, W. (2013). Effect of using a personal Johnson, B. W. (2006). On being a mentor: A guide for higher education faculty. Mahwah,
development plan on learning and development. Journal of Workplace Learning, NJ: Lawrence Erlbaum Associates.
25(3), 145–158. Kirkpatrick, D. (2007). Four levels of evaluation: Measurement and evaluation. Pewaukee,
Billett, S. (2002). Critiquing workplace learning discourses: Participation and continu- WI: American Society for Training and Development.
ity at work. Studies in the Education of Adults, 34, 56–68. Lamont, M. (2007). What are the features of e-portfolio implementation that can
Blair, K. (2001). Electronic portfolios in tenure and promotion decisions: Making a enhance learning and promote self-regulation? Paper presented at the European
virtual case. In Retrieved from ERIC database (ED 466 138). Institute for E-Learning (EIfEL) conference.
Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta- Lancaster, S., Di Milia, L., & Cameron, R. (2012). Supervisor behaviours that facilitate
analytic review. Journal of Management, 36, 1065–1105. http://dx.doi.org/10.1177/ training transfer. Journal of Workplace Learning, 25(1), 6–22.
0149206309352880 Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation.
Bound, H. (2011). Vocational education and training teacher professional develop- Cambridge: Cambridge University Press.
ment: Tensions and context. Studies in Continuing Education, 33(2), 107–119. Lohman, M. C., & Woolf, N. H. (2001). Self-initiated learning activities of experienced
http://dx.doi.org/10.1080/0158037X.2011.554176 public school teachers: Methods, sources, and relevant organizational influences.
Brookfield, S. D. (1995). Becoming a critically reflective teacher. San Francisco, CA: Jossey- Teachers and Teaching: Theory and Practice, 7(1), 59–74.
Bass. Marsick, V. J., & Watkins, K. E. (1990). Informal and incidental learning in the workplace.
Butler, P. (2006). A review of the literature on portfolios and electronic portfolios. London/New York: Routledge.
Palmerston North: Massey University College of Education Retrieved from McColgan, K., & Blackwood, B. (2009). A systematic review protocol on the use of
hhttp://www.eportfoliopracticequt.edu.au/docs/Butler%20-%20Review%20o- teaching portfolios for educators in further and higher education. Journal of
f%20lit%20on%20ePortfolio%20research%20-%20NZOct%202006.pdfi. Advanced Nursing, 65(12), 2500–2507.
Bye, B., Pushkar, D., & Conway, M. (2007). Motivation, interest, and positive affect in McGovern, T. G. (2006). Self-evaluation: Composing an academic life narrative. In P.
traditional and nontraditional undergraduate students. Adult Education Quarterly, Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching,
57(2), 141–158. research, and service (pp. 96–110). San Francisco, CA: Jossey-Bass.
Carraccio, C., & Englander, R. (2004). Evaluating competence using a portfolio: A Millis, B. J. (2006). Peer observations as a catalyst for faculty development. In P. Seldin
literature review and web-based application to the ACGME competencies. Teaching (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research,
and Learning in Medicine, 16(4), 381–387. and service (pp. 82–95). San Francisco, CA: Jossey-Bass.
Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional Murray, J. P. (1997). Successful faculty development and evaluation: The complete
growth. Teaching and Teacher Education, 18, 947–967. http://dx.doi.org/10.1016/ teaching portfolio. In ERIC Document Reproduction Service No. ED405759. Washing-
S0742-051X(02)0005307 ton, DC: Graduate School of Education and Human Development, The George
Cox, M. D., & Richlin, L. (Eds.). (2004). Building faculty learning communities: New Washington University.
directions for teaching and learning. San Francisco, CA: Jossey-Bass. O'Meara, K. A., & Braskamp, L. (2005). Aligning faculty reward systems and develop-
Cross, K. P., & Steadman, M. H. (1996). Classroom research: Implementing the scholarship ment to promote faculty and student growth. National Association of Student
of teaching. San Francisco, CA: Jossey-Bass. Personnel Administrators Journal, 42(2), 223–240.
Darwin, S. (2007). The changing contexts of vocational education: Implications for Opfer, V. D., & Pedder, D. (2011). Conceptualizing teacher professional learning.
institutional vocational learning. International Journal of Training and Research, 5(1), Review of Educational Research, 81(3), 376–407. http://dx.doi.org/10.3102/
55–71. 0034654311413609
Day, C. (1999). Developing teachers: The challenges of lifelong learning. London: Falmer Pallett, W. (2006). Uses and abuses of student ratings. In P. Seldin (Ed.), Evaluating
Press. faculty performance: A practical guide to assessing teaching, research, and service (pp.
Devanas, M. A. (2006). Teaching portfolios. In P. Seldin (Ed.), Evaluating faculty perfor- 50–65). San Francisco, CA: Jossey-Bass.
mance: A practical guide to assessing teaching, research, and service (pp. 111–130). Polytechnics Canada (2014). Research and innovation. Retrieved from hhttp://www.po-
San Francisco, CA: Jossey-Bass. lytechnicscanada.ca/polytechnic_advantage/applied_research_commercialization/
Engeström, Y. (2011). Activity theory and learning at work. In M. Malloch, L. Cairns, K. ProgramsCourses/FacultiesSchoolsCentres/TeachingLearning/Departments/
Evans, & B. N. O'Connor (Eds.), The SAGE handbook of workplace learning (pp. 86– Academici.
104). Los Angeles, CA: SAGE. Roberson, D. N., & Merriam, S. B. (2005). The self-directed learning process of older,
Eraut, M. (2004). Informal learning in the workplace. Studies in Continuing Education, rural adults. Adult Education Quarterly, 55(4), 269–287. http://dx.doi.org/10.1177/
26(2), 247–273. http://dx.doi.org/10.1080/158037042000225245 0741713605277372
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13
Sambrook, S. (2005). Factors influencing the context and process of work-related learning: Saroyan & M. Frenay (Eds.), Building teaching capacities in higher education: A
Synthesizing findings from two research projects. Human Resource Development comprehensive international model. Sterling, VA: Stylus Publishing.
International, 8(1), 101–119. http://dx.doi.org/10.1080/1367886052000352591 Van Woerkom, M., & Croon, M. (2008). Operationalising critically reflective work
Schön, D. (1983). The reflective practitioner: How professionals think in action. New York, behavior. Personnel Review, 37, 317–331.
NY: Basic Books. Wayman, J. C., & Jimerson, J. B. (2014). Teacher needs for data-related professional
Seijts, G., Taylor, L., & Latham, G. (1998). Enhancing teaching performance through goal learning. Studies in Educational Evaluation, 42, 25–34. http://dx.doi.org/10.1016/
setting, implementation and seeking feedback. International Journal for Academic j.stueduc.2013.11.001
Development, 3(2), 156–167. Webster-Wright, A. (2009). Reframing professional development through understand-
Seldin, P. (2006). Essential operating principles and key guidelines. In P. Seldin (Ed.), ing authentic professional learning. Review of Educational Research, 79, 702–739.
Evaluating faculty performance: A practical guide to assessing teaching, research, and http://dx.doi.org/10.3102/0034654308330970
service (pp. 20–34). San Francisco, CA: Jossey-Bass. Zeichner, K., & Liu, Y. (2010). A critical analysis of reflection as a goal for teacher
Shuell, T. J. (1986). Cognitive conceptions of learning. Review of Educational Research, education. In N. Lyons (Ed.), Handbook of reflective inquiry. New York, NY: Springer.
56, 411–436.
Smith, K., & Tillema, H. (2003). Clarifying different types of portfolio use. Assessment
and Evaluation in Higher Education, 28(6), 625–648.
Annemarieke Hoekstra Ph.D., is a Teaching and Learning Specialist at the Northern
Statistics Canada (2010). Projections of the diversity of the Canadian population, 2006 to
Alberta Institute of Technology (NAIT). Her research interest is in teacher professional
2031. Ottawa, ON: Author.
learning. She currently holds a grant from the Canadian Social Sciences and Research
Steinert, Y. (2000). Faculty development in the new millennium: Key challenges and
Council to study the professional learning of vocational educators. She obtained her
future directions. Medical Teacher, 22(1), 44–50.
Ph.D. in Education from Utrecht University in 2007. Her dissertation focused on
Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., et al. (2006). A
informal teacher learning in the workplace. From 2007 until 2009, she worked as
systematic review of faculty development initiatives designed to improve teaching
an instructor in the JR Shaw School of Business at NAIT.
effectiveness in medical education: BEME Guide No. Medical Teacher, 28(8), 497–
Stes, A., Min-Leliveld, M., Gijbels, D., & Van Petegem, P. (2010). The impact of
instructional development in higher education: The state-of-the-art of the re- Jocelyn R. Crocker M.Ed., has been an instructor with the department of Biological
search. Educational Research Review, 5(1), 25–49. http://dx.doi.org/10.1016/ Sciences at the Northern Alberta Institute of Technology (NAIT) since 2004. She has
j.edurev.2009.07.001 received multiple teaching awards including the NAIT Instructional Excellence Award.
Stiehl, R., & Lewchuk, L. (2008). The outcomes primer: Reconstructing the college She obtained a Master of Education from the Department of Secondary Education at the
curriculum. Corvallis, OR: The Learning Organization. University of Alberta in 2010; her thesis focused on the experiences of foreign-trained
Taylor, L., & Rege-Colet, N. (2009). Making the shift from faculty development to professional women in adult upgrading. Her current research interests are in meta-
educational development: A conceptual framework grounded in practice. In A. cognition and teacher professional learning.
Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
The current issue and full text archive of this journal is available on Emerald Insight at:
66,2 Two dimensional
efficiency measurements
in vocational education
196 Evidence from Australia
Received 24 September 2015
Revised 15 February 2016
Peter Fieger, Renato Andrin Villano, John Rice and Ray Cooksey
4 April 2016 University of New England, Armidale, Australia
Accepted 5 May 2016
Purpose – In Australia, the vocational education and training (VET) sector accounts for approximately A$8
billion of public spending, of which around A$6.6 billion is spent on government providers that include
Technical and Further Education (TAFE) institutes. The TAFE institutes in Australia are large, public VET
providers, generally funded and managed by state government. Measuring the efficiency and effectiveness of
TAFE institutes is of great interest to policy makers, regulators, consumers and to the institutions
themselves. The paper aims to discuss these issues.
Design/methodology/approach – In this study the authors use data relating to student cohort
demographics, institutional characteristics and educational outcome data, while employing stochastic frontier
analysis, to develop two distinct efficiency measures and models. The first model examines institutional
efficiency in the transformation of financial resources into teaching loads. The second model evaluates efficiency
in the transformation of institutional resources into post-study employment outcomes. K-means cluster analysis
is used to establish groupings of similar institutes and subsequent canonical discriminant analysis is employed
to develop a typology of these clusters.
Findings – In both models the authors find significant inefficiencies in the Australian TAFE system. The
relationship between both efficiency measures is then assessed. While there is no direct linear relationship, a
distinct pattern could be detected. Finally the authors develop a typology of efficient institutions.
Originality/value – This study contributes to the existing research by defining efficiency in vocational
education in two distinct ways and by the utilisation of the derived efficiencies in the development of a
typology of efficient institutes. In doing so, this research makes an original contribution to the understanding
of the drivers of efficiency in vocational education.
Keywords Efficiency, Productivity, Effectiveness, Input/output analysis
Paper type Research paper
1. Introduction
In Australia, the vocational education and training (VET) sector accounts for approximately
A$8 billion of public spending, of which around A$6.6 billion is spent on government
providers that include Technical and Further Education (TAFE) institutes (NCVER
Financial Information, 2014). The TAFE institutes in Australia are large, public VET
providers, generally funded and managed by state government. They are geographically
dispersed, providing important access to tertiary education in regional and remote areas, as
well as in urban locations.
In Australia's Federal system of government, education and training is a shared activity
of the states and the central (Federal) government. The VET element of education, however,
has traditionally been dominated by state-funded and managed providers although the
Federal government has taken an active role in its regulation and funding over recent
decades (Burke, 2015). Unfortunately, this surplus of regulatory and funding attention has
International Journal of led to a VET system that is arguably in crisis, exhibiting all of the characteristics of
Productivity and Performance
Management over-bureaucratisation, inefficiency, waste and ineffectiveness (Garrick, 2011). VET is thus a
Vol. 66 No. 2, 2017
pp. 196-215
politically contested domain, dominated by vertical duplication between states and Federal
© Emerald Publishing Limited
agencies, and significant variance between states in relation to public spending rates per
DOI 10.1108/IJPPM-09-2015-0139 capita and teaching quality (Billett, 2014).
To further complicate matters, various Australian jurisdictions (at both the national and Two
state levels) have been promoting a variety of market-based reforms to increase competition dimensional
and anticipated efficiency. Generally these deregulatory reforms have favoured private efficiency
education providers to the detriment of established, government owned training providers
like TAFE institutes. These changes have thus caused considerable uncertainty in the measurements
provision of VET in Australia (Toner, 2014). Even aside from these exogenous pressures,
finite governmental resources, and the demand for greater accountability in vocational and 197
higher education training spending, have led to increased scrutiny in relation to training
outcomes and measuring returns on investment in public training expenditure.
Quite clearly, while the skills generated by vocational education are important for the
economy and society, the overall efficacy of the vocational education system in generating
those skills is regularly questioned.
In this dynamic and politicised policy context, measuring the efficiency and effectiveness
of TAFE institutes is of great interest to policy makers, regulators, consumers and to the
institutions themselves. It is a matter of significant conjecture whether the market-based
reforms are indeed improving systemic efficiency and effectiveness (Harris, 2015). A key
problem in making this determination relates to the manner in which efficiency is
determined in publicly funded VET institutes like TAFEs. Knowledge about institutional
efficiency may aid government agencies in allocating funds and in assessing the impact of
funding decisions. Furthermore, institutions themselves may use information about their
own efficiency to benchmark themselves against other institutions and to make adjustments
to their own resource allocation.
Compounding the problems of measurement are the longstanding problems with VET's
status within the Australian educational sector. Generally considered the "poor cousin" to
university education (Harris, 2015), the outcomes from VET are often difficult to determine
due to its student body's lower socio-economic status and generally lower academic
credentials. As such, any appropriate set of performance measures must investigate both
educational processes/activities and also educational outcomes.
This paper seeks to provide such an analysis through the application of an integrative
approach to efficiency measurement to explore both activity and delivery efficiency
(teaching load) and outcome efficiency (employment outcomes).
In this study we will define two different types of efficiencies in the Australian TAFE
sector and employ parametric stochastic frontier analysis (SFA) to determine the respective
efficiencies of individual institutes. The first empirical model is designed to estimate
efficiency in the transformation of financial resources into teaching hours (from here on
termed "teaching load model"). The second model estimates the efficiency of the
transformation of teaching resources into post-study outcomes, namely, the employment
rate of TAFE graduates (from here on termed "employment outcome model").
Once both institutional efficiencies for each institute have been established we will
analyse whether there is a relationship between both types of efficiencies, and whether a
typology of efficient institutes can be developed. We will proceed in the following manner:
first, we will review the theoretical underpinnings of the technique used and identify and
describe the appropriate variables and data that are going to be used in the analysis. Then,
we will operationalise the models, discuss the resulting estimates, and establish groups that
share similar patterns of efficiency. A canonical discriminant analysis will follow to
determine which variables are related to membership in different groups of efficiency.
Finally we will consider what practical relevance the research results have and whether
concrete policy implications could emerge from our findings.
In the next section a review of the literature in relation to the analysis of productivity in
public institutions is provided, with a special focus on SFA. This is followed by a narrower
review of the extant empirical literature exploring efficiency in the Australian vocational
IJPPM education sector. The paper then proceeds in Section 3 to present an overview of the method
66,2 employed in the analysis and an outline of the data utilised to explore institute-level
efficiency. Section 4 presents the results of the analysis, drawing on a variety of
visualisations to illustrate the various drivers of efficiency, with Section 5 discussing the
practical implications of or findings for the Australian VET sector.
3. Method of analysis
We will be estimating "teaching hours" efficiency and "employment outcome" efficiency
based on the stochastic frontier methodology developed by Aigner et al. (1977). The main
contribution of these authors was the introduction of a new approach to the specification of
the error term, namely, its separation into a normal "noise" term and a one sided inefficiency
term. Stochastic frontier production functions are an extension to the classic Cobb-Douglas
(1928) function which can generally be expressed in this form:
b b
Y ¼ eb0 X 1 1 X 2 2 . . . X bnn ee (1)
This model can then be transformed by taking the log of both sides and the error term then
be disaggregated into the statistical noise portion v, and the non-negative technical
efficiency component u which is distributed independently from v. The technical efficiency
TEi of individual DMUs of ui can then be determined by:
TEi ¼ eui (2)
Once we have estimated the institutional technical efficiencies for "teaching hours" and
"employment outcome" we will analyse the potential relationship between both efficiencies.
This will include the graphing of both efficiency components and a cluster analysis to
determine "efficiency clusters". Finally, we will employ canonical discriminant analysis with
the aim of developing a typology of efficient institutions.
where, z represents the hypothesised K predictors of efficiency and δ the parameters that
needed to be estimated. In our model we hypothesised that predominantly demographic
factors influence efficiency, as these factors may require administrative adjustments to
TAFE operations. We therefore entered the variables with institutional indicators for
English as a second language, disability, remoteness, age and sex, into our efficiency model
(for descriptive statistics see Table II). Equations (4) and (5) enable us to estimate the
Model1 Model2 Model 3
Variables Est PW|t| Est P W|z| Est PW |z|
Stochastic frontier
Constant −0.167 0.8 0.285 0.651 0.228 0.836
Funding per hour 0.01 0.828 0.001 0.976 0.018 0.711
206 Completion rate −0.025 0.477 −0.03 0.309 −0.008 0.923
Cert III or higher 0.19 0.154 0.107 0.384 0.03 0.872
Graduates −0.092 0.024 −0.077 0.014 −0.008 0.794
Very large – – – – – –
Large 0.046 0.227 0.051 0.1 0.052 0.172
Medium 0.078 0.053 0.089 0.005 0.152 0.003
Small 0.073 0.084 0.077 0.014 0.052 0.103
Very small 0.05 0.401 0.023 0.638 0.007 0.917
Inefficiency model
Constant −1.61 0.871
English second language 0.078 0.112
Students with disability 0.416 0.061
Remoteness (ARIA) 2.233 0.004
Student age −0.493 0.076
Proportion of males −0.003 0.944
Funding per hour −0.044 0.331
Completion rate 0.048 0.495
Cert III or higher 0.041 0.642
Graduates 0.017 0.684
Load pass rate −0.025 0.735
Very large – –
Large 1.333 0.272
Medium 1.32 0.286
Small −0.689 0.685
Very small −4.861 0.275
Wald χ2 28.08 o0.001 22.87 o0.001
Table VI. σv 0.037 0.001 0.043 o0.001
Estimates for OLS σu 0.131 o0.001 0.122 o0.001
and SFA models – σ2 0.018 o0.001 0.165 o0.010
employment outcome λ 3.51 o0.001 2.838 o0.010
model γ 0.925 0.739
1 38 36 1 18 33 343049
71 4
50 65
5 286651 46
55 31
1110 43 44 64 20
26 35 74748 70
Teaching hours efficiency
52 57
0.4 60
Figure 1. 58
Location of institutes
in teaching hours and 0.2 74
employment outcome
efficiency graph 0.6 0.7 0.8 0.9 1
Employment outcome Efficiency
An interesting pattern became evident from this graph. There appeared to be three major Two
constellations: some institutes scored relatively low on "teaching hours" efficiency and high dimensional
on employment outcome efficiency, whereas others attained a high teaching hours efficiency efficiency
and low employment outcome efficiency, and the remainder rated relatively high on both
efficiencies. Interestingly, there were no institutions that displayed low scores on both types measurements
of efficiencies examined in this study. It was of interest to statistically separate these three
possible combinations of teaching hours and employment outcome efficiency (e.g. high/high, 207
high/low, and low/high) and to evaluate the institutions that constituted the pattern in
Figure 1 with respect to possible observable characteristics, including demographic,
educational and environmental variables as determinants of group membership thereof.
We performed a partition cluster analysis, using the k-means method with three target
clusters. This technique involved an iteration process in which each institute was initially
randomly assigned to a cluster, and then subsequently was allocated to the cluster with the
closest mean, as calculated using the Euclidean distance method. After this, new cluster
means were determined and the process iteratively continued until no institute changed
groups. The resulting clusters can be seen in Table VII.
The location allocation following from the clusters in Table VII can be seen in Figure 2.
We then employed canonical discriminant analysis to examine the extent to which several
covariates could be utilised to statistically differentiate between locations 1, 2, and 3. The
covariates entered into the discriminant function were age, completion rate, load pass rate,
disability (per cent), remoteness, graduates (per cent), age, male gender (per cent), satisfaction,
salary, indigeneity (per cent), SES, Certificate III or higher (per cent), English as a second
language (per cent), Australian born (per cent), the percentage of apprentices and trainees, and
the size of the institution as measured by the number of student delivery hours. The essential
statistics for the two resulting discriminant functions can be found in Table VIII.
Location Institutes
1 38 36 1 5 28 1833 34
51 49
71 4 14 24 43 6566 15
37 50 44 64 20
55 31
1110 29 26 23 35 74748 70
Location 2 3222
Teaching hours efficiency
Location 3
52 57
0.4 60
Location 1
0.2 74 Figure 2.
Institutes by cluster
0.6 0.7 0.8 0.9 1 location
Employment outcome Efficiency
IJPPM It could be seen that both discriminant functions were significant, but that the first
66,2 discriminant function captured 79 per cent of the variance. The discriminating ability of the
covariates was then be assessed by the evaluation of the standardised canonical
discriminant function coefficients (Table IX).
Generally, values close to 0 indicated diminishing discriminating ability to separate the
three locations. The percentage of disabled students, for instance, had thus a negligible
208 contribution to the separability of the three efficiency locations. The discriminant function
coefficients were graphed for easier interpretation (Figure 3). Variables near the origin of
Function 1 Function 2
Standardized discriminant function 2
0.5 Austr born
Location 1 2 3
True Classified Total
1 6 0 0 6
100 0 0 100
2 0 8 3 11
0 72.7 27.3 100
3 0 1 38 39
0 2.56 97.4 100
Total 6 9 41 56
10.7 16.1 73.2 100 Table X.
Priors 0.11 0.20 0.70 100 Confusion matrix
2 33 3
1 3
Discriminant score 2
. 3 33
1 3 33 3
3 Centroid 3 3
Centroid 1 1 3 3333 .
3 2 33 3 3 3
0 3 3 3 3 3
1 3 3 3
1 3
2 . 2 .
1 2 3
3 2 3 2 2
–2 Centroid 2 2
2 Figure 4.
Discriminant function
–6 –4 –2 0 2
score plot
Discriminant score 1
IJPPM Location means Location differences PW |t|
66,2 1 2 3 1v2 1v3 2v3 PW |F|
Load pass rate 78.6 79.3 82.7 1.000 0.471 0.420 0.169
Completion rate 15.5 36.2 26.6 0.001 0.061 0.033 0.001
Disability (%) 7.9 10.5 9.3 0.231 0.801 0.660 0.195
Remoteness 4.0 1.8 1.9 o0.001 o0.001 1.000 o0.001
210 Graduates (%) 27.2 49.6 37.7 0.004 0.218 0.031 0.004
Age 34.2 31.5 33.2 0.046 0.855 0.071 0.028
Male (%) 63.9 51.6 57.7 0.068 0.521 0.274 0.063
Satisfaction 4.3 4.2 4.2 0.036 0.188 0.481 0.041
Salary 68,814 53,225 55,990 o0.001 o0.001 0.442 o0.001
Indigenous (%) 24.3 6.3 3.0 0.002 o0.001 1.000 o0.001
Cert III or higher (%) 73.3 81.7 83.5 0.154 0.020 1.000 0.024
English 2nd language (%) 14.2 20.0 15.6 0.732 1.000 0.567 0.359
Table XI. Australian born (%) 84.3 77.5 79.7 0.538 0.878 1.000 0.402
Location means and Apprentices and trainees (%) 18.0 15.5 17.3 1.000 1.000 1.000 0.736
comparison tests Institute size (in million teaching hours) 0.7 8.0 5.6 0.001 0.013 0.210 0.002
statistically different between all three locations, with location 2 exhibiting the highest
completion rate. While discriminant function loadings (Table IX and Figure 3) indicated
the strongest discriminating ability for remoteness, average age, and the percentage of
indigenous and Australian born students, in terms of significant differences between their
location means these categories were unremarkable. It is further worth reflecting that
while institutes in location 1 displayed several traits that may be considered to have a
negative connotation (such as the lowest completion rate, lowest percentage of graduates,
and lowest percentage of students enroled in Certificate III or higher courses), in respect of
some outcomes these institutes scored exceedingly well. For instance, graduates of
location 1 institutes had higher satisfaction rates than students from other locations, and
attained significantly higher post-training salaries. The lack of a coherent association
between the demographic, institutional, and environmental variables on one side and
combined institutional efficiency (e.g. "teaching hours" efficiency and "employment
outcome" efficiency) could be associated with other unobservable variables, that could the
institutes scores on both types of efficiencies. This means that, in the practical evaluation
of the productivity in the vocational education sector it should thus be kept in mind that
TAFE efficiency is a multi-dimensional concept and its results depend on carefully
defined input and output measures.
Efficiencies should be defined carefully depending on the specific property that is
intended to be evaluated. In our study we defined two separate types of efficiency
and created rankings for the TAFE institutes under examination. We found that
efficiencies calculated under one definition are not necessarily an indicator for efficiencies
obtained via alternative definitions. It therefore seems prudent to conclude that any
results stemming from the efficiency analysis of Australian TAFE institutes, and by
extension the efficiency of any group of public institutions, should always be accompanied
by a carefully phrased explanation on how efficiency was specifically defined.
The validity of the research results presented in this paper is somewhat limited by the
small number of institutions that we were able to include in our models. While the number of
possible inclusions is naturally limited to the number of TAFE institutes in Australia, some
institutes did not contribute data to this analysis. To overcome this we did consider
assembling a panel data set covering a number of periods. However, recent frequent
changes in the Australian TAFE sector, including institutional split-ups, amalgamations, Two
formations and mergers with universities suggested that this approach would not yield a dimensional
superior data set. efficiency
5. Conclusion
In this study we have applied stochastic frontier models to estimate two types of efficiencies
of Australian TAFE institutes, focussing on the transformation of financial and 211
administrative inputs into teaching load outcomes on one hand, and the transformation
of institutional resources into employment outcomes on the other.
Noting the abovementioned caveats in relation to how efficiency has been determined in
this study, in both models we have observed some clear inefficiencies. These inefficiencies
were mainly related to the degree of remoteness and student characteristics, both of which
could be seen as exogenous to the TAFEs themselves. For example, the least efficient TAFE
institutes were more likely to be found in remote locations, had a higher percentage of
males, and a larger proportion of individuals from non-English speaking backgrounds.
We speculate these inefficiencies were driven by a combination of interrelated factors,
including geographic location, available infrastructure and the absence of occupational
diversity of graduates.
As TAFE institutes serve a student constituency that is generally of lower socio-
economic status, an acknowledgement of the higher costs and lower efficiencies inherent in
providing educational activities to TAFE's traditional student body is an important finding.
In the policy context where private, for-profit providers tend to "cherry pick" the most
academically able and most readily geographically serviceable students, TAFEs tend to
provide an "educator of last resort" service for students who would otherwise be
marginalised from the tertiary education sector. This clearly drives issues associated with
institutional efficiency.
In the second part of this paper we analysed the association between the institutional
efficiencies estimated earlier. While there was no linear relationship we could detect a
distinct pattern of efficiencies. We further demonstrated that a typology could be developed
that predicted the institutional membership in distinct groups of efficiency.
Our two types of efficiencies have been specifically defined for this study. Theoretically,
it is possible to define an almost infinite number of other efficiencies. We showed in this
paper that different types of efficiencies of the same institutes are not necessarily linearly
related. For policy makers it is therefore necessary to take a multi-dimensional approach
that takes into account the various aspects of different approaches to the concept of
efficiency when making policy decisions.
The TAFE sector in Australia has a long history of providing education to low SES
students, to students in regional areas and to students with low prior academic achievement.
Measures of efficiency must take into account a variety of internal and external outcome
measures in determining the efficiency of the institutions.
The wider, but related, notion of institutional effectiveness would integrate questions of
economic efficiency and also consider the next best alternative at the student, community
and national level that would flow from the absence of the TAFE institutions. In many
instances, the alternative to TAFE education and a skilled job may well be unemployment,
social and economic marginalisation and welfare dependency.
This emphasises that in the efficiency analysis of educational institutions it is necessary
that any efficiency model needs to be specified with a clear purpose in respect to which
particular aspect of institutional efficiency is going to be investigated, and these efficiency
aspects need to be considered in wide context that investigates both internal operational
elements and also the social benefits that may flow from the upskilling of an archetypical
TAFE student.
IJPPM Notes
66,2 1. Odd years feature an augmented sample of the SOS, containing about 300,000 questionnaires, of
which about one third receives a response. In these years the SOS is designed to enable estimates
at an institutional level. In even years the SOS sample contains about 100,000 questionnaires, and
the focus of estimates is the state level.
2. In the context of this study, the term "TAFE and TAFE like institute" refers to TAFE institutes,
212 TAFE divisions of a university, Skills Institutes and Polytechnics. From here on only referred
to as "TAFE".
3. In this paper, we consider a p-value of 0.05op o 0.10 "marginal".
Abbott, M. and Doucouliagos, C. (2002), "A data envelopment analysis of the efficiency of Victorian
TAFE institutes", Australian Economic Review, Vol. 35 No. 1, pp. 55-69.
Abbott, M. and Doucouliagos, C. (2003), "The efficiency of Australian universities: a data envelopment
analysis", Economics of Education Review, Vol. 22 No. 1, pp. 89-97.
Abbott, M. and Doucouliagos, C. (2009), "Competition and efficiency: overseas students and technical
efficiency in Australian and New Zealand universities", Education Economics, Vol. 17 No. 1,
pp. 31-57.
Aigner, D., Lovell, C.A. and Schmidt, P. (1977), "Formulation and estimation of stochastic frontier
production function models", Journal of Econometrics, Vol. 6 No. 1, pp. 21-37.
Avkiran, N.K. (2001), "Investigating technical and scale efficiencies of Australian universities through
data envelopment analysis", Socio-Economic Planning Sciences, Vol. 35 No. 1, pp. 57-80.
Battese, G.E. and Coelli, T.J. (1993), "A stochastic frontier production function incorporating a model for
technical inefficiency effects", Working Papers No. 69, Econometrics and Applied Statistics,
University of New England, Armidale.
Battese, G.E. and Coelli, T.J. (1995), "A model for technical inefficiency effects in a stochastic frontier
production function for panel data", Empirical Economics, Vol. 20 No. 2, pp. 325-332.
Bayraktar, E., Tatoglu, E. and Zaim, S. (2013), "Measuring the relative efficiency of quality
management practices in Turkish public and private universities", Journal of the Operational
Research Society, Vol. 64 No. 12, pp. 1810-1830.
Billett, S. (2014), "The standing of vocational education: sources of its societal esteem and implications
for its enactment", Journal of Vocational Education & Training, Vol. 66 No. 1, pp. 1-21.
Burke, G. (2015), "Australia's funding schemes in postsecondary education and disadvantaged
students", Journal of Educational Planning and Administration, Vol. 29 No. 1, pp. 5-27.
Carrington, R., Coelli, T. and Rao, P. (2005), "The performance of Australian universities: conceptual
issues, and preliminary results", Economic Papers, Vol. 24 No. 2, pp. 145-163.
Charnes, A., Cooper, W.W. and Rhodes, E. (1978), "Measuring the efficiency of decision making units",
European Journal of Operational Research, Vol. 2 No. 6, pp. 429-444.
Chen, C.F. (2007), "Applying the stochastic frontier approach to measure hotel managerial efficiency in
Taiwan", Tourism Management, Vol. 28 No. 3, pp. 696-702.
Chesters, J. and Smith, J. (2015), "Social capital and aspirations for educational attainment: a cross-national
comparison of Australia and Germany", Journal of Youth Studies, Vol. 18 No. 7, pp. 932-949.
Cobb, C.W. and Douglas, P.H. (1928), "A theory of production", The American Economic Review, Vol. 18
No. 1, pp. 139-165.
Cullinane, K. and Song, D.W. (2006), "Estimating the relative efficiency of European container
ports: a stochastic frontier analysis", Research in Transportation Economics, Vol. 16 No. 1,
pp. 85-115.
Farrell, M.J. (1957), "The measurement of productive efficiency", Journal of the Royal Statistical Society. Two
Series A (General), Vol. 120 No. 3, pp. 253-290. dimensional
Fieger, P. (2010), "An investigation of TAFE efficiency", 2010 AVETRA Conference, Surfers Paradise, efficiency
8-9 April.
Fieger, P. (2016), "Efficiency and effectiveness in the Australian technical and further and education
system", doctoral thesis, University of New England, Armidale.
Garrick, B. (2011), "The crisis discourse of a wicked policy problem: vocational skills training in 213
Australia", The Australian Educational Researcher, Vol. 38 No. 4, pp. 401-416.
Grosskopf, S., Hayes, K.J. and Taylor, L.L. (2014), "Efficiency in education: research and implications",
Applied Economic Perspectives and Policy, Vol. 36 No. 2, pp. 175-210.
Harris, R. (2015), "Quality in the Australian VET sector: what has been happening?", International
Journal of Training Research, Vol. 13 No. 1 pp. 16-34.
Horne, J. and Hu, B. (2008), "Estimation of cost efficiency of Australian universities", Mathematics and
Computers in Simulation, Vol. 78 No. 2, pp. 266-275.
Huang, C. and Liu, J.T. (1994), "Estimation of non-neutral stochastic frontier production function",
Journal of Productivity Analysis, Vol. 5 No. 2, pp. 171-180.
Izadi, H., Johnes, G., Oskrochi, R. and Crouchley, R. (2002), "Stochastic frontier estimation of a CES cost
function: the case of higher education in Britain", Economics of Education Review, Vol. 21 No. 1,
pp. 63-71.
Kumbhakar, S. and Lovell, C. (2000), Stochastic Frontier Analysis, Cambridge University Press,
Kumbhakar, S., Gosh, S. and McGuckin, J.T. (1991), "A generalised production frontier approach for
estimating determinants of inefficiency in US dairy farms", Journal of Business and Economic
Statistics, Vol. 9 No. 3, pp. 279-286.
NCVER Financial Information (2014), Australian Vocational Education and Training Series, NCVER,
Parsons, L.J. (2004), Measuring Performance Using Stochastic Frontier Analysis: An Industrial Sales
force Illustration, Institute for the Study of Business Markets, The Pennsylvania State
University, PA.
Pillay, H., Watters, J.J. and Hoff, L. (2013), "Critical attributes of public-private partnerships: a case
study in vocational education", International Journal of Adult Vocational Education and
Technology, Vol. 4 No. 1, pp. 31-45.
Reifschneider, D. and Stevenson, R. (1991), "Systematic departures from the frontier: a framework
for the analysis of firm inefficiency", International Economic Review, Vol. 32 No. 3, pp. 715-723.
Toner, P. (2014), "Contracting out publicly funded vocational education: a transaction cost critique",
The Economic and Labour Relations Review, Vol. 25 No. 2, pp. 222-239.
Waldman, D.M. (1982), "A stationary point for the stochastic frontier likelihood", Journal of
Econometrics, Vol. 18 No. 2, pp. 275-279.
Wang, H.J. and Schmidt, P. (2002), "One-step and two-step estimation of the effects of exogenous
variables on technical efficiency levels", Journal of Productivity Analysis, Vol. 18 No. 2,
pp. 129-144.
Worthington, A.C. and Lee, B.L. (2008), "Efficiency, technology and productivity change in Australian
universities, 1998-2003", Economics of Education Review, Vol. 27 No. 3, pp. 285-298.
Technical efficiency
Institute Teaching load efficiency Employment outcome efficiency
214 1
5 0.973 0.927
7 0.932 0.950
10 0.953 0.870
11 0.943 0.860
13 0.971 0.989
14 0.953 0.859
15 0.978 0.982
16 0.966 0.991
17 0.953 0.981
18 0.986 0.944
19 0.960 0.973
20 0.963 0.973
22 0.862 0.976
23 0.921 0.924
24 0.964 0.878
25 0.968 0.980
26 0.908 0.896
27 0.985 0.990
28 0.973 0.939
29 0.959 0.871
30 0.987 0.978
31 0.967 0.992
32 0.866 0.968
33 0.982 0.956
34 0.996 0.973
35 0.920 0.929
36 0.986 0.891
37 0.979 0.719
38 0.991 0.780
40 0.621 0.946
43 0.960 0.926
44 0.946 0.941
45 0.893 0.669
46 0.980 0.983
47 0.916 0.955
48 0.927 0.963
49 0.992 0.983
50 0.972 0.819
51 0.981 0.954
52 0.739 0.938
53 0.840 0.995
55 0.967 0.978
56 0.474 0.932
57 0.723 0.997
58 0.327 0.953
Table AI. 60 0.389 0.995
Teaching load 64 0.948 0.969
efficiency and
employment outcome
efficiency by institute (continued )
Technical efficiency dimensional
Institute Teaching load efficiency Employment outcome efficiency
65 0.977 0.940 measurements
66 0.979 0.947
70 0.918 0.986
71 0.978 0.724 215
74 0.198 0.885
77 0.983 0.983
110 0.423 0.994
Mean 0.888 0.929
SD 0.182 0.074 Table AI.
Corresponding author
Peter Fieger can be contacted at: firstname.lastname@example.com
For instructions on how to order reprints of this article, please visit our website:
Or contact us for further details: firstname.lastname@example.com
Reproduced with permission of the copyright owner. Further reproduction prohibited without