Last Friday our Credits for MOOCs project opened for students. From this week on our students can join a number of MOOCs and get credits for them. There are no additional costs for them.
Expanding Portfolio
The first 12 MOOCs for credits will be provided by Ecole Polytechnique Fédérale de Lausanne (EPFL), University of Queensland, Australia National University and TU Delft and they will start in February 2017. During the summer a broader variety of courses will be made available by universities such as Hong Kong University of Science and Technology and Rice University. Other top 100 universities have said they are keen to join.
We started this project in the fall of 2015 and after many conference calls, presentations, meetings and hard work of my colleagues and our partners we have managed to open it for enrolments. When MOOCs are infiltrating the university curriculum you have to pass many hoops and bypass many obstacles. Our students can find all the information about the process and which MOOCs are offered on this website.
Assessment
One important aspect is that we asked all the instructors of the MOOCs under what conditions they would offer credits for their MOOCs. For most MOOCs this means that there is an additional assessment requirement. That can be like a proctored exam, a skype interview, or a paper. This is also the reason why we have limited the number of students that can credits. Off course any learner can enroll in these MOOCs.
Virtual Exchange
All current exchange programmes are based of students to temporary move to another city. A great opportunity for students, but for many not possible due to time or financial constraints. With this programme we created a kind of virtual exchange programme. Our students can do courses of other top universities from our campus in Delft. They can learn across borders.
I think this offers a great opportunity for our students. It offers them courses that we as a technical university don't offer and it gives them flexibility.
There are many technologies flooding the market that help foster innovative teaching and learning. These tools, such as learning management systems, lecture capture systems, simulation creators, authoring, and video and audio tools, have flooded into the classrooms and lecture halls of higher education. However, the inference that these innovative tools aid learning should not be immediately assumed. With faculties’ full work load, learning and implementing new and often complex tools to improve their online pedagogy isn’t a priority. In fact, as the needs and tools of institutions have evolved, instructional designers have positioned themselves as pivotal players in the design and delivery of learning experiences. Instructional designers exist to bridge the gap between faculty instruction and student online learning. But who, exactly, are instructional designers? What do they do? Where do they fit in higher education?
Their findings:
Instructional designers number at least 13,000 in the U.S alone.
They are highly and diversely qualified.
Contrary to popular belief, they do more than just design instruction.
Above all, they struggle to collaborate with faculty.
One thing is certain: instructional designers are dedicated to improving learning with technology
I recognise their findings if I look at my team and their activities.
Highly and diversely qualified
What they name "instructional designer" we separated it in two different job titles:
learning developer: focused on creating the learning design of a course (constructive alignment) and often acting as project manager
instructional designer: expert in how to setup the learning design in the a specific platform (we have (open)edX, Blackboard and Brightspace.
All the learning developers have some form of educational science background. From a master in educational science to experience as lecturer with additional courses. The instructional designer have a much more technical background, one of my instructional designer is a watermanagement engineer. They all have finished a Bachelor and Master and some even have a PhD in education.
They do more than just design instruction
According to the report most instructional designers have four categories of responsibilities:
Design instructional materials and courses, particularly for digital delivery
Manage the efforts of faculty, administration, IT, other instructional designers, and others to achieve better student learning
Train faculty to leverage technology and implement pedagogy effectively
Support faculty when they run into technical or instructional challenges
The activities of my team perfectly aligns with 4 categories. Although sometimes they do even more, such as marketing, administration, policy advise.
Struggle to collaborate with faculty
With every course team an instructional designer has to win the trust of the faculty members. Faculty members are the experts on the topic of the course and have been teaching for many years. They have to accept that designing and delivering an online or blended course is a different cup of tea than classroom teaching. The instructional designer have been done this for many courses, so they know how to design an online course.
Dedicated to improving learning with technology
Despite the struggle with faculty they are always dedicated to make it work. Sometimes it means that the end result is not the best, but it was what could be reached within the time span and context of that course.
Recommendations
In the report there are a few recommendations. Two I want to strongly emphasize:
Institutional leaders and administration - Involve instructional designers early, often, and throughout your technology transition. Develop clear standards that are expressed to all participants — institutional leaders, instructional designers, faculty, and students. Also, think about incentivising faculty to work with instructional designers from the get-go. Survey respondents made clear that they’d like more resources allocated to their work, which could have a high return on investment in terms of student success.
Faculty - We know student success is top priority for you. An instructional designer can help you engage your students and give you more class time by using online tools. There is potential impact to be made for your students by collaborating and using new technologies that instructional designers can guide you to. They share your goal and want to see you shine for your students.
Since a couple of years the Open University creates an report about innovating pedagogy. Yesterday they published the fifth report. It is produced in collaboration with the Learning Sciences Lab at the National Institute of Education, Singapore, proposes ten innovations that are already in currency but have not yet had a profound influence on education.
The ten innovations they identified were rated on potential impact and in what timescale.
Trend
Potential impact
Timescale
Learning through social media Using social media to offer longterm learning opportunities
Medium
ongoing
Productive failure Drawing on experience to gain deeper understanding
High
Medium (2-5 years)
Teachback Learning by explaining what we have been taught
High
Medium (2-5 years)
Design thinking Applying design methods in order to solve problems
Medium
ongoing
Learning from the crowd Using the public as a source of knowledge and opinion
Medium
Long (4+ years)
Learning through video games Making learning fun, interactive and stimulating
Medium
Medium (2-5 years)
Formative analytics Developing analytics that help learners to reflect and improve
High
Medium (2-5 years)
Learning for the future Preparing students for work and life in an unpredictable future
High
Ongoing
Translanguaging Enriching learning through the use of multiple languages
Medium
Medium (2-5 years)
Blockchain for learning Storing, validating and trading educational reputation
High
Long (4+ years)
Reference Sharples, M., de Roock , R., Ferguson, R., Gaved, M., Herodotou, C., Koh, E., Kukulska-Hulme, A., Looi, C-K, McAndrew, P., Rienties, B., Weller, M., Wong, L. H. (2016). Innovating Pedagogy 2016: Open University Innovation Report 5. Milton Keynes: The Open University. ISBN 9781473022812
Personally I think that metaphor oversimplifies the process of a learner and it doesn’t help to solve the challenges that many universities face regarding their educational technology. A couple of reasons why it is flawed:
Not all educational technology tools that are used as part of the learning experience are created equally, there is much more variety in the ‘bricks".
Just sticking bricks together doesn’t make it a learning experience. The flow and interactive design are important.
Accessibility is much harder to guarantee, especially if instructors can add their own bricks via LTI.
Learning is much more complex.
We need to intelligently orchestrate all these different “bricks” to serve the student. In some situation it can just be as simple as IFTTT: if student has passed this quiz, he can go to the next chapter. But in many cases this process is far more complex than an IFTTT recipe of course, especially if you connect it to all the context. It is more like a central nervous system.
Next year there are elections in the Netherlands and political parties are already publishing their programmes. Due to the strict policy (budget cuts) and smart investments, government deficit has been reduced significantly. That means that there is money to spend and the different parties are claiming their priorities.
One of the priorities for many parties is more money for hiring more teachers in higher education. Although the workload of many teachers is high, I have my doubts if this will solve the problem.
Problem
Education is changing in higher education. One of the important changes is the digitization in education. I hear many universities and colleges say that blended education will be the default. This means that education will be a mix between classroom teaching and online learning. This change means that the role of a teacher is changing, but also that more disciplines need to be involved to develop and run a course. Teaching has been a one man job for many centuries and I think it is time to rethink our perception on this
Course Teams
Education has become a complex balance between content, technology and pedagogy. It is very hard to find people that are subject matter expert, as well as tech-savvy as well as up-to-date on pedagogy (so no urban myths) and think about effective ways of examining learning outcomes - all at the same time. In addition, in most universities staff members also need to be researchers, grant experts, and (PhD) student supervisors. These are too many roles to ask of one person.
So we need to disperse these roles to a team of people, each with their own dedicated role and expertise.
In the last two years at TU Delft Extension School we have pushed that a teacher forms a course team to develop and run a course. The minimum for a course team is:
Depending on the course objectives additional roles can be added. This is a list of the expertise we have in our support team to be added to the course teams:
Assessment expert: digital testing is different from exams on paper.
Instructional designer: expert on how to technically implement your course in your platform.
Copyright officer: to correctly use and reuse (open) content.
Data Analyst: lots of data is generated that can be analysed and used to improve the course experience.
Additional specialists for online labs, simulations, games, etc.
This approach has helped us to develop high quality online courses, but also lifted some of the burden from the shoulders of the teachers.
Conclusion: new perspective of teaching and support
Investing in extra teachers in higher education might seem like a proper way of spending extra budget. Investing in better course teams will have a much bigger effect to unburden teachers. Don’t invest in extra teachers, make existing teachers much more effective by properly supporting them. So better value for money!
To tackle the challenges of providing top-level university education, Leiden University, Delft University of Technology and Erasmus University Rotterdam (LDE) have joined forces. Through this partnership three universities with complementary specializations create a world-class platform for educational research, innovation and training: the Centre for Education and Learning (CEL).
The LDE Centre for Education and Learning now seeks to appoint a Professor of Higher Education with a focus on teaching and learning processes in a digital environment. The Professor will be formally appointed at Delft University of Technology.
You will work closely with the online learning team of TU Delft, the Online Learning Lab of University of Leiden and project Online Onderwijs of Erasmus University Rotterdam.
Job Description
First, the chair covers the whole domain of higher education and includes the scientific reflection on research and theory development in the broad field of higher education. Subareas of higher education research and theory development include 1) students and their study success, 2) teachers and their teaching success, 3) learning environments for on-campus, online and blended teaching and learning, 4) special topics such as assessment, internationalization, diversity, and educational policy, and 5) teaching and learning in individual disciplines.
Second, the chair has a focus on teaching and learning in a digital environment, for instance online and blended forms of education followed by data analytics of these learning processes. New knowledge is to be created in this relatively new field of research. The three LDE universities are among the forerunners in this field and consider it of the utmost importance that these innovations are as much evidence-based as possible. The chair holder can rely and build on existing expertise within CEL and its pillars in the individual universities. It is expected that the chair holder brings the various researchers together and promotes knowledge exchange and joint knowledge creation. The research will enable evidence-informed university policy making, evidence-informed development of new university teaching projects, evidence-informed continuous professional training of university teachers, and will ameliorate teaching and learning theory.
This week I'm attending the EDEN annual conference in the beautiful Budapest. Nelson Jorge, Sofia Dopper and me wrote a paper about the TU Delft Online Learning Experience. This is a pedagogical model that supports the development of our courses and strives for increasing quality. The creation of the OLE was an important step for TU Delft, contributing to the development of online courses in a more systematic and consistent way, guiding all course development teams through the realisation of several shared educational principles.
At the Gala Diner of the conference we received the EDEN 2016 Best Practice Inititative Award for our paper. The award is not just for the paper but for the whole initiative of designing the model and implementing it for our courses. I see it as a great appreciation for the work we are doing with the TU Delft Extension School.
Award for TU Delft
Online Learning Experience model
The OLE holds 8 principles to support course teams in the design and development of online courses. The model was elaborated based on the foundations established by distance learning experts (Moore, 1991; Keegan, 1996; Palloff & Pratt, 1999; Garrison, 2000; Peters, 2000; Anderson, 2003; Garrison & Anderson, 2003; Salmon, 2011; Salmon, 2013; Bates, 2015) and the know-how of the TU Delft Online Learning Course Development Team.
Jorge, Nelson; Van Valkenburg, Willem; Dopper, Sofia (2016). The TU Delft Online Learning Experience: From Theory to Practice in Teixeira, Szucs and Mazar (2016). Conference Proceedings EDEN 2016 Annual Conference. ISBN 978-615-5511-10-3. License CC-BY 4.0
Presentation
Here is the presentation Nelson presented at the conference.
And the link to the model explaining all the principles. Below is the video of teacher's perspective of Online Learning Experience
After an European tender procedure Delft University of Technology has selected a new LMS supplier. After 17 years we are saying goodbye to Blackboard and are going to migrate to the cloud-based platform of the Canadian Desire2Learn: Brightspace Learning Sytem. I'm very pleased with the result of our tender based on best value procurement. We have selected a partner that is eager to work with us for the next 10 years with a product that fits our strategy and is ready for the future. The new platform not only includes the full Brightspace Learning Environment (including ePortfolio, Learning Repository), but also their full Learning Analytics platform, including their predictive Learning Analytics system.
Best Value
Before we started with our tender we noticed that the traditional tender method of listing all our requirements gave such a long list, that no supplier would meet all requirements. So that selecting the best solution would be hard. That is why we changed to a best value procurement. In stead of listing all the requirement we wrote down our mission, strategy and goals we wanted to reach with the conditions (price ceiling). It was now up to the suppliers to use their expertise and know-how to provide us with the best solution they could offer within the conditions.
It also meant that they didn't need to provide us thick offers. It was limited to 2 pages for their performance substantiation, 2 pages for their risk file, and 2 pages for their opportunity file. Next to the paperwork each supplier could send 2 key persons that would be interviewed according to a standard list of questions (first question is why are you a key person?).
The most interesting is in the interviews and dossiers that we are looking at relevant dominant data. So no marketing talk, but real measurable data that can be verified. So for example, not "we have done many successful implementation", but "we have done 83 implementations in the last 2 years. Of which 79 were on time and within costs. The industry standard is 80%". This also meant that during the tender the people that will do the implementation would need to be involved. And that really improves the quality of their dossiers (if you involve the right persons).
The grades of the dossiers and interviews are based on a system that starts at 6. If you have dominent information it can go up to 8 or 10 or down to 4 or 2. So no dominant data means a 6. These grades are converted into a subtraction of the price. Combining that with the price of their offer leads to a ranking. The number one goes to the clarification & verification phase.
In the clarification & verification phase we worked together with the team of Desire2Learn to create the plan (a whole list of deliverables) and to verify their dossiers (If that doesn't work out or the verification shows error we would move on to the supplier that was ranked second). We are not buying a platform, but a plan to implement their platform. Yesterday we have finished this process and we have (provisional) awarded the tender to D2L.
Desire2Learn
Desire2Learn (D2L) is a Canadian based company founded in 1999, that is still run by the founder John Baker. According to the Ovum decision matrix for selecting an online learning platform D2L is:
Brightspace received the highest overall technology assessment score, obtaining at least a top-three rating in all 15 categories. Not unexpectedly, Brightspace received a perfect score for student performance and retention. D2L offers analytics-driven progress monitoring capabilities from within Brightspace, and in 2012 the company partnered with IBM to deliver the Smarter Education Solution, which incorporates an intervention management system and predictive analytics. Although D2L is ahead of other OLP providers when it comes to integrated analytics – and in particular predictive
analytics – the company upheld its promise to drive successful learning outcomes and its reputation for providing an open learning platform that can easily integrate with other education technologies by partnering with IBM. IBM is more attuned to predictive models and data systems, and together the two companies can help institutions leverage student data in meaningful ways. Separately, D2L also achieved a perfect score for accessibility. Its accessibility program is integrated into its R&D lifecycle, and designs are regularly reviewed with its Accessibility Interest Group, which demonstrates its commitment to this category. D2L combines all of its capabilities with impressive training and support services, and a high-touch approach to customer engagement. For example, D2L has designed custom training sessions at the request of some of its customers to help institutions learn more about topics such as accessibility. Ovum anticipates that as the industry moves into the next phase of OLP purchasing, vendors with strong support services around its solutions will be particularly appealing.
Although at its core D2L is a technology provider, it also has a strong focus on pedagogy and how enhanced learning experiences can help address the skills gap when students move on to employment. As a result, D2L received the highest score for the capacity to support next-generation online teaching and learning. The Brightspace platform moves away from a one-size-fits-all approach and is instead highly personalized to meet differing student needs. Furthermore, D2L was ahead of its competitors in addressing demand for competency-based learning and adaptive learning.
Ovum recommends that as a market leader, Brightspace by D2L should be included in an institution's list of OLPs. Moving from managing to improving learning, Brightspace meets the core functionality criteria defined in this ODM, and although its brand awareness could be stronger in certain regions it is certainly strong in North America and among its competitors. The company is continuously evolving its offerings to meet the needs of the higher education market.
Implementation & Migration
So after all the formal and legal stuff around the tender, now we can start the actual work. We have formed a great team of people from D2L and TU Delft that will run the project under the project management of Erna Kotkamp. We are very lucky we have someone as Erna in our team. With her passion, drive, skills and eye for detail, I'm convinced this will be a successful project that will give our lecturers and students a platform for the next ten years.
Update 30 June: Stand-still period has ended and the contract is signed.
On Thursday I presented at the Qualtrics Live Event in Amsterdam. I was asked to present about our MOOCs as inspiration for the other participants (about 25). At the end of the presentation I got the question what we are doing with Qualtrics. Although I gave the presentation I'm not the one that is handling our Qualtrics activities. It was a good thing that Sara and Jan-Paul had joined me at the event!
Use of Qualtrics
We use qualtrics in 5 different ways in our MOOCs.
Surveys
In all our online courses we have pre-, mid- and post-surveys. These surveys are mostly the same, although there are some custom questions per course. Before the surveys are added to the course, we ask the course teams if they have any specific questions to ask. In total we have more than 100,000 responses to these surveys.
Research Survey
The second category are surveys for course teams to get data for their research. Usually the questions in these MOOCs are related to the topic of the courses. This is a fast and cheap way to collect data from a very international group of learners. For example, the course team of framing included survey where participants were asked to respond to a certain 'frame'. Their interest was to find difference depending on the cultural differences of the learners.
As improvement of the EdX Quiz module
The EdX Quiz module is rather basic and lacks the advanced logics that qualtrics has to create custom paths through a survey. Because we link the user id of the edx platform to the specific survey response, we can link their response to their other results and activities in the course.
Marketing Surveys
On our website and in direct emails to our learners we use marketing surveys to get more insights about our learners. We even offer a professional education course about this, starting in a couple of days.
Support surveys for our learning interventions
Our research team does not only analyse the data, but also does learning interventions in some of our MOOCs. Around these interventions they use surveys to get additional information from the learners. One of the learning interventions was a learner tracker. The research team presented this at the Learning Analytics for Learners workshop last month in Edinburgh (paper).
Reference
Dan Davis, Guanliang Chen, Ioana Jivet, Claudia Hauff, Geert-Jan Houben (2016). Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback. In Learning Analytics and Knowledge 2016 Learning Analytics for Learners Workshop. [ Bibtex ]
our surveys include questions that are similar to those of the UW research
platform data
Survey data
If we look at the survey data (for 10 courses), the results are:
80% of both groups of students (developing and developed countries) reported they had taken an online course before;
12% of students from developed countries that had taken an online course before, never completed any course, and 16% from developing, which means that
88% of students from developed countries that had taken an online course before state they completed at least one course, and 84% of students from developing countries.
Here we see that many students actually completed at least one course in both groups, but overall percentages are still slightly “in favour” of students from developing countries.
Platform data
If we look at the platform data (example of one course):
The average grade of developing countries is 3,73% vs. 5,55% for developed, and the passing rate is 2,93% vs. 5,26%.
There are more students from developing countries that hadn’t really started the MOOC (i.e. had a grade = 0), 91% vs. 87%, but even among those that did (grade > 0), the average grade is lower (36% vs 40%), as well as passing rates (32% vs. 47%).
Only when you look only at people who received a certificate, the average grade is basically the same.
Because Sara is very good, she also had some comments about shortcomings of the original research:
It tries to compare the result of their survey about completing any course ever to per-course completion rates. While it acknowledges that they don’t really have the actual data for comparison, it is still misleading to mention it alongside it, because in reality, it doesn’t tell us anything. They have no idea how many students in general completed at least one course already in developed countries.
The comparison is problematic even further because of very different sources of information. While the completion rates are actual, true numbers, the survey is an estimate, that can be largely biased. We know that more students that receive a certificate in the end complete even the pre-survey (compared to the actual passing rates), which may also be true in this case, i.e. more engaged students completing their survey.
Also, they compare their “completion” to per-course “completion rates”. But, our completion rates are actually certification rates, while their “completion” is completing the course with not necessarily receiving a certificate. Furthermore, students may understand very differently, what does it mean to complete a course, possibly connected to their intentions. We have no idea how many students in our courses would say they completed the course. So this is another point why these numbers can hardly be compared to the regular “completion rates”.
On a per-course basis, the number of “registrants” is rather high (usually only around 50% of students does anything in our courseware), which is very far from the 2% of “registrants” they identified in their sample, which further shows that their completion numbers and course completion numbers can hardly be comparable.
Their study is conducted only on people between 18-35 years old. As we know there are many students above 30 (30 is usually the median) in our courses, this is hardly representative of all MOOC users.
Conclusion
The researchers based their conclusion solely on self-reported survey data, but tried to compare their result to actual per-course completion rates, which creates a false sense that students from developing countries actually complete more courses than their peers from developed countries. While the high completion rates among students from developing countries may still be a surprise, it is important to keep in mind what we are actually looking at. Both platform data and survey data of ours revealed that somewhat more students from developed countries complete courses, or receive a certificate. In most of our research we combine the survey data with the platform data to get more accurate results and less user bias.
