E-learn Weblog of Willem van Valkenburg

Recently the TU Delft Executive Board approved a policy document on learning analytics. This policy document is about learning analytics at TU Delft and is comprised of a vision on the topic, a policy framework with directional statements, and an overview of recommended next steps for the introduction of learning analytics at TU Delft.

Tthe field of learning analytics is still developing and will be in the coming years. This makes it difficult to predict how exactly TU Delft will be using learning analytics in a few years. Learning analytics policies must therefore take into account the dynamics in the field – we do not know what is possible in the future; but at the same time offer guidance – we know what we do not want.

Creating this document

Jan-Paul van Staalduinen was de principal author of this document. He based it on literature research and a series of interviews. An analysis of the current situation at TU Delft was based on available policy documents within TU Delft and discussions with various stakeholders. An overview of external developments was based on available (academic) literature and discussions with experts and researchers in the area of learning analytics. To get an idea of what other institutions had already done in this area, available learning analytics policy documentation was analysed and policy staff and decision makers of those institutes were interviewed.

Defining Learning Analytics

Learning analytics is about “collecting traces that learners leave behind and using those traces to improve learning (Duval, 2012)”. George Siemens (2011) argues that the broader field of educational data mining (EDM) includes both learning analytics and academic analytics (including institutional analytics), the latter focusing on information provision to government agencies, finance agencies, and management, rather than students, teachers, and faculties. We focused exclusively on learning analytics.

The field of learning analytics focuses on how data generated by digital learning environments, student systems, and other resources can be used to better understand, support, and improve students’ learning experiences. Learning analytics is more than collecting and analysing data. The purpose of learning analytics is to understand what happens during learning processes: to provide deeper insights so that better decisions based on facts can be taken – a data-driven approach to education.

Opportunities and goals

The use of learning analytics offers opportunities for TU Delft in several ways.

For example, learning analytics at the institutional level can contribute to better study support, coping with student population growth, and improving data literacy:

• Better study support
  By incorporating new sources of information, learning analytics can help to create more nuanced and richer images of the highly varied student population at the university, thereby creating a better overview of where and which supportive resources are most needed.
  
  
• Coping with student population growth
  The growth of university student populations brings new challenges to education. The use of learning analytics can help teachers handle this growing student population, for example, by analysing and identifying which individual students need additional attention, support, or information - or by supporting students through more personal feedback and learning systems, to ease the workload for the teacher.
  
  
• Improving data literacy
  Actively working with learning analytics and taking action based on analyses can help students and teachers in developing 'data literacy' – the ability to interpret and judge analysed data. This can encourage students to think more critically e.g. about how data is generally used, what form al consent means, and how algorithms work when combining data sets to create profiles of individuals. Supporting teaching staff in developing skills for working with learning analytics applications is an investment in institutional capacity and leadership.

At the curriculum and module level, learning analytics can help increase the quality of education:

• Increase the quality of education
  Learning analytics can be used to assess the effectiveness of the educational design of a subject or program. Analyses of student activity can be used as part of the course evaluation and course development process, or as a form of monitoring and feedback during the course run, for example. This can be used to take a step towards the data-driven design of education. Learning analytics can thus be used to increase educational quality and improve the student experience.

For students learning analytics enables personalised feedback to support better study progress:

• Personalised feedback
  Learning analytics can help in customising messages and support that students get from their institution, and provide more personalised feedback that helps students with self-reflection and their study planning.
  
  
• Better study progress
  In addition to more personalised feedback, learning analytics can help improve progress and retention, and promote satisfaction and well-being by helping students develop critical reflection skills and take responsibility for their own learning. Of the universities that already work with learning analytics, a significant number use learning analytics specifically to combat drop-outs and to increase retention.

Issues and potential risks

The use of learning analytics also has a number of issues and potential risks, which are explained below:

• Technological and policy developments
  Learning analytics is a relatively new discipline that is still developing. This means the field will change over time, introducing new best practices, proven technologies, and related educational approaches. In addition, there are many technological developments to be expected in the field of learning analytics, partly because the digital footprint of students and teachers is increasing. This means that this field will continue to evolve in the coming years. It is therefore crucial for an institution to continue learning and gaining experience with learning analytics, to keep up with this field and to make use of the right opportunities. High standards for consent and data protection can constitute barriers to experimentation, thereby inhibiting the learning ability of the organization. A balance must be struck here.
  
  
• Data-driven prejudices
  Another point of concern is the risk that data may contain prejudices, for example, about specific groups of students. Another potential risk is that insights into a student's individual study behaviour may cause potential prejudice in determining the final grade by the teacher – consciously or unconsciously. Transparency about how analyses are made can be a mitigating measure in this case, i.e. by allowing students and staff to review whether the analysis is correct, proper reservations with interpretation are applied, and what data is used.
  
  
• Data-driven performance culture
  A particular issue mentioned by experts and other institutions is the risk that insights into their own performance data may cause anxiety in students. Strikingly enough, at these institutions this issue a greater concern than a subject such as data protection. An educational institution must therefore carefully consider how the analysis outcomes are delivered to students and whether students have sufficient data literacy to use them.

From the conversations with other institutions it appears that a clear and focused strategy is needed for implementing learning analytics in education, with specific policy being a prerequisite. This includes involving students, teachers, and other stakeholders, in order to build a shared awareness and joint knowledge of learning analytics. Such a process must therefore be both policy-forming and educational. In the coming years pilots and experiences are expected to lead to policy changes at all interviewed institutions, as they continue to learn from the evaluation of the progress and impact of learning analytics within the institution.

POLICY FRAMEWORK

The policy framework consists of general policy principles for learning analytics, and policy principles for specific themes, including privacy and data protection, student consent, data usage and transparency, and stakeholder management.

General policy principles
With regards to using learning analytics at TU Delft the following general policy principles are proposed:

1. Learning analytics is a discipline with ethical dimensions, the application of which must be in line with the core values of TU Delft.
2. TU Delft uses learning analytics to provide targeted support for students during education and learning processes, and to improve the student experience.
3. TU Delft is transparent about how data is collected and used, where student permission is applicable, and where responsibility for ethical use of data lies.
4. TU Delft aims to keep model formation, algorithms, and interventions based on data analysis clear and free of bias.

Privacy and data protection
With regards to privacy and data protection when using learning analytics at TU Delft the following policy principles are proposed:

5. TU Delft operates within the legal framework of the law (including the Higher Education and Scientific Research Act and the Personal Data Protection Act) for the protection, storage, and processing of personal data. If, for some reason, this cannot be met immediately, the university acts as much as possible in the spirit of the law.

Student consent
With regards to student consent when using learning analytics at TU Delft the following policy principles are proposed:

6. TU Delft explicitly requests students to consent for the use of their personal data for applications of learning analytics.
7. Students have the opportunity to withdraw their given consent for the use of their personal data.

Data usage and transparency
With regards to data usage and transparency when using learning analytics at TU Delft the following policy principles are proposed:

8. TU Delft collects data on student study behaviour solely for a particular purpose, uses the collected data only for applications of learning analytics within that purpose, and informs students in advance about this use.
9. TU Delft limits access to and use of data: data sets used by analytical systems and analysts remain restricted to what is necessary for the specific application of learning analytics for which they are processed.
10. TU Delft is transparent about how data is stored and used, and makes this information available, for example via a website:
    1. Which student data is collected.
    2. For what purpose this data is collected.
    3. How this data is used for applications of learning analytics
11.  The TU Delft is also transparent about used methods and algorithms in applications of learning analytics, and makes this information available, for example via a website.

Stakeholder management
With regards to involving stakeholders in using learning analytics at TU Delft the following policy principles are proposed:

12. TU Delft involves representatives of staff and students in designing, developing, and evaluating the use of learning analytics and their governance.
13. For using learning analytics in education, TU Delft employs a governance structure aimed at ensuring that learning analytics projects and implementations are conducted ethically and in line with the strategy, policies, and values of the university.
14. TU Delft consults the Human Research Ethics Committee (HREC) with any new learning analytics pilot project for the ethical perspective, and takes this into account during the execution of the project.

Next steps

Five next steps are recommended which TU Delft can take to begin using the opportunities that learning analytics in education provides in a well-considered way:

1. Create a strategic project plan: Determine at the institutional level how to introduce learning analytics on-campus, and develop an implementation strategy for this.
2. Create an advisory board for learning analytics.
3. Work on transparency and consent.
4. Start with small projects to gain experience.
5. Increase knowledge and awareness about learning analytics within the organisation.

This document is just the beginning of Learning Analytics at the TU Delft. I think it is essential for a university to have such a policy in place, especially with the new privacy regulation GDPR.

References

• Van Staalduinen, J.P. (2017). Learning Analytics at TU Delft - Concept Vision and Strategic Framework. TU Delft internal document.