Learning Analytics: Notes on the Future (Athabasca U Course 2011

LearningAnalytics.net Open Course, Feb. 2011, (Athabasca U)

Learning Analytics:
Notes on the Future
Simon Buckingham Shum

Knowledge Media Institute
Open University UK
http://simon.buckinghamshum.net
http://open.edu

http://creativecommons.org/licenses/by-nc/2.0/uk 1

The lost key
One night a neighbor of Mullah Nasrudin was walking home and found
Mullah squatting on the ground beside a lamp post evidently looking
for something.
"What's the matter mullah?" asked the concerned neighbor. "I have lost
my keys" replied mullah
"Oh! Here let me help you" and the kindly neighbor got down on his knees
and started searching for Mullah's keys as well.
After some time spent looking the neighbor straightened up and quite
puzzled asked, “Are you sure you dropped your keys here?"
"Oh, I didn't drop them here" replied Mullah.
"Where did you drop them?!" exclaimed the now bewildered neighbor.
"Over there…" and Mullah pointed to the front of his house that was in
darkness.
"So why are you looking for them here??!!" shouted the angry neighbor.
"Because there is light here." replied mullah.
2

Ethics

Predictions
commoditization
institutionalization
academic shaping

Emerging
Examples
3

On the ethics of analytics,
and acting on them

Analogous to the ethics of constructing any simplifying
abstraction of reality, and acting on it: cf. economic
modelling, migration simulations, insurance risk…
— what can we learn from such fields?
4

Ethical Dilemmas

  Ensuring that what’s “good’ for the organization is not
bad for students/staff — or learning/real work
  learning is a mysterious process: beware the lamp-post
  institutions love to count stuff and demonstrate “impact”
  models only provide probabilities and averages, but in the hands of
people with a little knowledge…

  Ensuring that students are aware of and have
consented to the use of data
  in possible tension with our ethical duty to understand students, and
use that knowledge to maximise their success
  reporting student feedback in an unbiased way, protecting
confidentiality

5

Prediction 1:

Commoditization of analytics
services and tools

6

Social analytics start to become a commodity
service

http://www.mzinga.com/software/tour.asp 7

Social analytics start to become a commodity
service

http://www.mzinga.com/software/tour.asp 8

Organizational network analytics start to become
a commodity service

http://www.orgnet.com 9

Organizational network analytics start to become
a commodity service

http://www.orgnet.com 10

Commodity analytics/recommendation services?

  Browsing, discussing, tagging, friending,
following, shopping, rating, media
consumption…

  Because intense effort is going into these, the
associated analytics and recommendation
engines will become commodity services,
including open source and publicly available
algorithms

  There will be a value-added service industry to
help tune these to your needs
11

Prediction 2:

Embedding of institutional
analytics services and
diffusion of lessons learnt
from robust patterns

12

OU Analytics service: Predictive modelling

  Probability models help us to identify patterns of
success that vary between:
  student groups
  areas of curriculum
  study methods
  Previous OU study data – quantity and results – are the
best predictors of future success
  The results provide a more robust comparison of
module pass rates and support the institution in
identifying aspects of good performance that can be
shared and aspects where improvement could be
realised

OU Student Statistics & Surveys Team, Institute of Educational Technology 13

OU Analytics service: Effective Interventions

  Proactive measures targeted at specific points in the student
journey are associated with improved retention and progression

  Telephone contact with students considered to be potentially ‘at risk’
before the start of their first course is associated with around a 5%
improved likelihood of course completion.

  Additional tutor contact mid-way through a course is associated with
between 15% to 30% improved likelihood of course completion.

  Additional tutor contact around course results is associated with
between 10% to 25% improved likelihood of registering for a further
course.

  Contact with students intending to withdraw before course start is
associated with retaining 4% of students on their current course.

OU Student Support Review 14

OU Analytics service: Engaging faculties

  Various delivery methods:
  Self-guided presentations on a website
  Workshops and briefings
  Ad-hoc queries: available and approachable
  Graphics help non-experts use complex statistics

OU Student Statistics & Surveys Team, Institute of Educational Technology 15

Prediction 3:

Emergence of analytics and
recommendation engines
grounded in theories of
learning and sensemaking

16

Do we simply take what the vendors offer?

While we can gratefully reuse
generic web/business/social analytics
in educational and business institutions

— isn’t there anything special about…
learning
scholarship
sensemaking

? 17

Moreover, are many of us not also questioning
conventional definitions of “authentic learning”
and “scholarship”?...

  In learning/research/org-life, there are conventional
success indicators which are easy to measure
  course completions, passes, withdrawals…
  citations, grants, editorial boards, invited keynotes…
  no. customers, contracts secured, projects completed…

18

Moreover, are many of us not also questioning
conventional definitions of “authentic learning”
and “scholarship”?...
  In learning/research/org-life, there are conventional
success indicators which are easy to measure

  course completions, passes, withdrawals…
  citations, grants, editorial boards, invited keynotes…
  no. customers, contracts secured, projects completed…

  But the emerging “2.0” landscapes for learning,
scholarship and knowledge work, and new pedagogies,
demand new, more meaningful indicators

  social capital, critical questioning/reasoning, citizenship
values, habits of mind, resilience, collaboration skills,
creativity, emotional intelligence…
19

Raising our game

Lower level Higher level
events patterns

Investigate analytics that build on lower level events to define
higher level patterns tuned to the dimensions of learning/
scholarship/knowledge work which distinguish it from other activity

Raising our game

Lower level events Higher level patterns

People like you… People not like you in
particular ways…
(because you need to
be stretched out of
your comfort zone)

Raising our game


Other webinars with Other tutorials in which
matching the mentor played a
keywords… decreasing role and
newcomers played an
increasing role…

Video conference analytics (OU’s Flashmeeting)

23

Video conference foreign language tutorials:
Which mentor would you want to have?...
Mentor 1 Mentor 2
AV Chat AV Chat

1
Session

2

3

24

Speech-to-text transcript and semantic analysis

Ruben LAGATIE, Fridolin WILD, Patrick DE CAUSMAECKER & Peter SCOTT (2011).
Exposing Knowledge in Speech: Monitoring Conceptual Development in Spoken 25
Conversation. International Conference on ICT for Africa 2011, Mar 23-26 2011, Nigeria

Raising our game


Other blogs with Other scholars working
matching on the same open
keywords to your question that you
post… blogged about:

Villa, Mosaic, “Why Hadrian did not
Hadrian, Picts, invade Scotland
Romans sooner remains a
matter of debate”

Raising our game


Viewed 5 comments Challenged a peer’s
assumption with a
Replied to 3 good critical question
Posted 2 new ones Introduced a counter-
example

Socio-cultural discourse analysis (Mercer et al)

•  Disputational talk, characterised by disagreement and
individualised decision making.

•  Few attempts to pool resources, to offer constructive criticism or make
suggestions. Disputational talk also has some characteristic discourse
features - short exchanges consisting of assertions and challenges or
counter assertions ('Yes, it is.' 'No it's not!').

•  Cumulative talk, in which speakers build positively but uncritically
on what the others have said.

•  Partners use talk to construct a 'common knowledge' by accumulation.
Cumulative discourse is characterised by repetitions, confirmations and
elaborations.

Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
28

Socio-cultural discourse analysis (Mercer et al)

•  Exploratory talk, in which partners engage critically but
constructively with each other's ideas.
•  Statements and suggestions are offered for joint consideration.

•  These may be challenged and counter-challenged, but challenges are
justified and alternative hypotheses are offered.

•  Partners all actively participate and opinions are sought and considered
before decisions are jointly made.

•  Compared with the other two types, in Exploratory talk knowledge is made
more publicly accountable and reasoning is more visible in the talk.

Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
29

Structured deliberation and debate in which
Questions, Evidence and Connections are
first class entities (linkable, addressable, embeddable, contestable…)

30

Analyst-defined visual connection language

31

— node creation via web annotation


33


34

seeing the connections people make as
they annotate the web using Cohere

Visualizing all the connections that a
set of analysts have made
— but unfiltered, this may not be very
helpful

— semantic filtering of connections
Visualizing multiple learners’
interpretations of global
warming sources

Connections have been filtered
by a set of semantic
relationships grouped as
Consistency

De Liddo, A. and Buckingham Shum, S. (2010). Cohere: A prototype for contested collective intelligence. In: ACM Computer Supported Cooperative Work
(CSCW 2010) - Workshop: Collective Intelligence In Organizations, February 6-10, 2010, Savannah, Georgia, USA. http://oro.open.ac.uk/19554

— an agent reports a connection of interest

Concept Social
Network Network

Social
Discourse
Network

— discourse-centric analytics

De Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf.
Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff

Next step…

introducing automated analysis of discourse
moves which may signify deep/shallow learning
and knowledge construction

44

Analytics for identifying Exploratory talk
Elluminate sessions can It would be useful if we could
be very long – lasting for identify where learning seems to
hours or even covering be taking place, so we can
days of a conference recommend those sessions, and
not have to sit through online chat
about virtual biscuits

45

Analytics for identifying Exploratory talk

Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within 46
Synchronous Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff

Discourse analysis with Xerox Incremental Parser (XIP)
Detection of salient sentences based on rhetorical markers:
BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:
Recent studies indicate … ... new insights provide direct … little is known …
evidence ... … role … has been elusive
… the previously proposed …
... we suggest a new ... approach ... Current data is insufficient …
… is universally accepted ...
... results define a novel role ...
CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:
… unorthodox view resolves … studies ... have provided The goal of this study ...
paradoxes … important advances Here, we show ...
In contrast with previous Knowledge ... is crucial for ... Altogether, our results ...
hypotheses ... understanding indicate
... inconsistent with past valuable information ... from
findings ... studies

GENERALIZING: SURPRISE:
... emerging as a promising We have recently observed ...
approach surprisingly
Our understanding ... has grown We have identified ... unusual Ágnes Sándor & OLnet Project:
http://olnet.org/node/512
exponentially ... The recent discovery ... suggests
... growing recognition of the intriguing roles
importance ...

Human and machine annotation of literature

Document 1 19 sentences annotated 22 sentences annotated
11 sentences = human annotation

Document 2 71 sentences annotated 59 sentences annotated
42 sentences = human annotation

Ágnes Sándor & OLnet Project:
http://olnet.org/node/512

Raising our game


People like you who People like you who are
watched “Roman working on their
Britain” also critical thinking rated
viewed these these discussion
videos… groups as challenging
but supportive…

Raising our game


Failed 1 assignment Demonstrates
increased resilience
Passed 2 with merit when challenged
Passed 1 with Demonstrates ability to
distinction apply learning across
Graduated with contexts
honours Reports a growing
sense of herself as a
learner

Learning to Learn: 7 Dimensions of “Learning Power”
Factor analysis of the literature plus expert interviews: identified seven
dimensions of effective “learning power”, since validated empirically with
learners at many levels. (Deakin Crick, Broadfoot and Claxton, 2004)

Being Stuck & Static Changing & Learning
Data Accumulation Meaning Making
Passivity Critical Curiosity
Being Rule Bound Creativity
Isolation & Dependence Learning Relationships
Being Robotic Strategic Awareness
Fragility & Dependence Resilience

Professional development in schools, colleges and business:
ViTaL: http://www.vitalhub.net/vp_research-elli.htm

ELLI profile showing pre/post stretch following
mentoring and targetted intervention
ELLI: Effective Lifelong Learning Inventory (Ruth Deakin Crick, U. Bristol)
A web questionnaire generates a spider diagram summarising the learner’s
self-perception: the basis for a mentored discussion and strategic priorities

Changing and
learning

Critical Learning
Curiosity relationships

Meaning
Making Strategic
Awareness

Creativity Resilience

52
ViTaL: http://www.vitalhub.net/vp_research-elli.htm

EnquiryBlogger (Learning Futures programme)
Wordpress multisite plugins tuning it for learning
to learn and personalised enquiry

http://people.kmi.open.ac.uk/sbs/2011/01/digital-support-for-authentic-enquiry 53

The blobs on the EnquirySpiral and ELLI Spider plugins will change
size and colour when you categorise blog posts, as shown

Connecting

Choosing

54

(ELLI profiles can be
added to the blog using
the images plugin) 55

Users with teacher status can view the group’s
EnquiryBlogger plugins in their Dashboard

56

Users with teacher status can view the group’s
EnquiryBlogger plugins in their Dashboard

57

Raising our game


Students assigned Students assigned
to groups based based on predicted
on balancing vulnerability, in order
gender and age to balance mentor
workload, and match
mentor skillsets

Raising our game


In the week prior to Both successful and
every Assignment, failing students
forum posts and showed stressful
Helpdesk calls status updates, but the
spike successful ones
engaged in more
informal interaction,
while the failing
students stopped
talking to anyone

OU Facebook app: Tony Hirst, Liam Green-Hughes, Stuart Brown
http://apps.facebook.com/myoustory

My OU Story

60

EnquiryBlogger: Changing your “mood” in the
Mood View plugin prompts you to explain why,
which will then be added as a new blog post 61

future trajectories

62

Institutional Analytics: the future

Future trajectories we’re working on…

  Triangulating data sources to enrich context: e.g. staff skills, curriculum
design, student profile, real time usage, course feedback, CRM
transcript analysis… ( linked data)

  Just in time interventions using analysis of online engagement

  Greater personalisation using demographic details and previous study
history

  Multimedia indexing to detect the use of common images/videos

  Merging data from cloud applications to which the user grants access

  Data mining and recommendation engines around social learning

…balanced by the ethical principles around
data fusion, confidentiality, conflicting interests…

…and the intellectual challenge: does what we are measuring have
integrity as indicators of authentic learning?
64

Learning Analytics: Notes on the Future (Athabasca U Course 2011

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (19)

Similar to Learning Analytics: Notes on the Future (Athabasca U Course 2011

Similar to Learning Analytics: Notes on the Future (Athabasca U Course 2011 (20)

More from Simon Buckingham Shum

More from Simon Buckingham Shum (20)

Recently uploaded

Recently uploaded (20)

Learning Analytics: Notes on the Future (Athabasca U Course 2011