Research on the Use of Khan Academy in Schools

Research on the Use of

Khan Academy in Schools

Research Brief March 2014

Developed by SRI Educaon with funding from

the Bill & Melinda Gates Foundaon.

Although it originally focused on helping

individual users learn on their own, outside of

formal instuons, Khan Academy is now also

working closely with schools to explore ways

of transforming how instrucon is organized,

delivered, and experienced by both students

and teachers. This study examines how Khan

Academy and a group of California schools

collaborated to pilot innovave approaches

to teaching and learning in classrooms. These

approaches seek to accelerate, personalize,

and deepen student learning. The study also

examines how Khan Academy sought to use

its collaboraon with parcipang schools to

create a product that beer met both educator

and pupil needs.

Today teachers around the globe have access

to more online learning resources than ever

before, and more products are in the pipeline

as entry costs decline and public access to

high-speed bandwidth devices increases. With

an ever-growing array of choices, educators

need more and higher-quality informaon

about which products can work best in dierent

sengs and for dierent students, the factors

that support more eecve use in schools,

and the opportunity costs and challenges of

choosing one product over another.

While Khan Academy is sll used

predominantly by individuals for informal

study outside of schools, its use in schools and

other instuonal sengs has been growing.

In September 2011, the Bill & Melinda Gates

Foundaon contracted with SRI Internaonal

to study the development and use of Khan

Academy in K-12 schools.

Our two-year study involved nine sites, 20

schools, and more than 70 teachers over the

course of school year (SY) 2011-12 and SY

Introducon

In recent years, Khan Academy has become one of the most prominent pioneers

among a new generaon of digital learning organizaons in K-12 educaon. Khan

Academy had about 10 million unique users per month as of February 2014, up

from about 144,000 per month in early 2010. Around 65% of users are in the

United States. The sheer volume of the trac, as well as its rapid growth, indicates

the worldwide hunger for quality online instrucon, especially in math, and the

value that Khan Academy provides to its users.

Khan Academy is

working closely with

schools to explore

ways of transforming

how instrucon is

organized, delivered,

and experienced by

both students and

teachers.

This brief describes work done for the Bill & Melinda Gates Foundaon based on “Research on the Use

of Khan Academy in Schools,” Robert Murphy, Lawrence Gallagher, Andrew Krumm, Jessica Mislevy

and Amy Haer, Menlo Park, CA: SRI Internaonal. This research was funded by the Bill & Melinda

Gates Foundaon. The ndings and conclusions contained within are those of the authors and do not

necessarily reect posions or policies of the Bill & Melinda Gates Foundaon.

2012-13 to invesgate Khan Academy’s use as

a supplemental educaonal resource in the

classroom. All of the sites were volunteers that

elected to parcipate in the study. Four sites

parcipated both years, and the other ve

parcipated in the research for one year. Pilot

school sites were selected to represent a range

of governance structures and school types—

public, charter, and independent schools; and

elementary, middle, and high schools. Students

in grades 5 to 10 were the focus of the study

with around 2000 students parcipang in

each study year. A majority of the sites served

students from low-income communies, and

several were using Khan Academy specically

to support the math instrucon for students

with the greatest learning needs. One site,

a public elementary school district, had the

largest level of parcipaon, involving 8

schools, more than 50 teachers, and over a

1000 students. In the other sites, parcipaon

ranged from a single school and teacher, to

two to three schools and ve to six teachers.

Our report’s ndings are based on data

we collected from the nine research sites

during SY 2011-12 and SY 2012-13. We

collected data during site visits, including

classroom observaons; interviews with site

administrators; and interviews and informal

conversaons with teachers, parents, and

students. We also conducted teacher and

student surveys, and analyzed available

standardized test scores and user data

collected when students worked on Khan

Academy acvies.

A focus on implementaon rather than

impact on learning. During SY 2011-12 and

2012-13, SRI conducted an implementaon

study rather than an evaluaon of Khan

Academy’s impact on learning, given that

implementaon of Khan Academy varied

across sites, Khan Academy tools and

resources evolved over me, and schools,

Khan Academy, and researchers collaborated

regularly. Ongoing communicaons and

ght relaonships between the parcipang

teachers, students, and school leaders with

Khan Academy and SRI Internaonal resulted

in both a more useful tool for teachers and

students, and a beer understanding of the

various ways Khan Academy can be used in

school sengs to promote student learning.

During the study, Khan Academy worked

with schools parcipang in the study to

update and rene its tools and resources.

Simultaneously, teachers and students were

using Khan Academy tools and resources in

considerably dierent ways across the nine

study sites, and some of the sites also changed

the ways they used it during the course of

the two-year study. For these reasons, it

was methodologically unsound to conduct a

rigorous evaluaon of Khan Academy’s impact

on learning during the study period, including

any use of randomized control trials, which

would have required Khan Academy tools and

resources to remain unchanged during the

study and for teachers and students to use

Khan Academy the same way. Moreover, at

all but one of the sites, Khan Academy was

principally used as a supplementary tool—not

as the core primary curriculum—so the eects

of Khan Academy cannot be separated from

those contributed by other elements of the

math curriculum.

Our two-year study

involved nine sites,

20 schools, and more

than 70 teachers

to invesgate Khan

Academy’s use as

a supplemental

educaonal resource

in the classroom.

Report Overview

This research brief describes Khan Academy

resources and funcons; provides examples of

how three of the nine study sites used Khan

Academy; and summarizes what educators and

students found both helpful and challenging

about using Khan Academy, and how Khan

Academy responded to that feedback. It

concludes by briey idenfying issues that

Khan Academy and other developers of

digital products for schools should consider

as they look to the future. More detailed

informaon about the diversity of models

used at the school sites; factors facilitang the

implementaon of Khan Academy; how Khan

Academy’s design evolved during the study;

ndings about the impact Khan Academy

had on supporng teaching and learning;

and implicaons for further research, are

available in the study’s implementaon report,

“Research on the Use of Khan Academy in

Schools”, which is available at hps://www.

sri.com/work/projects/research-use-khan-

academy-schools.

About Khan Academy

Founded in 2006, the nonprot Khan Academy is

now one of the globe’s most popular educaon

websites, and describes its mission as providing “a

free world-class educaon for anyone, anywhere.”

Khan Academy oers more than 5,500

instruconal videos—of which approximately

3,500 are about math—and they connue to

be extremely popular. However, Khan Academy

is more than a massive repository of teaching

videos; it also oers more than 100,000 pracce

math problems that students can complete at

their own pace. In 2013 alone, users worked on

more than 700 million problems. Khan Academy

has expanded its inial focus on math, economics,

and science to include other subjects such as art

history and entrepreneurship. Addionally, Khan

Academy is now partnering with a diverse array of

instuons including museums, universies, and

thinktanks.

Khan Academy also recently introduced

addional “coaching” features developed with

schools in mind but also available to home

users for informal instrucon. The coaching

secon includes materials to guide teachers,

tutors, parents, and others in how to use Khan

Academy to meet their students’ learning

goals. It allows these users to assign specic

material to their students, automacally

sends students electronic alerts about their

new assignments, and provides dashboards

allowing teachers and students to monitor

student progress.

For more informaon about Khan

Academy, please see:

hp://www.khanacademy.org/about

hp://khanacademy.desk.com/customer/

portal/arcles/441307-press-room

How did the study sites use

Khan Academy?

Variaon in how schools and teachers

implemented Khan Academy across the nine

sites we studied was an important aspect of

this research, aecng what each site learned

as a result of its experience. Parcipang

sites used Khan Academy’s resources in many

ways and for quite dierent purposes—as an

addional pracce tool, as an intervenon

for students who had fallen behind, as an

enrichment acvity for advanced students, and

as an accountability tool to monitor student

progress. Each school found varying ways

to employ the Khan Academy instruconal

resources, with its choices relang to both its

instuonal goals and the goals of individual

teachers within the school.

Each school began working with Khan Academy

at a dierent starng point, including varying

levels of teacher training and experience,

student and teacher comfort in the use of

technology, and access to computers. Use in

some schools evolved as students and teachers

became more at ease with using Khan Academy

or gained access to more computers in their

classrooms. Some teachers materially adapted

their use of Khan Academy as they learned

more about it. Use paerns also evolved as

developers added new features and content

in response to ongoing communicaons with

educators. All of this variaon aected student

and teacher interacon with Khan Academy,

and in turn each implementaon model

oered diering benets and posed diering

challenges.

Even though Khan Academy is primarily known

for its video library and has been associated with

the “ipped” classroom model (i.e., teachers

assign students videos about new concepts

to watch as homework, and use class me to

extend the video lectures with discussion and

interacve acvies), teachers parcipang in

the research were more focused on exploring

how online, personalized pracce opportunies

for students could be incorporated into their

exisng instruconal acvies. In most cases,

when students used the videos, they did so in

class to review concepts as they worked through

the Khan Academy problem sets. Few teachers

used the videos in their lessons to introduce new

concepts and skills.

Students primarily used Khan Academy problem

sets to pracce and rene skills individually

and with classmates, while geng immediate

feedback that resulted in a range of observed

outcomes across the research sites including:

• Learning new math skills

• Filling in gaps in learning and shoring up

weak spots from past instrucon

• Tracking and monitoring school work to

hold themselves accountable for their

performance

• Spending more me in peer teaching and

collaborave work with their classmates

• Receiving more opportunies to direct their

own learning, and

• Allowing teachers to spend more me

assisng individual students or small groups

of students.

The following pages provide examples of

how three of the sites we studied used Khan

Academy. They showcase three contrasng

models of how schools are using this technology

to support core instrucon. Addional details

about these models and descripons of Khan

Academy use in the other six sites are discussed

in the study’s implementaon report.

Parcipang sites

used Khan Academy’s

resources for

dierent purposes—

a pracce tool, an

intervenon, an

enrichment acvity,

and an accountability

tool to monitor

student progress.

Site 2: Using Khan Academy

to facilitate self-directed

learning

The site: We examined an innovave 9th and

10th grade math program in these two small

charter high schools that opened in fall 2011.

The schools are collocated in a neighborhood

where 45% of students qualied for free or

reduced-price lunches. These trailblazing

schools emphasize a self-paced learning

model, in which students take on signicantly

more personal responsibility for direcng and

managing their own learning than is typical at a

tradional school. Technology plays a signicant

role in facilitang this approach.

How it used Khan Academy: This site was

by far the most disncve of the nine sites

parcipang in the study because of its

emphasis on preparing students for college

using a self-paced, self-directed learning

model. The site’s mission focused on helping

students learn how to set learning goals, hold

themselves responsible for meeng those goals,

and evaluate their own progress connuously

along the way. More broadly, these schools

encourage students to learn how to advocate

for themselves as learners. Its math program

was designed to support students’ simultaneous

development of content knowledge, academic

skills, and crical non-cognive skills.

During the rst year of the study, when use of

Khan Academy was mandated for all students,

the schools had the highest use levels among

study sites, with students spending about 22%

of their instruconal me on Khan Academy

acvies, compared to less than 10% at all of

the other sites. In the second year, this site’s

students used Khan Academy for fewer hours

than the rst year, because the schools changed

their math instrucon model to an enrely

self-paced, self-directed approach, and Khan

Academy use was now enrely at the discreon

of students. While students used it for shorter

periods of me during the second year of the

study, they used it in more independent and

innovave ways. This approach was more

ghtly aligned with the site’s mission, and may

be helpful to schools invesgang how to use

technology to personalize learning and make it

more student-directed.

Khan Academy was the primary resource used

in the schools’ heavily personalized learning

environment for math instrucon. Math took

place during a 2-hour me block with up to

200 students from two grades all meeng in

one large open classroom. Students spent a

signicant poron of the math block working

on their own or in small groups, oen at

laptops. Student conversaons were frequent,

but much of their talk was about math, about

nding parcular learning resources, or about

navigang the digital learning environment.

Teachers primarily served as facilitators and

sources of support, with far less emphasis on

tradional forms of direct instrucon such as

lecturing at the front of the room. Instead,

teachers and “learning coaches” met with

students as needed in the large classroom, as

well as in smaller sengs in adjoining rooms. By

spring, the site had divided the two-hour math

block into one hour that was student-directed

and one that was teacher-directed. During the

student-directed me, the students had access

to two teachers or adult volunteers with math

backgrounds who answered student quesons

and provided tutoring.

Teachers developed digital “playlists” to help

students direct their learning me. A playlist was

a set of links to recommended online acvies

and educaonal resources that were directly

related to the topics students were studying. The

amount of me it took a student to complete

playlist materials could vary from a few hours to

several days, depending both on the topic and

on the student’s pace of work. The playlists were

not limited to Khan Academy materials, but our

observaons and interviews indicated that Khan

Academy was the most frequently used resource,

parcularly the problem sets. A student’s playlist

might include Khan Academy videos and problem

sets, as well as other digital resources, including

other websites similar to Khan Academy, online

textbooks (which include readings, videos,

and problem sets), PDFs of teacher-created

worksheets, and interacve graphing calculators.

Each student working on a topic was assigned

the same playlist, but individual students chose

their own pathways through materials. Teachers

did not mandate the use of any materials,

nor did they spulate a specic order or

pathway for using them. Students could work

on whichever acvies and use any resources

they found helpful, selecng as many or as

few as they needed. Students could idenfy

addional materials related to the topic on

their own and work on those if they preferred.

Whenever students felt ready, they could take

a school-created test on the learning objecve.

Once they answered four out of ve quesons

correctly, they were deemed ready to move on

to the next topic.

Time was set aside during self-directed work

me (one hour of a two-hour instrucon block)

for students to idenfy their learning goals

and plan what they needed to do to meet

them, as well as to reect on their progress

toward those goals. Students spent the rst 10

minutes of each self-directed session planning

how to use their class me and idenfying

which digital or other resources they needed

to meet their specic objecves for that day.

Aer they nished their work, they spent 10

minutes reecng on what they had learned,

including wring about that day’s learning

experience, compleng a survey, or otherwise

self-evaluang their progress. By the second

semester, students who were not making

adequate progress were required to ll out

forms that described their step-by-step plans for

catching up.

Lessons the school learned during the study:

The high use of Khan Academy at Site 2 in the

rst year, SY 2011-12, relave to the other

sites and study years was supported by several

factors: (1) anyme access to one-to-one

compung (one computer for each student) in

classrooms, (2) mandated compleon of Khan

Academy goals with consequences for failure to

do so, (3) close teacher monitoring of progress

toward goals, (4) a well-planned integraon

with the core curriculum and (5) extended

instruconal blocks (90 minutes dedicated to

daily math instrucon).

In the second year, the site wanted to focus on

its goal of building students’ ability to direct

their own learning in other areas in addion to

mathemacs. By giving students more control

over what resources they used and for how

long, the schools learned that some students

needed addional support to do this kind of

independent work. As a result of the site’s

experiences using Khan Academy technology

to facilitate this model, it is currently working

to idenfy the best ways to serve those

students who need more help in self-direcng

their own learning, including increasing the

amount of face-to-face me teachers have

with students, and rening its student data

management system to allow sta and students

to beer monitor progress against curriculum

benchmarks, and to alert sta sooner if students

are falling behind.

This site is sll experimenng, and its

unconvenonal learning structures are very

much a work in progress, but it is planning to

scale and spread the self-directed, self-paced

learning model to other schools in its network.

Site 8: Using Khan Academy

to support teacher-led

whole-class instrucon

The site: Site 8 is a high school serving a

predominantly high-poverty populaon. More

than 80% of its students qualify for free or

reduced-price lunch, and 75% come from

families whose primary language is not English.

The school’s sta felt that Khan Academy

could help meet their urgent need to help

students catch up, given the signicant number

of students who were several grade levels

behind in math upon entering the school in 9th

grade. These gaps in crical skills were holding

students back and making learning grade-level

content extremely dicult.

How it used Khan Academy: Khan Academy

was used on a daily basis by two educators: one

teaching an algebra readiness class for ninth

graders and a learning lab class, and the other

teaching a mixed 9th and 10th grade Algebra

1 class, a mixed 9th and 10th grade geometry

class, and a 10th grade Algebra 2 class.

The school primarily used Khan Academy

to support teacher-directed, whole-class

instrucon. During the daily math class, all

students focused on the same pracce problem

sets at the same me. Rather than using

tradional worksheets, however, they worked

online using Khan Academy exercises. This

approach may be helpful to schools exploring

how to use technology to give students more

opportunies to pracce their math skills

in a way that is integrated into the exisng

curriculum and also reinforces their teacher-led

lessons. The two teachers at this site primarily

employed Khan Academy as a resource to help

students devote more structured, producve

me to pracce acvies designed to help them

ll in gaps in their math knowledge and skills

and reinforce skills covered by the teacher in

their daily lesson.

The school’s educaonal vision emphasizes

building students’ self-discipline, sense of

individual responsibility, and overall work ethic.

Teachers viewed Khan Academy as a praccal

tool that could help students hold themselves

accountable for staying on top of their

assigned work. Both the me students spent

working on Khan Academy and the progress

they made were automacally archived and

easily monitored by teachers in real me. The

students could also view the informaon easily.

Teachers could use the data to show their

students the tangible academic results that

occurred when they completed their work—

as well as what happened when they did not

complete their work or spend enough me on a

parcular task.

By providing a digital plaorm that makes

it easier to document and analyze student

performance on specic assignments over me,

Khan Academy helped teachers pinpoint not

just whether students had completed assigned

problem sets, but also whether and how well

they were mastering specic math skills such as

factoring or graphing linear funcons. This made

it easier for both students and their teachers

to visualize and track academic progress. In

turn, the data helped students make a clearer

connecon between doing their work and

improved academic performance.

Students also knew that there were specic

consequences for not nishing assigned work.

For example, if students did not complete

their Khan Academy problem sets, they had

to remain aer school to nish them, thus

receiving addional movaon to stay on task

during the school day. As one of the teachers

said, even “if students don’t care about their

grades and they’re happy with a D, they do care

about their me.” In addion, Khan Academy

provided students more informaon about

their performance, which helped students

idenfy precisely what they needed to do to

make progress. This combinaon of direct

consequences with beer informaon helped

steer students in the right direcon.

Lessons the school learned during the study:

The school found that tangible reporng of

results brought home to students the direct

connecon between pung more me into

their work and posive results in their math

performance.

In addion, preliminary evidence suggests

that the changes made to the school’s math

instrucon, including the use of Khan Academy,

were associated with improved test scores.

For a group of 9th grade Algebra I students

who rst began using Khan Academy in SY

2011-12 in a summer academic program and

connued to use it as 10th graders within their

geometry class, we found moderate to large,

stascally signicant dierences in spring test

scores compared to students who aended the

school before the change in math instrucon.

While it is not possible to separate out the

eect of the use of Khan Academy from other

changes made to the school’s math program,

Khan Academy is clearly playing a central role

in what appears to be a very eecve approach

to improving student achievement through

improving student accountability. Addional

details regarding this analysis can be found in

the study’s implementaon report.

Site 9: Using Khan Academy

to facilitate small-group

instrucon

The site: This school opened several years ago

with grades 6 to 8, and will eventually oer

grades 6 to 12. The school’s mission emphasizes

academic preparaon for higher educaon

because many of its students are seeking to

be the rst in their families to aend college.

It serves a predominately low-income, Lano

populaon. This prole highlights a model of

Khan Academy use implemented by the school’s

6th grade math teacher.

How it used Khan Academy: This middle

school used a “rotaon” model for its 6th

graders, largely because it did not own enough

computers for each student to have one. There

were only about a dozen netbooks—inexpensive

laptop computers designed for Internet access

and wireless communicaon—available for

a classroom of about 25 to 30 students. The

school devoted a 2-hour daily block of me

for 6th grade math instrucon, which typically

began with a 20-minute whole-class warm-

up period consisng of announcements and

“mental” math exercises (problem solving

without calculator or pen-and-paper). The class

then divided into three groups of 8 to 9 students

for the rotaon staons. Each group spent

about 30 minutes at each of the three staons,

one of which was Khan Academy. At a second

staon, students worked in a small group with

a teacher who gave a mini-lesson, and at the

third students pracced independently on math

worksheets or took an assessment. One day of

class each week was reserved for whole-group

instrucon, acvies, or tesng.

The school’s 6th grade math teacher used Khan

Academy to adapt her instrucon to beer

meet the needs of individual students. The

teacher assigned students to groups based on

whether they were performing below, at, or

above grade level in math. The teacher would

alter her lesson accordingly. Khan Academy

data was one informaon source she used for

making these decisions, and she also used the

Khan Academy data reports throughout the year

to monitor students’ performance and progress.

This school employed Khan Academy largely as

a supplemental instruconal acvity, but its use

was integrated into the weekly math lessons.

At the start of the Khan Academy staon me,

students would log on to the online classroom

management portal Edmodo to view the

problem sets the teacher had posted for them.

Each week, the teacher assigned students

about three Khan Academy problem sets to

solve. Usually, two problem sets focused on the

specic math topics being taught that week.

The third problem set varied, depending on the

level at which a student was performing, and

whether that student needed more pracce in

specic areas.

The Khan Academy staons enabled students

to work at their own pace; students could easily

skip content on which they had demonstrated

mastery and move on to new material. Or, if

they were struggling with a concept and got

stuck, they could use the “hints” feature to

reveal, one step at a me, a sample approach

to solving the problem. The value created by

this immediate support contrasts with the

experience of compleng a pencil-and-paper

worksheet: if students hand in a worksheet

with many blanks or errors, they may wait a

day or more before receiving feedback or extra

help. Conversely, with Khan Academy, students

learned right away whether their soluon was

correct and used the hints feature to clear up

confusion and get themselves back on track

rapidly and independently, without the need

to wait for a teacher. This me-saving measure

helped facilitate the eciency of the small

group rotaon model.

By the second half of the year, the school had

acquired addional netbook computers so that

it had enough for half the class. At that stage,

the teachers reduced the number of rotaon

staons from three to two: half the students

worked individually on Khan Academy acvies

at the computers, and the other half worked

oine in a group with the teacher. The groups

traded places aer about 45 minutes.

This school’s approach can serve as an example

for schools that do not have enough computers

available for individual use in a class. Even

schools with one-to-one technology could use

this approach to create me for the teacher

to work with small groups while part of the

class is producvely occupied with other math

assignments on the computers.

Lessons the school learned during the study:

At this school, teachers found a rotaon model

to be a producve way to use Khan Academy

without having one-to-one compung. This

was facilitated by established rounes that

ensured the smooth moving of students from

their small group me with the teacher, to their

independent me working on Khan Academy.

The school planned to adopt this rotaon model

across grades 6 to 8.

Teachers at this school singled out the

immediate feedback students received as the

most important benet of Khan Academy. The

school was also considering supplemenng

Khan Academy with other online resources

to provide students with a variety of digital

learning experiences.

Summary of Findings

While it is sll too early to expect

transformave change at this early stage—

given teachers’ lack of experience in

integrang digital resources eecvely in the

classroom and the absence of broadly adopted

research-based implementaon models—

our research has enhanced understanding

about what schools and teachers will need

for eecve use of emerging personalized

learning technologies such as Khan Academy.

This includes how developers organize and

align digital content for use in the classroom,

and what kind of training and other supports

teachers need to help them integrate the

technologies into their daily lessons.

A summary of key research ndings across

all sites follows. More detailed informaon

about the ndings is available in the study’s

implementaon report “Research on the Use

of Khan Academy in Schools” at hps://www.

sri.com/work/projects/research-use-khan-

academy-schools.

Benets of Khan Academy

Use for Teaching and

Learning

Student Use and Percepons

• Student percepons of their me spent on

Khan Academy was highly posive. Overall

71% of students reported that they enjoyed

using Khan Academy, and 32% agreed they

liked math more since they started using

Khan Academy.

• Students’ engagement level was generally

high during Khan Academy sessions. A high

level of engagement was evident during a

majority of our classroom observaons for all

grade levels. In focus groups with students in

the lower grade levels, they oen commented

that they enjoyed their “Khan me,” and

the teachers we interviewed and surveyed

conrmed that atude. In SY 2012-13, 8 in

10 teachers surveyed reported that students

liked the me they spent working on Khan

Academy and, across all grade levels, that

students were moderately (62%) or highly

(25%) engaged when using Khan Academy.

• Students perceived that use of

Khan Academy encouraged greater

independence in learning. Immediate

feedback, hints, and access to videos meant

that if students were struggling with a

parcular problem in Khan Academy, they

were not stuck for long and could experience

success even when the content became

challenging: 45% of student respondents

said that with Khan Academy they were able

to learn new things about math on their own

without the help of a teacher.

Overall 71% of

students reported

that they enjoyed

using Khan

Academy, and 32%

agreed they liked

math more since

they started using

Khan Academy.

• The amount of me students spent working

on Khan Academy varied considerably

across and within sites, and also by school

year.

Student use (measured as viewing

videos and working on problems) ranged

from a low of 11 minutes per week for the

median student at Site 1 in SY 2012-13, to

a high of 90 minutes per week at Site 2 in

SY 2011-12. Few teachers expected their

students to use Khan Academy outside of

the regular school day. Median student use

outside of the school day ranged from a low

of a few minutes a week at several schools,

to as high as 25 minutes per week at Site 8.

Teacher Use and Percepons

• A majority of teachers were happy with

their Khan Academy experience and

planned to use the Academy with their

students in the upcoming school year.

Among the parcipang teachers, 86%

would recommend it to other teachers, and

89% planned to use it with their students

during the next school year. Educators

reported that the fact that the resource

was free was an important factor in their

decisions to use Khan Academy, given limited

budget resources for new technologies.

Two other key factors facilitang Khan

Academy use were the availability of one-to-

one access to computers in the classroom,

and the allocaon of extended me for

math instrucon. (Four of the nine sites

dedicated 90 minutes or more a day to math

instrucon.)

The phrase “working on Khan Academy” refers to the total me

students spent in watching videos or working on problem sets. It

excludes the me students spent on the website logging in and

out or engaging in other acvies (e.g., updang their prole

page, searching for content). This represents a measure of the

amount of me students spent acvely engaged in a direct

instruconal experience on the website.

• Teacher percepon of Khan Academy’s

impact on students varied across dierent

learning areas. Teachers reported the

strongest impacts on students’ overall

understanding of math topics taught, ability

to work and learn independently, and

acquision of procedural skills. A strong

majority of teachers, 85%, reported that the

use of Khan Academy had posively aected

their students’ learning and understanding

of the material overall: 37% reported a

“strong posive impact” on student learning

and understanding, and 48% reported a

“somewhat posive impact.”

• Teachers reported that integrang Khan

Academy into their instrucon has

increased their capacity to support their

students in a number of areas. Across

the two study years, the majority (91%) of

teachers indicated that using Khan Academy

increased their ability to provide students

with opportunies to pracce new concepts

and skills they had recently learned in class.

About eight in ten teachers also reported

that Khan Academy increased their ability

to monitor students’ knowledge and ability,

thus helping to idenfy students who were

struggling or ahead of the rest of the class.

Similarly, eight in ten teachers said it helped

them expose advanced students to concepts

beyond their grade level, while nearly two-

thirds of teachers reported that it increased

their capacity to help students who were

behind to catch up.

• Teachers who viewed the Khan Academy

reports regularly found them useful. Slightly

more than half of the teachers (56%) said

they reviewed the student performance

data at least once a week. But about four in

ten parcipang teachers at the pilot sites

said they reviewed the data about once a

Student use ranged

from a low of 11

minutes per week

for the median

student at Site 1 in

SY 2012-13, to a high

of 90 minutes per

week at Site 2 in SY

2011-12.

Among parcipang

teachers, 86% would

recommend it to

other teachers, and

89% planned to use

it with their students

during the next

school year.

month or less or not at all. Of the teachers

who reviewed the data a few mes a month

or more oen, slightly more than half

(51%) said it was “very useful,” in informing

instrucon and the remaining 49% found it

“somewhat useful.”

Student Outcomes

• A posive associaon was found between

more Khan Academy use and more problem

sets completed and two outcomes: (1)

improvements in student test scores, and

(2) improvements in three of the four self-

reported nonachievement outcomes – math

anxiety, math self-concept, and academic

ecacy (i.e., belief in one’s ability to

succeed in academic endeavors). Evidence

from exploratory analyses of data available

from two sites suggested that students who

spent more me on Khan Academy and

successfully completed more Khan Academy

problem sets to prociency experienced

more posive than expected outcomes in

terms of math test scores, reduced math

anxiety, and had higher condence in their

ability to do math. Prior achievement tests

and atude measures taken in the fall were

used to predict end-of-year outcomes for all

students, and then the sample was split into

those whose end-of-year scores were beer

than predicted, and those whose end of year

scores were less posive than expected.

– At Site 1, we found that h graders

with beer than predicted end-of-year

achievement test scores had spent an

extra 12 hours over the school year using

Khan Academy and that sixth graders

exceeding their predicted achievement

level had spent an extra 3 hours on Khan

Academy, compared to grade-level peers

with lower-than-expected end-of-year test

scores. Similarly, h graders with higher

than expected achievement test scores

had completed 26 addional problem

sets (39% more), and sixth graders with

higher than expected achievement had

completed 20 addional problem sets (or

22% more).

– Likewise, students at Site 1 who reported

lower than expected anxiety about doing

math in the spring compared to their

reports in the fall, higher than expected

beliefs about their own math ability (math

self-concept), and greater condence

in their ability to learn math even when

the concepts become dicult (academic

ecacy), had completed 10% to 20% more

Khan Academy problem sets than their

grade-level peers who showed less posive

than expected atudes in these areas.

– These analyses are exploratory, and

the results preliminary; they cannot be

used to make denive claims about the

eecveness of Khan Academy resources.

Other plausible explanaons could account

for these associaons that the analyses

did not consider. Although the results are

not denive, they do suggest associaons

that are worthy of future invesgaon

using more rigorous designs to beer

understand the potenal ecacy of the

use of Khan Academy in the classroom.

Exploratory

analyses in two sites

suggested: students

who spent more me

on Khan Academy

and successfully

completed problem

sets experienced

more posive than

expected outcomes

in terms of math

test scores, reduced

math anxiety,

and had higher

condence in their

ability to do math.

What did the educators at

the study sites nd helpful

about using Khan Academy

in the classroom?

Over the course of the study, educators at the

nine study sites shared their thoughts about

the aspects of Khan Academy they found most

benecial in a classroom seng:

• Using modular problem sets to provide

addional opportunies for students to

pracce math skills: Teachers told us that

they liked the modular nature of the videos

and problem sets. Because the materials

were not presented in a predetermined

sequence, teachers could use them

separately and independently, arranging the

materials as they needed into customized

playlists. Rather than following a set order,

students could focus their aenon on areas

they needed to pracce more, or skip ahead

if they were ready to move on.

• Facilitated dierenaon: The modular

aspect of Khan Academy resources also

helped facilitate dierenaon in instrucon:

during a class period, dierent students could

focus on dierent skills at the same me.

• Rapid feedback: The virtually instantaneous

nature of the feedback provided to students

while working on the problem sets was

aracve to educators as well as students.

The feedback was provided much faster than

the me required for a teacher to grade and

return a set of textbook problems, or even

the me associated with students trading

papers to check each others’ work. Many

educators reported that it was this aspect of

Khan Academy they found most valuable.

• Self-directed learning: Khan Academy gave

students the opportunity to self-direct

their learning, and build condence in their

ability to work and learn independently.

Students could also use the expedited nature

of the feedback to inform and drive their

understanding, enabling them to assert more

control and ownership over the direcon of

their learning.

What challenges did

educators using Khan

Academy encounter?

Although many aspects of Khan Academy

were helpful, teachers did encounter some

challenges in classroom use, and some

features were underused.

• Alignment of content to grade-level

curriculum: Lack of alignment of Khan

Academy content with core curriculum

posed a signicant challenge for integrang

Khan Academy into the classroom. Two-

thirds of teachers surveyed across study

years reported that a lack of alignment

between the Khan Academy resources and

their school’s curriculum had a moderate

to signicant negave eect on their ability

to use Khan Academy eecvely with their

students. From a formal school curriculum

perspecve, content gaps existed in both

the videos and the problem sets during the

rst year of the study, and to a lesser extent

during the second year as well.

• Organizaon of the content: Teachers need

online content that is curated, assignable,

and clearly mapped to grade-level content

The virtually

instantaneous nature

of the feedback

provided to students

while working on

the problem sets

was aracve to

educators as well as

students.

standards. The Internet oers a vast amount

of digital content, of which Khan Academy is

just one resource, but teachers do not have

me to curate all of this content. During

most of the SY 2011-12 school year, many

teachers expressed diculty locang the

videos and problem sets that were relevant

for their upcoming lessons and were at the

appropriate level for their students. During

the study period, Khan Academy also lacked

an assignability or “recommendaon”

funcon. Teachers could not easily specify

the Khan Academy content they wanted

students to work on, such as topics recently

covered in a lesson or skills that students

needed to develop or improve.

• Data reporng funcons: Khan Academy’s

data reports are intended to give teachers

beer informaon to help improve how

they monitor both individual and class

understanding, adapt their instrucon

accordingly, and also provide beer feedback

and support to students. These data funcons

are being promoted as one of the primary

benets of these new technologies, but these

features are being relavely underused, both

in this study and our other studies on blended

learning. As noted above, during our two-year

study period, while a majority of teachers are

reviewing student performance data captured

by the system on a regular basis, about four in

ten of the teachers parcipang in the study

reviewed the Khan Academy student data

infrequently (once a month or less) or not at

all. Among those who never reviewed the data

or who reviewed it infrequently, 70% indicated

that they did not use the reports more

frequently because they relied on informaon

outside the system to gauge student progress,

such as their own observaons and formave

assessments.

As a result of these challenges,

what product features did

Khan Academy change?

In response to feedback from the educators at

the study sites, Khan Academy implemented

a wide array of changes, adding or adapng

features to facilitate the product’s use in the

classroom.

In July 2013, Khan Academy launched a major

redesign of its website with the release of its

grade-level “missions” and a new “learning

ow” and “learning dashboard.” The redesign

was introduced to help students focus on

working in the appropriate content area. From

their dashboard, students can select a mission,

and while in the mission, the system will only

display the videos and problem sets mapped

to a single grade level (e.g., the Grade Six

Mission) or course content (Algebra I) along

with videos and problem sets associated with

any prerequisite skills.

Khan Academy also created considerable new

videos and problem sets specically designed

for each grade level, mapped content to the

Common Core State Standards (full coverage

of the K-12 standards is currently planned for

fall 2014), added search capabilies, created

a problem set browser to help teachers

nd aligned content, and, by the end of SY

2011-12, reorganized its math content into

“tutorials” intended to contain related videos

and problem sets that teachers can use or

modify to support an instruconal unit on an

important topic. It also gave users the ability

to slow down and fast forward videos through

playback, to make it easier for them to digest

the content.

Two-thirds of

teachers surveyed

across study years

reported that a

lack of alignment

between the Khan

Academy resources

and their school’s

curriculum had

a moderate to

signicant negave

eect on their ability

to use Khan Academy

eecvely with their

students.

Khan Academy also revised its teacher reports

to provide more simplied, customizable

summaries of student data at both the class

and individual levels, and created the ability for

teachers to recommend specic Khan Academy

content to their students. To ease the burden on

teachers, Khan Academy now emails student

progress reports to teachers.

For students, Khan Academy added a goal-

seng feature, and other tools to help visualize

progress within the new grade-level missions.

Summary and Implicaons

Our implementaon study focused on

understanding and documenng the models

that educators are experimenng with as they

use Khan Academy’s math resources in schools;

considering the benets and challenges of doing

so; and examining how this can inform future

decisions about whether and how to adopt

Khan Academy for classroom use.

Overall, the study idened many posive

ndings relevant to educators, developers, and

educaon leaders, and shows that the schools

serving diverse student populaons can

make use of Khan Academy as a component

in their mathemacs instrucon. Teachers

in the pilot schools reported that they found

value in using Khan Academy to support their

instrucon, that it helped their students, and

that they planned to connue to experiment

with dierent models for integrang Khan

Academy into their math curricula. Students

also indicated in focus groups and on surveys

that they liked using Khan Academy. In

addion, early evidence from one site suggests

that a math instrucon approach using Khan

Academy in combinaon with close teacher

monitoring and extended periods for math

instrucon can improve student learning.

For now, these preliminary ndings should

be interpreted cauously and should not be

over-generalized. No single implementaon

model was used across all the sites, and Khan

Academy was not used as the sole, or even

primary source of math instrucon at most

sites, making it dicult to isolate its eects.

Teachers and schools leaders are aracted to

using Khan Academy because it is available

for free, oers a modular set of resources,

engages students, provides immediate

feedback, and oers opportunies for students

to direct their own learning. At the same me,

fundamental challenges remain that constrain

how schools use Khan Academy. These

challenges include accountability pressures,

restricons on instruconal me, and limited

access to one-to-one compung.

Taken together, the study’s ndings have

several broader implicaons:

• The Khan Academy school pilot program

demonstrated that a collaborave

partnership between a lean technology

startup and schools can result in substanve

feedback from students and teachers which

in turn leads to beer learning and teaching

experiences and improved digital educaon

oerings.

• Schools and teachers adopng Khan

Academy can benet from detailed use

cases, describing how Khan Academy can

be implemented under dierent me and

technology constraints and with dierent

instruconal goals.

Our learnings in this

study made it clear

that the teacher’s

role is sll central

even in the wake of

the adopon of new

technologies.

• Teachers like having a source of extensive,

curated digital content but want to maintain

responsibility for leading instrucon, as well

as control over students’ use of the content.

• Most students are not yet used to acng as

independent learners; as a result, teachers

implemenng Khan Academy typically need

to intenonally orient their students to the

types of independent learning pracces and

habits necessary for success in this learning

environment.

Our learnings in this study made it clear that

the teacher’s role is sll central even in the

wake of the adopon of new technologies. The

achievable classroom benets of using new

technologies can include building stronger

connecons with students, and developing

a clearer and deeper understanding of what

students actually know. At their best, the new

technology tools can enable teachers to do

what they nd most fullling: interacng with

students to have a posive impact on their

learning experience.

With this in mind, our study shows that

teachers sll need support in integrang online

instruconal resources into the curriculum;

they need digital content that is curated

and aligned with grade level standards,

and models of use that demonstrate the

resource’s value with students like theirs. As

we move toward greater classroom use of

self-paced instruconal resources, students

will also need addional support to navigate

this transion, which may vary depending on

the t between the individual student and

the online environment. To understand the

supports needed so that all students can excel

in self-directed online learning environments,

research should be pursued to understand

the roles that non-cognive student

characteriscs—movaon, persistence,

resourcefulness—play in student success in

these environments. Finally, experimental

studies of the impacts of dierent Khan

Academy implementaon models and other

digital learning tools like Khan are needed

to determine eects not only on math

achievement but also on students’ atudes

toward mathemacs and their capacity for self-

directed learning.

Research should be

pursued to understand

the roles that non-

cognive student

characteriscs—

movaon,

persistence,

resourcefulness—play

in student success in

self-directed learning

environments.

Bill & Melinda Gates Foundaon is a registered trademark

in the United States and other countries.

Developed by SRI Educaon with funding from the Bill & Melinda Gates Foundaon.

SRI Internaonal • 333 Ravenswood Avenue

Menlo Park, CA 94025 • www.sri.com

Phone: 650.859.2995 • Email: educa[email protected]om

For more informaon visit:

• SRI Educaon’s Khan Academy Project website:

hps://www.sri.com/work/projects/research-use-khan-academy-schools

• SRI Educaon: www.sri.com/educaon