Adaptive Learning: Context and Complexity

Charles Dziuban, Colm Howlin, Patsy Moskal, Connie Johnson, Mitchell Eid, Brandon Kmetz


This article describes a research partnership between the University of Central Florida and Colorado Technical University, with their common adaptive learning platform provider, Realizeit. The study examines component scores at the two institutions in mathematics and nursing based on a number of Realizeit system metrics. Although the principal components across disciplines and universities remained constant, student scores on those dimensions varied considerably. This indicates that adaptive learning is influenced by context and complexity. The context aspect helps frame student learning regarding knowledge, engagement, communication, and growth as they experience variability from faculty approaches to instruction. Complexity indicates a nonlinear learning pattern for the adaptive process in which the emergent property shows that interactions among the individual elements result in a more realistic model for explaining how students function in contemporary higher education. The authors raise a number of implementation issues for adaptive learning.

Adaptive learning (AL) technologies impact higher education by creating responsive learning environments that allow students to accelerate or extend their studies, thereby challenging the usual time constraints in the learning cycle. John Carroll (1963) identified this when he contended that if learning time were held constant, then knowledge, skill and concept acquisition would vary. But, if the constant is some prespecified level of achievement, then learning time will become the variable. In the vernacular of higher education, if you give all students one semester to learn College Algebra, there will be important differences in the knowledge each of them acquires. Determining mastery is a challenging assessment problem, but for the purpose of this paper the authors assume that it is relatively constant. An expanded adaptive model would place no constraints on knowledge acquisition or time, a combination that would impact much of higher education as it currently exists (Creative Destruction, 2014).

Without effective technology, implementing adaptive learning may be daunting because instructors cannot manage the modality without support. Fortunately, that help is available in a number of good functioning adaptive platforms (Dziuban, Moskal, Cassisi & Fawcett, 2016) that:

  1. Personalize the educational experience,
  2. Customize content,
  3. Continually assess student progress.

A number of important questions underlie these three simple components, however:

  1. What role does social learning play?
  2. What cognitive parameters are involved?
  3. How do students behave in the adaptive environment?
  4. Can adaptive learning be scaled?
  5. What is adaptive learning's impact on access to education?
  6. How do students perceive this learning structure?
  7. What are the elements of student affect?
  8. How is time modified?

Although comprehensive, these elements are by no means exhaustive because higher education contexts vary considerably throughout the world. Because of complexity issues, examining these elements individually will underrepresent adaptive learning.

Taleb (2018) puts it this way:

The main idea behind complex systems is that the ensemble behaves in ways not predicted by its components. The interactions matter more than the nature of the units. Studying individual ants will almost never give us a clear indication of how an ant colony operates. For that, one needs to understand an ant colony as an ant colony, no less, no more, not a collection of ants. This is called the emergent property of the whole by which parts and whole differ because what matters are the interaction between such parts. And interactions can obey very simple rules (p. 69).

From an operational perspective Forrester (1991) asserts three principles:

  1. The impossibility of anticipating how an intervention will ripple through a complex system;
  2. Often outcomes will be counterintuitive;
  3. There will be unanticipated side effects.

This article addresses these principles in the adaptive learning environment as they impact higher education.

University Collaboration

University of Central Florida (UCF)

The University of Central Florida is one of 12 public universities in Florida's State University System. It is located in Orlando and is the largest in Florida with over 68,000 students. UCF is a Hispanic serving institution with an average age of 24 with 22% of students over the age of 25 (UCF Facts, 2018).

In 2014, UCF began investigating adaptive learning as a means to improve student success. Realizeit is the university's enterprise platform, allowing the faculty control and flexibility over course content (Bastedo & Cavanagh, 2016). A team of Personalized Adaptive Learning (PAL) instructional designers at UCF's Center for Distributed Learning (CDL) provides support and guidance to the faculty as they implement the course design process. Faculty who wish to use adaptive learning participate in a faculty development program (PAL6000) and are assigned an instructional designer who is experienced with Realizeit. The support team provides assistance with the workload of adaptive course creation in order to ensure quality design (Chen, Bastedo, Kirkley, Stull & Tojo, 2017). CDL also provides video, graphics, and technology support as faculty redesign and teach their courses.

Colorado Technical University (CTU)

Colorado Technical University is a for-profit institution providing industry-relevant programs to a diverse student population of approximately 25,000 students. The university began offering online courses in 2000 and now offers over 50 online or blended programs. The older student population has an average age of 36 and is 60% female.

CTU's open enrollment results in students who enter with varying levels of expertise; therefore, the university began investigating adaptive learning in 2012. The flexibility of this approach provides students with unique learning paths that adjust to their varied knowledge and preferences to improve CTU's nontraditional students' online instructional experience. They are introduced to adaptive learning during orientation and, if needed, are provided with help guides and additional training in using the technology. Also, faculty must successfully complete a separate asynchronous training prior to teaching a course with adaptive technology.

Table 1 provides a summary of Realizeit use across the two institutions. The joint use of Realizeit provided CTU and UCF with an opportunity for collaborative research. Although the demographics of both universities vary, the learning analytics provided by Realizeit and a shared student reactions survey allow for common variables to be examined across these demographics. This collaboration has helped inform Realizeit's product development and recognition of customer needs as well as CTU and UCF's research and development in the adaptive learning environment.

Table 1. Realizeit adaptive learning use at UCF and CTU.

Started with adaptive learning Fall 2014 Fall 2012
Number of adaptive courses 26 (75 instances) 268 (4,157 instances)
Typical course length 12 weeks (summer) or 15 weeks (fall or spring) 5.5 weeks
Number of students 6,758 132,996
Number of enrollments in courses 7,514 933,154
Enrollments per student 1.11 7.02

Source: data involve cumulative totals provided by Realizeit; correct as of September 27, 2018.

The Adaptive Learning Partner: Realizeit

Realizeit is both an adaptive and adaptable learning platform. Institutions can bring their existing courses into Realizeit and make them adaptive or they can build adaptive courses from scratch. The platform is adaptable in that it does not impose a pedagogical approach on the course but can be customized to suit the needs of each instructor, course or institution. The platform supports approaches ranging from competency-based learning to self-directed approaches, as well as various models for learning in corporate settings.

The principle underlying all these strategies in Realizeit is the separation of curriculum from content (Howlin & Lynch, 2014). Traditionally, learning is content driven, with structure dictating the same linear pathway through the material for all students. In Realizeit the curriculum drives the direction of learning and uses content to help students acquire knowledge. The platform defines the curriculum using a hierarchical model known as the Curriculum Prerequisite Network - a directed acyclic graph in which the nodes represent the concepts to be learned, and the boundaries represent the prerequisite relationships that exist between them. Thus, Realizeit creates a map that shows a student many non-linear pathways to move through the concepts.

Just as an instructor can teach a concept in many ways, Realizeit provides multiple pieces of content and resources for each concept in the curriculum. The design is content agnostic - it is applicable in any learning domain and can deliver learning content in multiple formats.

Within Realizeit, the interaction of the learner with both the curriculum and the content generates a comprehensive stream of data that powers the Adaptive Intelligence Engine (AIE). This enables the platform's algorithmic adaptivity, personalization, guidance and feedback. The AIE discovers and adapts to each learner's changing achievement, behavior and preferences following a loop structure described in VanLehn (2006) and du Boulay (2006). It incorporates students' initial baseline results to estimate their prior knowledge. As students progress, additional outcomes enable Realizeit to suggest alternative learning trajectories. This results in continuous updates of students' ability estimates, the knowledge they have acquired, and objectives that still require mastery and recommendations for optimal paths through the course material.

At almost every point in the process, the student has final control over learning and subsequent steps. They may alter their learning path progression (trying new concepts) and alternatively undertake a review (revising/practicing previous concepts). In addition, they have access to supplemental learning material including adding, removing and reordering course elements within the content. However, this is not a completely open landscape for students, but a structured platform provided by Realizeit for optimal learning that allows some flexibility. The platform directs students towards ability-appropriate activities to increase their potential for success.

The courses examined

At UCF, Intermediate Algebra helps develop algebra skills and focuses on students who need remediation prior to enrolling in College Algebra - the primary math general education credit course taken by undergraduate students. Pathophysiology is required for students in the Bachelor and Master of Science in Nursing programs, and addresses abnormalities in physiologic functioning of the human body. All pathophysiology classes have content fully developed in Realizeit. The CTU analog to UCF's Intermediate Algebra is termed Introductory Algebra and provides skills and concepts necessary to succeed in further mathematics studies. Analytic Algebra is a basic algebra course that is specifically required for engineering and information technology students, and focuses on linear, rational and quadratic equations. The CTU Nursing course content includes a change management project pertaining to nursing practice and focuses on leadership skills. The comparison model is presented in Table 2.

Table 2. CTU/ UCF Course Comparison

Introductory Algebra Intermediate Algebra
Analytic Algebra College Algebra
Trends in Contemporary Nursing Pathophysiology

Source: Reprinted by permission from Online Learning.

The basis for this study

This study augments the 2018 findings of Dziuban et al. (2018), demonstrating that adaptive learning stabilizes the underlying components of several outcome metrics across universities and subject areas. The authors considered several indices produced as students use Realizeit for courses delivered in the adaptive modality at the University of Central Florida (UCF) and Colorado Technical University (CTU): The measures used in that study are presented in Table 3.

Table 3. Explanation of Variables

Variable Explanation
Knowledge State (KS) A measure of student ability. The mean level of mastery that the students have shown on topics they have studied.
Knowledge Covered (KC) A measure of student progress. The mean completion state of each of the course objectives.
Calculated (CA) An institution-defined combination of several metrics, mainly KS and KC, used to assign a grade to students.
Average Score (AS) The mean result across all learning, revision, practice, and assessment activities.
Determine Knowledge (DK) The percentage objectives on which the student completed a Determine Knowledge operation.
Knowledge State Growth (KSG) The extent by which a student's KS has changed from the start of the course. Can be positive, negative, or zero.
Knowledge Covered Growth (KCG) The extent by which a student's KC has changed from the start of the course. Can be positive or zero.
Interactions (IN) The engagement level of the instructor(s) with the student. The total number of interactions.
Messages Sent (MS) The number of the interactions sent by the instructor that were simple messages.
Total Activities (TA) The total number of nonassessment activities started by the student.
Total Time (TT) The total time spent on nonassessment activities started by the student.
Number Revise (NR) The total number of node-level activities that are classified as revision.
Number Practice (NP) The total number of objective-level practice activities.

Source: reprinted by permission from Online Learning.

To make comparisons of latent patterns in the aligned courses, the authors derived principal component solutions within and across each institution and computed similarity coefficients (Chan, Ho, Leung, Chan & Yung, 1999), finding a high degree of correspondence among components for all disciplines in both universities (average Tucker index=0.92). All student samples with courses combined from each university produced virtually identical results. Table 4 shows a prototypical pattern matrix encountered by the authors.

Table 4. Transformed (Promax) Pattern Matrix for the Realizeit Indices, Entire Sample at UCF (n = 1,528)

Index KA EA C G
Calculated .95 .04 -.01 .12
Knowledge covered .95 .02 .02 .13
Knowledge state .91 .01 -.10 .02
Determineknowledge .79 -.06 .12 -.21
Average score .37 .02 -.20 -.15
Total activities -.05 .97 -.02 -.09
Num. revised -.02 .90 -.15 .00
Num. practiced .11 .61 .16 -.25
Interactions -.01 -.02 .98 .01
Messages sent -.01 -.02 .98 .01
Knowledge covered growth .05 -.11 .04 .93
Knowledge state growth -.06 -.05 -.09 .92
Total time -.04 .30 .24 .44

Source: the table reprinted by permission from Online Learning.

For every comparison, four components emerged. Pattern coefficients absolutely equal to or greater than 0.30 were used as the criteria for index salience. Those values are identified in Table 4; however, for ease of interpretation they are also listed after each component name in this section.

Knowledge Acquisition (KA): Comprised of the Calculated, Knowledge Covered, Knowledge State, Determine Knowledge and Average Score indices. This component relates to educational achievement and has a mastery element associated with it. Knowledge acquisition in adaptive learning assesses learning prior to, during and upon completion of a course and forms the benchmark for student success. In addition, it serves as the basis for the decision engine's recommendation about the appropriate learning path for students and an early indication of possible difficulties in the learning sequence.

Engagement Activities (EA): Comprised of the Total Activities, Number Revised, Number Practiced and Total Time indices. This component bears a strong relationship to what Carroll (1963) called the time students spend in actual learning and relates to how much energy a student expends in the learning process. If one could hold ability level constant, a reasonable assumption might be that students who are more engaged in learning activities will score higher on knowledge acquisition.

Communication (C): Comprised of the Interactions and Messages Sent indices, communication emerges in the Realizeit platform, enabled by messages sent and interactions. This is the social dimension of adaptive learning and the way students communicate with each other and their instructors. At another level this component underlies the effort expended communicating during the courses.

Growth (G): Comprised of the Knowledge Covered Growth, Knowledge State Growth and Total Time indices, growth is a clear expectation for any course. Measuring change in student knowledge can result from many baseline measures and is an important element of the learning cycle. Growth is change in what information a student has mastered and is the key bellwether for student progress in their adaptive learning courses.

The methods for this study

Based on the four components and their stability as found in the Dziuban et al. study (2018), the authors have concluded that deriving the component scores for like courses between the two universities would add context to those results. A component score for each student in the study gives an indication of his or her spacing or the degree to which a student relates to each dimension. For instance, a student with a higher score on knowledge acquisition has more affinity for that dimension than a student with a lower score. A student with higher scores on engagement and communication may be more concerned with the class climate rather than knowledge acquisition or growth. This comprises useful information because, although the components are stable, it does not follow that the component scores will reproduce a similar pattern.

The Anderson Rubin (Anderson & Rubin, 1956) component score derivation was used because it is best suited for solutions where the dimensions are correlated to some degree. Most procedures yield unit normal variates with a mean of 0 and a standard deviation of 1. Knowledge acquisition scores were computed for alignment in beginning Algebra, College Algebra and Nursing courses at both UCF and CTU and the procedure was repeated for knowledge growth, engagement and communication. The unit normal scores were linearly transformed to have a mean of fifty and standard deviation of ten (the T score transformation). The score means and standard deviations for courses at each university were derived and tested for significance. However, large sample sizes resulted in high power for those tests, so effect sizes were determined according to the Hedges' g procedure (Hedges, 1982). Values that reached .5 were considered noteworthy, consistent with normally accepted guidelines. Error bar graphs were used to provide a visual model for making decisions about the comparability of the component scores.

Component Score Comparisons

The following analysis examines the mean component scores for each course on each of the 4 components found by Dziuban et al. (2018). Once again, these scores determine, on average, how each course related to each component--how big a role did each component play in the way learning took place? This is not a case of one course outperforming another, indicating that the organization and context of the courses' scores on the components require careful consideration. For example, little or no communication between the instructor and students can result in a low mean score on the communication component. This low score on communication might have several explanations, (e.g., reduced instructor engagement or communication taking place outside of the adaptive platform).

Additionally, scores with a high degree of variability among students within each course were evaluated. Where instructive, the standard deviation of the scores will be discussed. The following tables and graphs provide the component mean score and standard deviation for each course. Although this was a combined university study, all findings for component score levels were disaggregated by individually aligned courses for both universities. The results reflect that separation.

Knowledge acquisition (KA)

Knowledge acquisition is the first principal component, explaining the largest proportion of variance for each of the six courses. This dimension represents a cluster of metrics that capture student progress, ability and grade. The summary statistics for each course on KA are given in Table 5 and Figure 1.

Table 5. Knowledge acquisition analyses

  n SD n SD p g
Analytic Algebra/ College Algebra 363 43.1 13.4 4,486 47,5 11.2 .00 .39
Intro to Algebra/ Intermediate Algebra 302 49.3 9.5 6,993 51.3 8.9 .04 .19
Trends in Contemporary Nursing/ Pathophysiology 537 53.1 7.5 303 55.7 5.5 .00 .38

Noteworthy effect sizes
Source: authors' own study.

Figure 1. Knowledge Acquisition. The points represent the course mean and the vertical bar represents the ±1 standard deviation from the mean. CTU is represented by red and UCF by blue.

Source: authors' own study.

Despite being the first principal component, substantial differences between the course comparison pairs do not result. Each of the CTU courses is higher on KA that their corresponding course in UCF, although none of the effect sizes are noteworthy. The same pattern emerges across both institutions with the Nursing courses relating most highly, followed by the introductory Algebra courses, with the more advanced Algebra courses last. The standard deviations appear relatively similar on each pair of comparisons.

Findings indicated that both Nursing courses show the smallest variability, while the College Algebra courses have the largest variance. Nursing students relate to KA more highly, but measured along these components they are much more like each other than College Algebra students. The reasons for the diminished variance in Nursing courses may be due to the subject matter; nursing students are both highly invested and interested in the content. Students in College Algebra are taking the course as a general education requirement so their interest level and perception of relevance in this course varies. Additionally, because of the narrow discipline focus, nursing students are more likely to have a similar student profile and may be older and further along in an academic program or profession.

Engagement activities (EA)

Engagement Activities represents the second principal component in each of the solutions. It defines the cluster of measures indicating the number of different activities attempted, and time students spent engaged with the learning content. The summary statistics of each course on EA are given in Table 6 and Figure 2.

Table 6. Engagement Activities

  n SD n SD p g
Analytic Algebra/ College Algebra 363 49.7 10.7 4,486 31,4 1.3 .00 5.8*
Intro to Algebra/ Intermediate Algebra 302 45.6 4.4 6,993 79.3 9.8 .04 3.5*
Trends in Contemporary Nursing/ Pathophysiology 537 53.2 11.1 303 44.1 4.9 .00 1.9*

*Noteworthy effect sizes
Source: authors' own study.

Figure 2. Engagement Activities: The points represent the course mean and the vertical bar represents the ±1 standard deviation from the mean. CTU is represented by red and UCF by blue.

Source: authors' own study.

Each of the paired course comparisons has a noteworthy effect size (>0.5). College Algebra at UCF relates to this component significantly more than its corresponding course in CTU, Analytic Algebra. Remarkably, there is very little variability among the 4,486 students in this CTU Algebra course. Almost all students relate to this component similarly, meaning the general level of engagement of these students, when measured across a range of metrics, is approximately the same. Some may spend more time and some may do more activities, but it reduces to the same general level of engagement. At CTU similar results may occur because only students pursuing the Bachelor's degree in Information Technology are required to take Analytic Algebra. At UCF however, College Algebra is one of the math courses in the general education program. The majority of undergraduate students take it, resulting in a variety of student demographics and academic interests.

The scores reverse themselves when looking at the Introductory Algebra courses. The CTU course, Introduction to Algebra, relates significantly higher to this component than the UCF course, and while not as low as the previous comparison, the UCF course has small variability. CTU is an open enrollment institution and the average age of students is in the mid-to- late-thirties. The higher level of engagement may be a response, in some cases, to students who have not been exposed to College Algebra for over fifteen years (students are required to take this course unless they successfully complete a college entrance exam-CTU does not require the SAT or ACT). UCF's Intermediate Algebra course is required only by those who fail the math placement exam and does not fulfill the math credit requirement for undergraduates; students must successfully complete it before progressing to College Algebra.

Again in Nursing, the effect size is noteworthy but not as large as the other two comparisons. The UCF course is higher on this component and again the lower course (CTU's), has smaller variability. Some possible reasons for the low levels of variability among students in CTU Analytic Algebra, UCF Intermediate Algebra and CTU Trends in Contemporary Nursing are that in these courses there is a general ceiling effect for the level of engagement that is required from the students. In the UCF course this could be because an artifact of the fact that Intermediate Algebra is essentially a remedial prerequisite course that prepares students for College Algebra but is not credit-bearing. Students may be putting in the minimum level of effort required to complete the course. The relatively short contact time of 5.5 weeks for the CTU courses may also impact results, as there is little time for students to engage in protracted engagement efforts.

Communication (C)

Communication represents the cluster of metrics that captures the level of communication from the instructor to the students; the metric is passive on the student side and mostly dependent on metrics driven by the instructor. The summary statistics of each course for this component are given in Table 7 and Figure 3.

Table 7. Communication

  n SD n SD p g
Analytic Algebra/ College Algebra 363 60.5 15.2 4,486 52,9 9.9 .00 0.73*
Intro to Algebra/ Intermediate Algebra 302 50.2 2.7 6,993 49.1 9.5 .04 .12
Trends in Contemporary Nursing/ Pathophysiology 537 46.2 0.6 303 47.7 10.4 .00 .19

*Noteworthy effect sizes
Source: authors' own study.

Figure 3. Communication: The points represent the course mean and the vertical bar represents the ±1 standard deviation from the mean. CTU is represented by red and UCF by blue.

Source: authors' own study.

A high score on this component for an individual student may indicate substantial communication from the instructor due to student needs and preferences or instructor communication style; a student might require a higher level of direction from the instructor or remediation, and a strong student might receive extra material--an instructor might prefer to send regular updates.

In the UCF College Algebra course, students on average relate more highly to this component than the corresponding course in CTU with an associated effect size of 0.73. There is also a high level of variability among students on this component in the UCF course because the instructor is highly engaged and exhibits all the behaviors previously listed, personalizing the level of communication to the needs and requirements of each student.

Note that in the CTU course, students relate approximately the same to this component with similar variability. At CTU instructors communicate with students at least weekly, as outlined in the university's faculty expectations, because the length of CTU courses is 5.5 weeks (many instructors engage every few days). If students fall behind, time becomes a debilitating factor in the ability to succeed. Intermediate Algebra and Pathophysiology exhibit very low variability. This suggests that the instructor broadcasts general messages versus personalized communications.

Knowledge Growth

Knowledge Growth (G) represents a group of metrics that define each student's progress during the course. The summary statistics of each course on this component are given in Table 8 and Figure 4.

Table 8. Knowledge Growth (G)

  n SD n SD p g
Analytic Algebra/ College Algebra 363 49.1 9.9 4,486 51.4 10.2 .00 0.23
Intro to Algebra/ Intermediate Algebra 302 55.4 11.8 6,993 49.2 9.8 .04 .63*
Trends in Contemporary Nursing/ Pathophysiology 537 49.3 8.4 303 47.9 7.9 .02 .17

*Noteworthy effect sizes
Source: authors' own study.

Figure 4. Knowledge Growth: The points represent the course mean and the vertical bar represents the ±1 standard deviation from the mean. CTU is represented by red and UCF by blue.

Source: authors' own study.

The only comparison with a noteworthy effect size is Intermediate Algebra at UCF when contrasted with Introduction to Algebra at CTU. This is not surprising since the UCF course is more remedial. Those students without the necessary knowledge to attempt College Algebra must complete the Intermediate Algebra course. This would imply a relatively reduced level of prior knowledge meaning that they also have an opportunity to grow. Prior knowledge will always be negatively correlated with knowledge growth. That is, students who know most of the course material at the start have less to gain than those who know very little and now have the opportunity to learn predominantly more. The variability is approximately equal across all the comparisons and is large enough to show that students in these courses have a wide range in different levels of progress and/or changes to their mastery level.

Discussion and Implications

Previous work by the authors (Dziuban et al., 2018) and this article show that adaptive learning involves stable dimensions (knowledge acquisition, learning engagement, growth and communication) across varied disciplines and learning contexts. However, results reported here suggest that adaptive learning is complex because variables are involved, some of which are observable and some that cannot be measured. Multiple colleagues have commented that the scores have been derived from dimensions that cannot be directly observed. While true, these methods seem reasonable in order to understand the interactions that define the emergent property of adaptive learning. Often, important elements must be constructed. Further, there is a predictive element in this work that describes how various student prototypes will respond to adaptive learning. The researchers offer explanations such as highly motivated nurses, general education for undergraduates and the characteristics of working adults, but explications are better cast as working hypotheses. Cause and effect allude the researchers in this study.

Several circumstances, however, do appear to impact adaptive learning environments. Although the two universities and their common adaptive platform provider do not comprise a comprehensive sample, their diversity suggests that the interaction of institutional strategic initiatives and the capability of the adaptive platform constitute a reasonable framework for predicting how students will learn, engage, grow, and communicate. For instance, CTU can require faculty members to perform certain functions, whereas UCF has much less control over how instructors conduct their courses. This is true for students as well. Acquiring some entry level baseline for the determine knowledge metric gives the decision engine in Realizeit much better parameters for guiding a student through a course. Making the pretest optional at UCF impacts the instructional design algorithm.

Student cognitive, affective and behavioral characteristics impact how adaptive learning constitutes the educational climate in the classroom. The large populations of older and mostly working adults who enroll at CTU have many demands that compete for their time and attention. Although certainly motivated to learn, seemingly a primary goal for them would be an educational credential that advances them professionally. Therefore, adaptive learning allows them to control their learning space, accommodate their need for workplace or job progression and receive an excellent education. At UCF, College Algebra is a major stumbling block for undergraduate students, especially when used as a general education requirement with no intention to pursue further mathematics study. Not meeting the math placement requirement is an event that seriously alters their educational program. Required enrollment in the noncredit bearing Intermediate Algebra course can lengthen a student's completion time, causing them to change majors or drop out. Sequential adaptive Intermediate and College Algebra courses create the possibility to complete both in one semester and remain on track, thereby reducing student ambivalence about mathematics and education in general.

These results also have implications for both faculty and student development. The level at which students relate to knowledge is relatively equal across courses and universities but demonstrates considerable individual student variability, because engagement levels tend to vary widely across disciplines and universities. Similarly, communication varies across discipline and institution. Gains reported as individual differences are substantial but university levels are generally equivalent except for Introductory Algebra. Taken as a collective, the researchers have found that although the same dimensions define the adaptive learning environment, how students relate to those dimensions is key to understanding their voice.

These diversities strengthen adaptive systems, forcing students and faculty to become more flexible and agile because small inflection points during the learning process can result in dramatic changes in the process. The non-linearity inherent in adaptiveness presents problems for prediction and determination but creates an autocatalytic learning system that generates continuous feedback loops that create momentum for the system.

Limitations inherent in this study constrain the robustness of these results that should be validated across multiple universities and adaptive platforms. Nothing in these analyses assesses the psychometric adequacy of the indices in Table 3. Do they represent an adequate sample from a domain of importance?

Like most studies in higher education, this research raises many questions. Can adaptive learning help reduce the growing ambivalence in the student population about obtaining a post-secondary education? Can it lesson ambiguity among students about understanding the rules of engagement in their courses? Is there some way, through adaptiveness, to further inspire teachers? Can the relationship among faculty and students be improved? Can interaction and communication be refreshed, creating a more energized learning environment? Ultimately, can adaptive learning expand the productive learning horizon for students?

While these questions represent aspects of complexity, their answers will address more fundamental ones. Does adaptive learning have a bona fide place in higher education and if so, what is the potential value added verses opportunity cost? These larger questions are best answered through research partnerships not only between universities and vendors, but with professional and governmental agencies. Multiple perspectives, although conflicting, can add clarity to an issue and provide better guidance for research that is authentic, contextual and reflective.


  • Anderson, R.D. & Rubin, H. (1956). Statistical inference in factor analysis. Proceedings of the Third Berkeley Symposium of Mathematical Statistics and Probability, 5, 111-150.
  • Bastedo, K. & Cavanagh, T. (2016). Personalized Learning as a Team Sport: What IT Professionals Need to Know. EDUCAUSE Review. Retrieved from
  • Chan, W., Ho, R.M., Leung, K., Chan, D.K., & Yung, Y. (1999). An alternative method for evaluating congruence coefficients with Procrustes rotation: A bootstrap procedure. Psychological Methods, 4(4), 378-402. DOI:10.1037/1082-989X.4.4.378.
  • Chen, B., Bastedo, K., Kirkley, D., Stull, C. & Tojo, J. (2017). Designing personalized adaptive learning courses at the University of Central Florida. ELI Brief. Retrieved from
  • Carroll, J.B. (1963). A model of school learning. Teachers College Record, 64(8), 723-723.
  • Creative Destruction. (2014, June 28). Retrieved from
  • du Boulay, B. (2006). Commentary on Kurt VanLehn's 'The Behaviour of Tutoring Systems.' International Journal of Artificial Intelligence in Education, 16(3), 267-270.
  • Dziuban, C., Moskal, P., Cassisi, J., & Fawcett, A. (2016). Adaptive Learning in Psychology: Wayfinding in the Digital Age. Online Learning, 20(3), 74-96.
  • Dziuban, C., Howlin, C., Moskal, P., Johnson, C., Parker, L., & Campbell, M. (2018). Adaptive Learning: A stabilizing influence across disciplines and universities, Online Learning, 22(3), 7-39.
  • Forrester, J.W. (1991). System dynamics and the lessons of 35 years. In K.B. de Greene (Ed.), Systems-based approach to policymaking. Norwall, MA: Kluwer Academic.
  • Hedges, L.V. (1982). Estimation of effect size from a series of independent experiments. Psychological Bulletin, 92(2), 490-499.
  • Howlin, C.P., & Lynch, D. (2014). A framework for the delivery of personalized adaptive content. International Conference on Web and Open Access to Learning (ICWOAL), 20, 1-5.
  • Taleb, N.N. (2018). Skin in the Game: Hidden Asymmetries in Daily Life. New York: Random House.
  • UCF Facts, 2018. (n.d.). Retrieved from
  • VanLehn, K. (2006). The behavior of tutoring systems, International Journal of Artificial Intelligence in Education. 16(3), 227-265.

Dr. Charles Dziuban

The Author is the Director of the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where has been a faculty member since 1970. Since 1996, he has directed the impact evaluation of UCF's distributed learning initiative, longitudinally examining the impact of web technologies on teaching and learning. In 2000, Chuck was named UCF's first ever Pegasus Professor for extraordinary research, teaching, and service and in 2005 received the honor of Professor Emeritus. In 2005, he received the OLC award for Most Outstanding Achievement in Online Learning by an Individual and was named an inaugural OLC Fellow in 2010. Dr. Dziuban has made over 100 presentations at universities throughout the world on how modern technologies impact teaching and learning. Currently he is co-directing, with Dr. Patsy Moskal, the evaluation of UCF's pilot project in adaptive learning.

Colm Howlin

As Principal Researcher at Realizeit, Colm leads the research and analytics teams. His research powers the models that underpin Realizeit's personalized learning engine. Colm also works with academic researchers to better understand the impact and opportunities related to the usage of the Realizeit engine and adaptive learning in general. Colm holds a B.Sc. in Applied Mathematics & Computing and a Ph.D. in Applied Mathematics.

Dr. Patsy D. Moskal

The Author is the Associate Director for the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where she has been a faculty member since 1989. Since 1996, she has served as the liaison for faculty research of distributed learning and teaching effectiveness at UCF. Patsy specializes in statistics, graphics, program evaluation, and applied data analysis. She has extensive experience in research methods including survey development, interviewing, and conducting focus groups and frequently serves as an evaluation consultant to school districts, and industry and government organizations. She has also received funding from several government and industrial agencies including the National Science Foundation and the Alfred P. Sloan Foundation. In 2011, she received the honor of being named an OLC Fellow and she now serves on the OLC Board of Directors. She has co-authored numerous articles and chapters on blended and online learning and is a frequent presenter at conferences and to other researchers.

Dr. Connie Johnson

The Author is Colorado Technical University's (CTU) Chief Academic Officer and Provost, working with both online and ground degree programs. She has oversight of academic affairs, including faculty, curriculum, classroom experience and accreditation. During her time at CTU, Connie has initiated adaptive learning technology implementation, effective leadership of academics, women's leadership, leading academics through change, and effective technology implementation in online classroom. Connie's educational background comprise of an EdD, Education / Organizational Leadership (2010) an MBA, Management (1991), Nova Southeastern University; and a BS with honors, Criminal Justice, Florida State University. Connie has served in higher education for over 20 years with extensive experience in teaching, administration, and leadership. Additionally, Connie has extensive experience in regional accreditation, curriculum implementation, and faculty training and development. She is a trained peer evaluator for the Higher Learning Commission, has completed in the ACE Chief Academic Officer program, and is a member of the CTU Board of Trustees. Prior to joining the world of higher education, Connie was a licensed stockbroker for ten years. Her first career out of college was as a probation officer in South Florida.

Mitchell Eid

The Author is a first year Masters' student at the University of Central Florida. Mitchell is currently studying Industrial/Organizational psychology and working as a research assistant in the UCF's Research Initiative for Teaching Effectiveness center. He also attended UCF for his Bachelors' degree in Psychology prior to graduate school. Before switching careers and beginning his studies in Psychology, Mitchell attended the University of Florida and Miami to study Accounting.

Brandon Kmetz

The Author is a first year Masters' student at the University of Central Florida. Brandon is currently studying Industrial/Organizational psychology and working as a research assistant in UCF's Research Initiative for Teaching Effectiveness center. Brandon Kmetz received his Bachelor of Science in Psychology from the Florida State University.

About the article


The article is in the printed version on pages 13-23.

pdf download PDF

pdf read the article (English)

How to cite

Dziuban, CH., Howlin, C., Moskal, P., Johnson, C., Eid, M. & Kmetz, B. (2018). Adaptive Learning: Context and Complexity. e-mentor, 5(77), 13-23. DOI: