data driven

Paper presentation @ PLoP 2015 writer’s workshop

/ scupelli

October 27, 2015 3:36 PM

Paul Inventado and Peter Scupelli presented “A Data-driven Methodology for Producing Online Learning System Design Patterns” at the 22nd Conference on Pattern Languages of Programs (PLoP) 2015 in Pittsburgh, Pennsylvania.

Abstract:

Online learning systems are complex systems that are frequently updated with new content, upgraded to support new features and extended to support new technologies. Maintaining the quality of the system as it changes is a challenge that needs to be addressed. Design patterns offer a solution to this challenge by providing guides to stakeholders responsible for making design changes (e.g., system developers, HCI designers, teachers, students) that will help them ensure system quality despite changes. Although design patterns for online learning systems exist, they often focus on one aspect of the system (e.g., pedagogy, learning). The data-driven design pattern production (3D2P) methodology utilizes data for producing design patterns in collaboration with stakeholders, addresses stakeholders’ concerns, and ensures the system’s quality as a whole. The paper presents five patterns produced by applying the methodology on the ASSISTments online learning system namely: all content in one place, just enough practice, personalized problems, worked examples, and consistent language. We made two changes to the pattern format: added in-text references in the forces section, and added an evidence section. The references allow the reader to learn more about the force in question. The evidence section highlights key findings uncovered from the 3D2P methodology. Four sources of evidence were considered in the pattern format: (a) literature – existing research on the problem or solution, (b) discussion – expert opinions about the problem or solution, (c) data – measures of the problem’s recurrence, and the solution’s effectiveness based on collected data; and (d) evaluation – assessment of the pattern’s performance when it was applied on an existing system. The changes to the format highlight linkages between pattern elements, theory, and empirical evidence. We believe that links further justify the design pattern, and make it easier for multiple stakeholders to understand them.

Last updated: 3:36 pm

Poster presentation @ Learning with MOOCs II – 2015

/ scupelli

October 16, 2015 8:29 PM

Paul Inventado and Peter Scupelli presented “Addressing MOOCs’ Sustainability Issues Using Data-driven Design Pattern Production” at Learning with MOOCs II – 2015 in Teachers’ College Columbia University, New York.

Abstract:

Despite the popularity of MOOCs, some higher education institutions have started moving away from it. Academic leaders seem to be less convinced of MOOCs’ sustainability. Four challenges need to be addressed to improve MOOC sustainability: (a) representing and communicating MOOCs best practices, (b) evaluating MOOC quality, (c) testing, validating, and refining MOOC designs, and (d) fostering research and collaboration around MOOCs. The data-driven design pattern production (3D2P) framework discussed here, can be used to address these four issues and can lead to the development of more sustainable MOOCs.

Last updated: 8:29 pm

Paper presentation @ EuroPLoP 2015 writer’s workshop

/ scupelli

July 11, 2015 3:54 PM

Peter Scupelli and Paul Inventado presented “Data-Driven Design Pattern Production: A Case Study on the ASSISTments Online Learning System” in a writing workshop 20th European Conference on Pattern Languages of Programs (EuroPLoP) 2015 in Bavaria, Germany.

Abstract:

Online learning systems popularity increased rapidly in recent decades in multiple domains such as cognitive tutors, online courses, and massive open online courses (MOOCS). The design quality of online learning systems is difficult to maintain. Multiple stakeholders are involved (e.g., software developers, interaction designers, learning scientists, teachers), the system is complex, there are rapid changes in software, platforms (e.g., mobile, tablet, desktop) and learning subject content, and so forth. Many existing online learning systems collect a significant amount of data that describe learning outcomes and student behaviors, which are indirect measures of system quality. Data analysis on online learning systems data can uncover linkages between particular design choices made and student learning outcomes. In this paper, we describe the Data-Driven Design Patterns Production (3D2P) methodology to prospect, mine, write and evaluate design patterns for online learning systems. Pattern prospecting helps designers decide what type of possible meaningful outcomes and features to scan for in the data and helps to focus on specific data subsets to limit the search space for pattern mining. Design patterns identified with 3D2P methodology can guide the addition of new content and the modification of system designs to maintain the online learning system’s quality. We present a case study of the ASSISTments math online learning system to illustrate the 3D2P methodology and discuss its benefits and limitations.

Last updated: 3:54 pm

New publication in eLearning Papers #42

/ scupelli

June 15, 2015 7:56 PM

Paul Salvador Inventado and Peter Scupelli’s paper entitled “Towards an open, collaborative repository for online learning system design patterns” has been published in eLearning Papers #42.

Abstract:

Design patterns are high-quality solutions to known problems in a specific context that guide design decisions. Typically, design patterns are mined and evaluated through four methods: expert knowledge, artifact analysis, social observations, and workshops. For example, experts discuss: knowledge, interpretations of artifacts, social patterns, and clarity of patterns. In this paper, we introduce a fifth method, a data-driven design pattern production (3D2P) method to produce design patterns and conduct randomized controlled trials as a means to evaluate applied design patterns. We illustrate the 3D2P method in the context of online learning systems (OLSs) that are difficult to create, update and maintain. To overcome such challenges, we propose an open repository for OLS design patterns, evaluation data, and implementation examples. On the repository, researchers can collaborate in the six stages of the pattern lifecycle (i.e., prospecting, mining, writing, evaluation, application, applied evaluation). The repository provides five benefits: researchers from different backgrounds can (a) collaborate on design pattern production; (b) perform distributed tasks in parallel; (c) share results for mutual benefit; (d) test patterns across varied systems and domains to explore pattern generalizability and robustness; and (e) promote design patterns to elevate OLS quality.

Last updated: 7:56 pm