A lot of research within the learning sciences focuses on deriving design-related contextual pedagogical theory (also known as local theory). This produces knowledge that sits in-between theory and practice with increasing translational value, though it is time-consuming and results in contextual knowledge that lacks in generalisable/scalable applications. That is, to derive each of these design-related insights, the researcher needs to go through a lengthy process of performing a literature review, recruiting participants, creating different designs, conducting the experiment and evaluating the design results. Here, I propose an idea to leverage practice and automate (from the researchers' perspective) the derivation of design-related contextual pedagogical theory, as well as supporting the incorporation of such theory within practice. Note, when referring to automation it can often conjure images of simulated students, though this proposal focuses on interaction with real students in real environments to ensure the naturalistic validity of derived theories. The automation is within the research processes. ## 1. Situating the knowledge Within the learning sciences there are different types of contributions. Here, I organise some types of contributions, to better situate the proposition. The only tangibles are the instantiated design, the way the student engages with the design, and the respective learning primitives that are triggered. However, learning is far too complex to be able to automatically derive the ideal design for a given scenario by purely reasoning as to how it can be mapped to learning primitives, with respect to particular objectives. Firstly, learning primitives happen at such a granular level, that designing with respect to forcing the student to move through these primitives, leads to us ignoring the forms of pedagogical insight that are difficult to explicitly operationalise; akin to the reductiveness that we see with teaching machines [[@wattersTeachingMachinesHistory2023|(Watters, 2023)]]. Secondly, some forms of educational objectives are difficult define in concrete ways, which could lead to the over-prioritisation of explicit/measurable declarative or procedural knowledge over implicit social knowledge. Hence, theories like constructionism or communities of practice reason on a higher-level of abstraction; that is, they focus on the environments, the surroundings, the social communities, where changes to the learning ecology [[@postmanFiveThingsWe1998|(Postman, 1998)]] can increase the likelihood of the learning primitives being triggered, rather than explicitly controlling/mapping to them. For example, constructionism is not concerned with controlling the specific cognitive learning primitives, but rather lets go of control. It posits that through the student constructing an external artefact, relevant behaviours like reflection will take place that lead to knowledge construction within the student's mind [[@papertMindstormsChildrenComputers1980|(Papert, 1980)]]. Therefore, the **theoretical learning contributions** can be seen as differing ways of thinking about education at differing levels of abstraction. * **Primitives**, often found in foundational psychological research, such as [[Spaced repetition|spaced repetition]], [[Active Recall|active recall]], [[Imitation|imitation]] and [[Observation|observation]]. * **Frameworks**, operate on a higher-level of abstraction and view learning in different ways, such as [[Constructionism|constructionism]] and [[Communities of practice|communities of practice]]. * **Meta-frameworks**, act as ways of thinking about the use of frameworks and primitives, such as [[Conjecture Mapping|conjecture maps]], [[The Knowledge-Learning-Instruction Framework|the knowledge-learning-instruction framework]], and [[Educational modelling language|educational modelling languages]] (such as [[Orchestration graphs|orchestration graphs]]). Where the learning designer can inform their decisions through frameworks and meta-frameworks, which act as proxies to the core learning primitives. ![[Pasted image 20250725102214.png]] Apart from theoretical learning contributions, there are also **practical contributions**; that is, the produced designs such as lesson plans. Though in-between theoretical and practical contributions, sits **contextual design-related pedagogical knowledge**. It provides pragmatic answers to learning design problems in a particular context (eg. learning objective, domain, student age). These are not usually foundational named theories but are nonetheless important to inform our designs, such as the difference between instruction-first or problem-solving-first instruction within algebra K-12 contexts. Lastly, embedded within the designs are practitioners implicit insight. They are often too vast, tacit, implicit and extensive to be operationalised, but can be captured and shared by proxy of the designs [[@dalzielLAMsCommunityBuilding2013|(Dalziel, 2013)]]. These different forms of knowledge can be visualised in the design process as following. ![[Pasted image 20250721112713.png]] Akin to an hour glass, it can be turned upside down where the instantiated design flows down with student engagement to trigger the learning primitives. These are the only tangibles, everything else is knowledge that strives to reach these tangibles. ![[Pasted image 20250725102145.png]] This is a proposed flow/organisation of knowledge in the process of creating designs. Though note, it is not a taxonomy of learning science knowledge. There are a multitude of different contributions that are not discussed, such as instantiating theory with technology (eg. [[Spaced repetition|spaced repetition]] through the [[FSRS]] algorithm, or [[Socratic questioning|socratic questioning]] by [[LLM-control methods|fine-tuning LLMs]]). ## 2. The problem As the name implies, contextual design-related pedagogical theories are not generalisable. However, due to its contextual-nature there are a very large number of questions to ask, such as, what differences do these differing activity orderings produce, and what tradeoffs are produced by using activity X instead of the usual Y. Though as knowledge that is situated much closer to the design process, they are very useful and easier to translate to practice. Hence, the solution is not to move up levels of abstraction to frameworks and meta-frameworks, but to stay within the contextual-details and perform it at a drastically faster rate with simpler processes for the researcher. That is, an approach through which we can rapidly test permutations, thereby, motivating the following research question. **RQ1. How can we automate the experimentation processes for deriving contextual design-related pedagogical theories?** However, unlike frameworks where designers can be taught about core theories like constructionism, it is impractical to teach designers about a plethora of permutations and their respective learning effects; thereby, motivating the following research question. **RQ2. How can we support learning designers in better making use of vast array of contextual design-related pedagogical theories?** ## 3. Relation to prior work This idea sits within [[Design-based research|design-based research (DBR)]], in that it is a method that values developing contextualised theory (RQ1) in a manner that can be accumulated and used to inform practice (RQ2) [[@thedesign-basedresearchcollectiveDesignBasedResearchEmerging2003|(The Design-Based Research Collective, 2003)]]. DBR takes an iterative approach to research, with continuous cycles of (re)design, enactment, analysis. As illustrated below. ![[Pasted image 20250725102243.png]] DBR is not highly prescriptive of specific methods, but rather directs the researcher to think about methods that 'link processes of enactment to outcomes' [[@thedesign-basedresearchcollectiveDesignBasedResearchEmerging2003|(The Design-Based Research Collective, 2003)]]. Here, the motive is to produce a new kind of method that can be done in an automated manner, though this comes with reductionist tradeoffs to each stage of (re)design, enactment and analysis. Some have tried to get at the essence of these research questions, that is connecting practice to theory in a scalable manner, albeit through different angles. The LAMS community shares and adapts the practical contributions, that are the specific designs or lesson plans which have contextual design-related pedagogical knowledge embedded within them [[@dalzielLAMsCommunityBuilding2013|(Dalziel, 2013)]]. Meanwhile, pedagogical patterns describe reusable solutions to recurring teaching problems within particular contexts that can be instantiated for use by the designer/practitioner [[@laurillardTeachingDesignScience2013|(Laurillard, 2013)]]. However, the bi-directional seamless integration between practice and research is lacking. The former, the sharing of designs, acts as a way of accumulating practical contributions which need to be analysed to extract theory. Whilst the latter, the pedagogical patterns, acts as a method of disseminating theoretical insights into practice. The proposed idea attempts to bridge this gap and produce a cycle, where we leveraging the vast amount of existing practice that occurs as a power source for generating theory, which then can be used to refine practice. Akin to a waterfall (practice) that uses power turbines to generate electricity (theory), where the produced electricity is used to purify the water (practice) upstream. For scalability and automation, we need to make use of [[Educational modelling language|educational modelling languages (EML)]]. That is, a way of operationalising our design in a machine-executable format and acts as a medium through which we can express our proto-theory, which are our design and theoretical conjectures about the effects of an intervention in practice [[@sandovalConjectureMappingApproach2014|(Sandoval, 2014)]] (for RQ1). Additionally, the formal representation of derived theories can allow for algorithms that can support learning designers in the incorporation of theory into designs (for RQ2). This idea proposes orchestration graphs as the chosen EML [[@dillenbourgOrchestrationGraphsModeling2015|(Dillenbourg, 2015)]], which provides an expressive and extensible vocabulary that integrates theories from the learning sciences, as well as modelling for the different social planes on which learning takes place. Hence, derived theories can expand beyond the individual or cognitive dimension to collaborative learning. This motive to automate the derivation of design-related effects is also seen within human-computer interaction (HCI), with A/B testing (also known as split testing) [[@kohaviControlledExperimentsWeb2009|(Kohavi, 2009)]]. They similarly have to deal with the messiness of human behaviour to inform their UI/UX design. A/B testing are randomised experiments, where the content that is given to the user (eg. website, app) will be varied with respect to a feature, design-element, and so on; and be measured against a pre-defined metric such as click-through rate. It is often performed through automated experimentation platforms that aids the UI/UX researcher in producing multiple different variants, performs traffic allocation and monitor/evaluate the execution with respect to the metrics. It could also have capabilities to automatically vary traffic allocation depending on what design qualities are determined to be the winner. We also see automated experimentation systems through the idea of the self-driving lab within the natural sciences, for tasks such as drug discovery. They often contain robotic arms performing a range of actions such as pipetting, with software used to design, schedule, execute, monitor and analyse experiments [[@abolhasaniRiseSelfdrivingLabs2023|(Abolhasani & Kumacheva, 2023)]]. **To my knowledge, such an automated experimentation platform tailored to the qualities of educational design and theory production, does not exist.** However, education is a social science with much of its theory existing within a different epistemological plane, that is not suited towards such automated experimentation. However, for specific forms of theory, namely 'design-related contextual pedagogical theory', research automation that leverages existing practice could be highly fruitful. ## 4. The idea Expanding upon the analogy of a waterfall (practice) that uses power turbines to generate electricity (theory), where the produced electricity is used to purify the water (practice) upstream. This idea, demands the creation of a power turbine that is capable of tapping into the relentless, massive and never-ending flow of existing practice, and leveraging this to produce theory (RQ1). It also demands a purifier, that is capable of taking the vast amounts of theory that is continuously generated and be able to improve this relentless flow of practice upstream (RQ2). However, as researchers, we are often playing with individual trickling streams. Therefore, to engage in this proposed scale requires a method that bridges over the fields of learning design, learning analytics, and tutoring systems. Whilst the core of the idea takes research automation to the extreme, I will also discuss the potential of having human involvement in a low-effort manner that could retain scalability whilst combatting the inherent reductionism in automating these stages. In particular, many of simplifications created through automation, could open the potential for practitioners to take on the role of researchers, which is central to scalability. I first discuss forms of operationalising knowledge, which is core to understanding the idea. After which, I detail the idea for RQ1, with respect to the stages of (re)design, enactment and analysis. Finally, I denote how such generated theory can be used to address RQ2. #### 4.1 Operationalising knowledge There are two primary forms of knowledge that concerns this idea: the design artefacts and the pedagogical knowledge (primitives, frameworks, meta-frameworks and contextual design-related pedagogy). For the needs of RQ1 and RQ2, we require a formal well-defined language in which both forms of knowledge can be expressed. The former knowledge, that is the design artefacts (such as lesson plans), can be represented as a sequence of activities through a chosen educational modelling language. ![[Pasted image 20250726144940.png]] The latter knowledge, the pedagogical theory, is harder to operationalise. Some attempt to bypass this difficulty by using the notion that embedded within the design are pedagogical qualities, such as the decision to recall the students prior knowledge, introduce learning objectives, or use example-based learning. Hence, they choose to share the designs themselves that act as a proxy to the pedagogy. In normal learning science papers, the expression of pedagogic theory is performed through natural language, which facilitates nuance and opens up the role for interpretation. This is very powerful in that the designer's implicit knowledge about the context can be used to guide the manner through which the theory can be appropriated to meet their specific goals. But for the accumulation of knowledge in a manner that can be derived at scale (RQ1) and reasoned upon to make use at scale (RQ2), demands a stricter and formal operationalisation of pedagogy for which there are two main approaches I can think off, pedagogical patterns and symbolic representations. **Pedagogical patterns,** represents abstract sequences of activities that present solutions to reoccurring problems. Whilst there is an inherent causal relationship, that is, if X problem, then use Y pedagogic pattern, often this knowledge of application is somewhat tacit and difficult to operationalise in a formal manner. That is, existing work on pedagogical patterns sees it as a manner through which we can accumulate libraries of patterns that rests partially on the designers capacity to make appropriate use of it. So whilst it is easier to collate as a library, if we are to accumulate vast amounts of more specific theory, it is much more difficult to effectively make use of. **Symbolic representations,** are much more demanding in the explicit operationalisation of theoretical insights. That is, it demands not solely the consequence of the relevant pedagogical patterns, but also the conditions under which such a pattern should be triggered with respect to some well-defined goal. Now the difficulty lies in the demand for such explicit operationalisation without excessive reductionism of education; which is a difficult task. Additionally, some may express scepticism for symbolic methods as being a step to the past, as being the driver of many old intelligent tutoring systems (ITSs) that were highly reductive of education [[@wattersTeachingMachinesHistory2023|(Watters, 2023)]]. Though in ITSs, symbolic methods were often used to represent both the **enactment** of the design (eg. generating natural language in a symbolic way for dialogue tutoring systems), as well as the pedagogical approach. This complexity, means that the pedagogical strategies had to be highly perspective and deterministic, leading us to design primarily with respect to simplest and most predicable form of pedagogical knowledge; that is, the learning primitives. But to separate enactment means that the symbolic representation of pedagogy can occur at a much higher-level of abstraction that could overcome some of these older issues of reductionism, whilst preserving the core benefits of symbolic representations. That is, a form that inherently facilitates interoperable reasoning with large amounts of knowledge. Additionally, I am not demanding any human to operationalise their own pedagogic intuitions and knowledge in a symbolic form; a very difficult task! But rather, the demands on humans are of expressing and enacting designs, which is much more natural. The derivation of theory is performed through the analysis of the results of enactment, with respect to the qualities of the design. Here the system should derive and represent theory in a manner that can add and accumulate to an extensive knowledge-base, which would otherwise be very difficult for humans in a manner that adheres to a formal syntax/vocabulary. Whilst there are many symbolic representations of knowledge, one that could be of particular use is answer set programming (ASP). It is a declarative logic programming language that is designed for knowledge representation and solving complex combinatorial search and optimisation problems [[@gelfondStableModelSemantics1988|(Gelfond & Lifschitz, 1988)]], which I see lying at the heart of education. Efficiency and optimisation is often seen as stripping the humanity and soul of education, and I believe these narratives are somewhat justified when looking the lens of educational history where efficiency was strongly connoted with reductivism with an obsessiveness of simple observable measures over all else. Though for most things, a child can learn, feel and reason in particular ways if given enough time, even if educational approaches are poor. Thereby the meaning of efficiency and effectiveness converge to the same idea, assuming that the manner in which education is measured is not overly reductive. #### 4.2 RQ1. How can we automate the experimentation processes for deriving contextual design-related pedagogical theories? I detail the idea with respect to the stages of (re)design, enactment and analysis. For each, I start by describing the automated approach, denoted by 'a)', before expanding on how humans can be involved, denoted by 'b)' and so on. ##### 4.2.1 (Re)design In the design stage, we do not solely perform design with respect to practice, but the design is performed in a manner that could lead to theory production. Therefore, in the design stage, we create formal variations of designs in such a manner that be can used to test proto-theories, that are, the initial hypotheses that guide the design. Hence, there are two primary challenges, that is to come up with suitable proto-theories and produce design variants that are useful for their testing. a) In a fully automated manner, the ASP representation of pedagogical knowledge opens up a search space with many possible design-related contextual pedagogical theories, which are represented as satisfiable answer sets. These theories can then be instantiated in application to designs to produce variations in activity orderings, substitutions and so on. Hence, the decision of the proto-theory could be done with respect to deriving knowledge that is most useful for narrowing the search space. Finally, the enactment and analysis of the design variants can both derive theory and also spawn further proto-theories for investigation. b) Though, with the search space being so vast, the intuitions developed by teachers and designer can be used to facilitate the selection of proto-theory for the satisfiable answer sets, in order to narrow our search space and direct our investigation into areas that may be more fruitful. That is, the designer's intuitions are heuristics. Given, the simplistic demands, teachers could also take the role of researchers, which is another approach to scale. c) Lastly, designers could express their own proto-theories that interest them. Given the formal nature of ASP, this could be performed through an chatbot interaction which derives relevant information that can be used to encode the proto-theories in a formal manner, and thereby produce relevant design variants. In many automated experimentation platforms for UI/UX, the humans are still responsible for generating the experiments that are of interest, with the platform simply performing the experiment. Whilst this is much more time consuming than the former a) and b), it can still save significant time for researchers of the learning sciences where setting up and enacting studies is greatly time consuming. Hence they could use it as an instrument for performing targeted and specific research. ##### 4.2.2 Enactment Enactment is the process of instantiating and performing the design. However, I am not proposing the automation of students. Rather, I question how enactment can leverage existing practice at scale to power the automated derivation of theory. Whilst many current approaches are time-consuming and requires researchers to find willing classrooms, discuss with teachers, and enact disruptive studies. I propose that we need to blend research and practice. That is, to harness the constant unrelenting flow of practice and insert the process of research in a natural and non-disruptive manner. a) In a fully automated manner, individual tutoring systems can be used to enact designs and their respective variations for activities like homework. Whilst this bypasses the educator and hence limits the types of derived theory (ignores social dimensions), it simplifies the need for a a teacher to conform to the pedagogical structures that we wish to test, but still allows for naturalistic conditions within individual studying contexts, such as homework or private study time. b) To improve the kinds of theory that can be derived, particularly in taking account of social theories of learning, requires enactment within social contexts by educators. This could be performed through integration with curriculum providers and creating relevant variations for theory production, since many teachers follow lesson plans and slides that are externally designed, thereby already conforming to existing structures; which produces a natural and non-disruptive flow into enactment. ##### 4.2.3 Analysis The data collected in the process of enacting the design, should be analysed with respect to the proto-theories, to generate theoretical insights that can be added to the pedagogical knowledge base. a) The lesson plan could contain formative assessments or activities that produce data, in a manner that can be mapped onto measurable qualities, which can be used as a proxy to measure the areas of interest. For example, our representation through EML allows a design to integrate pre/post tests. The difficulty lies in the value-laden tradeoff between producing designs that are most meaningful for student learning, whilst being able to elicit information for analysis. b) To improve data richness, we can elicit the practitioners insights, observations and opinions. This information could be translated by LLMs into a symbolic form that can be inductively reasoned upon with the pedagogical knowledge base to derive theory. c) In the most time-consuming manner, a researcher could be involved in interviewing, talking to the teacher, analysing data, and deriving insights. Which then through discussion with LLMs can be translated into a symbolic form for our collective interoperable knowledge. Whilst this is time-consuming, it may be be less time-intensive that the standard research processes, whilst being facilitating the generation of deep and specific theories that are of interest to the researcher's domain. #### 4.3 RQ2. How can we support learning designers in better making use of vast arrays of contextual design-related pedagogical theories? We can teach designers about frameworks and ways of viewing education, such as constructionism. However, it is impractical to teach the entirety of contextual design-related pedagogy, because this contextual nature leads to vast accumulations of theory. Rather representation with a formal EML, allows for the application of algorithmic methods to support their use. Here, I see two interconnected approaches pedagogical patterns and reasoning with symbolic representations. ##### 4.3.1 Pedagogical patterns Pedagogical patterns, as abstract sequences of activities that present solutions to reoccurring problems, can be accumulated as sets of libraries for specific use cases. After which metadata can be added to patterns to help denote their conditions for use and their relevant pedagogic properties. After which, the patterns can be made accessible to designers by means of filtering, sorting and searching. Ultimately relying on the designer's expertise and intuitions to make the final appropriate decision. ##### 4.3.2 Reasoning with symbolic representations Whilst patterns are used to represent design, the symbolic representation is used to encode pedagogical theory. The primary goal would be to produce a relevant bridge between the two. That is, by reasoning through the symbolic representation of pedagogical theory, with respect to contextual-conditions and objectives, we can derive appropriate pedagogical patterns for effective design. With optimisations in ASP, we can not only find satisfiable answer sets, but ordered with respect to reasoned appropriateness in a given context. These can be used as suggestions from which the designer can make the final decision with respect to their implicit and tacit pedagogical knowledge. ## 5. What could this mean? Education is a social science. However within specific areas of pedagogy, there lay consistent cause and effect structures which can be reasoned upon. That is, irrespective of whether the purpose of education, the role of technology or otherwise, sits within the paradigm of interpretivism, pragmatism or critical theory; laden within design-related contextual pedagogical theory is logical and empirical positivism. Much of the success of the formal (logical) and natural (emperical) sciences comes in the form that knowledge can build upon each other and accumulate. Hence, certain forms of pedagogical theory that operate at lower levels of abstraction closer to the design space, with less role for interpretation, could have similar accumulative effects. Thereby motivating this idea to build a knowledge base of theory in a faster manner (RQ1), that can actually interoperate by rationalist methods to simplify dissemination of theory in practice (RQ2). If such ideals are realised, it can aid in closing the gap between research and practice, where communities can not only work together collaboratively, but cooperatively! This could aid in educational technology becoming a field that can truly build and accumulate knowledge, through interoperability and extensibility, rather than being a disperse set of inconsistent ideas. Though following our prior analogy of harnessing a waterfall of practice to turn a turbine of theory production which is applied to improve water quality upstream, is not a simple activity; rather, a large infrastructure project. It requires: having relevant stakeholders on board, developing systems of theory production, learning design tools to apply theory, infrastructure for scalable deployment, usability and more. It is an interdisciplinary learning engineering approach which extends beyond solely academic expertise. Whilst such grand infrastructure visions and investments are seen in the fields of engineering and natural sciences, it is seldom seen within educational technology. Though of course, such scepticism is justified for education where there are ethical tradeoff of diverting limited taxpayer money from other highly certain methods of improving education. To divert significant resources with risky dreams is reckless, as we have seen with the effects of one laptop per child [[@amesCharismaMachineLife2019|(Ames, 2019)]]. Hence, research can start with smaller prototypes to test feasibility and the quality of derived theory. Though, such investment should be made with the understanding that the potential cannot be seen without testing its ability to produce network effects. Without a platform as a vehicle of research, we will be constrained to prototyping and testing theories with great effort to derive theory that remains in natural language and seldom used, with all engineering efforts thrown away as the next project repeats the cycle. This idea aims to break this cycle and challenge our epistemological capacity of pedagogical discovery, operationalisation for accumulation of theory and its dissemination in practice. ## References Abolhasani, M., & Kumacheva, E. (2023). The rise of self-driving labs in chemical and materials sciences. _Nature Synthesis_, _2_(6), 483–492. [https://doi.org/10.1038/s44160-022-00231-0](https://doi.org/10.1038/s44160-022-00231-0) Ames, M. G. (2019). _The charisma machine: The life, death, and legacy of one laptop per child_. Mit Press. Dalziel, J. (2013). The LAMs community: Building communities of designers. In _Rethinking Pedagogy for a Digital Age_ (pp. 254–267). Routledge. [https://www.taylorfrancis.com/chapters/edit/10.4324/9780203078952-28/lams-community-building-communities-designers-james-dalziel](https://www.taylorfrancis.com/chapters/edit/10.4324/9780203078952-28/lams-community-building-communities-designers-james-dalziel) Dillenbourg, P. (2015). _Orchestration Graphs: Modeling Scalable Education_. Amazon Digital Services LLC - Kdp. Gelfond, M., & Lifschitz, V. (1988). _The stable model semantics for logic programming._ _88_, 1070–1080. Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: survey and practical guide. _Data Mining and Knowledge Discovery_, _18_(1), 140–181. [https://doi.org/10.1007/s10618-008-0114-1](https://doi.org/10.1007/s10618-008-0114-1) Laurillard, D. (2013). _Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology_. Routledge. [https://doi.org/10.4324/9780203125083](https://doi.org/10.4324/9780203125083) Papert, S. (1980). _Mindstorms: children, computers, and powerful ideas_. Basic Books. Postman, N. (1998). _Five things we need to know about technological change_. Sandoval, W. (2014). Conjecture Mapping: An Approach to Systematic Educational Design Research. _Journal of the Learning Sciences_, _23_(1), 18–36. [https://doi.org/10.1080/10508406.2013.778204](https://doi.org/10.1080/10508406.2013.778204) The Design-Based Research Collective. (2003). Design-Based Research: An Emerging Paradigm for Educational Inquiry. _Educational Researcher_, _32_(1), 5–8. [https://doi.org/10.3102/0013189X032001005](https://doi.org/10.3102/0013189X032001005) Watters, A. (2023). _Teaching Machines: The History of Personalized Learning_. MIT Press.