Technical literacy for educators — understanding how adaptive learning, recommendation, and tutoring systems shape student learning
An adaptive learning platform promised personalized learning for every student. The system used AI to recommend which lesson each student should study next based on their previous performance. What the system actually did was more complex: it collected granular data on every student action, tracked what they did and how long they spent on each problem, built models of student knowledge, and determined the optimal next problem to maximize the chance of correct performance. This worked — the system got students to solve more problems correctly. But teachers noticed something else: some students became dependent on the system, waiting for recommendations rather than taking initiative. Some students who struggled with a concept stopped trying once the system routed them away from that topic. The system had optimized for short-term performance, not long-term learning or independence.
The educational AI had worked technically. It had not worked educationally.
Educational AI systems operate on a pipeline: Data collection: Every interaction a student has is data. How long they spend on a problem, whether they get it right, whether they ask for hints, whether they give up. Student modeling: AI infers what the student knows and does not know from their behavior. If a student consistently gets algebra problems wrong but geometry problems right, the system infers they understand geometry but not algebra. Recommendation: Based on what the system thinks the student knows, it recommends what they should study next. The goal is typically to maximize some metric: correct answer rates, time-on-task, learning gains. Adaptation: The system changes its behavior based on student performance. It might make problems easier if students are struggling, harder if they are succeeding, or recommend different content entirely.
Educational AI systems are often optimized for metrics that are easy to measure: correct answers, time on task, engagement. But these metrics do not always align with education goals. A system that maximizes correct answers might be doing so by steering students away from challenging material. A system that maximizes engagement might be addictive rather than educational. Educators need to understand what the system is actually optimizing for and whether it aligns with their learning goals.
Collecting granular data on student learning reveals patterns, but it also creates risks. Privacy: Detailed learning data is sensitive. What topics a student struggles with, what they research, how long they take to answer questions — this is personal information that needs protection. Bias: If student modeling systems are trained on historical data, they perpetuate historical biases. If students from certain groups have historically been tracked into certain courses, the system will recommend those same courses to new students from those groups. Surveillance: Monitoring every student action creates a surveillance environment. Students may feel watched and may change their behavior (being less willing to try hard problems if their struggles are recorded). Data persistence: Once data is collected, it persists. A student who had a bad year in math might be tracked by the system's inferences about their math ability for years afterward.
Educators do not need to implement AI systems. Educators need to understand how AI systems they use shape student learning. What data are they collecting? What is the system optimizing for? What metrics would prove the system is actually supporting learning, not just measuring performance? These questions are where educator expertise is essential — not computer science expertise, but expertise in learning.
Understand how an edtech AI system works and what impact it has on learning
You are analyzing an educational AI system your school or institution is considering using. Your job is to understand what it actually does and what impact it will have on student learning. Choose a real edtech platform with AI (Duolingo, Khan Academy, Knewton, Coursera, or another).
For that system: (1) Describe what data it collects on students. (2) Explain what the system optimizes for (what metric guides its recommendations). (3) Identify potential impacts on learning — both positive and concerning. (4) List questions you would ask the vendor about the system before deploying it with students.
How teachers can create AI-enhanced learning experiences using existing tools — no engineering required
A high school English teacher wanted to give students more feedback on their writing. Manually grading 150 essays took 20 hours a week, so detailed feedback was impossible. She discovered that she could use an LLM to draft feedback on student essays — not to grade them, but to give her starting points for feedback. She built a system: students submit essays through a form, an LLM generates initial feedback on structure, grammar, and argument clarity, she reviews and personalizes the feedback, then returns it to students with her own voice and expertise. The system did not replace her judgment. It freed up time so she could provide better feedback than she could before. Her students received more feedback, and she spent less time on the mechanical parts of grading.
A teacher had built an AI-enhanced learning experience without a technical team.
Feedback generation: Teachers can use LLMs to draft feedback on student work — essays, problem solutions, projects. The AI generates initial feedback, teachers personalize it. This multiplies the feedback teachers can give. Adaptive content: Teachers can create learning materials that adapt based on student answers. A quiz where the LLM generates follow-up questions based on student errors. An interactive scenario where the AI responds to student choices. Assessment design: Teachers can use AI to generate test questions, rubrics, or assessment ideas. This is not replacing teacher expertise — it is multiplying it by providing starting points teachers can adapt. Differentiation: AI can help teachers understand what different students need. Analyzing student performance data to identify patterns that suggest some students need more challenge, others need more support.
Educational AI works best when it amplifies teacher judgment, not replaces it. A system that auto-grades essays removes teacher judgment about what matters in writing. A system that auto-determines what each student should study next removes teacher knowledge of the individual student. Teachers must remain in control of the pedagogical decisions that matter: what goals matter, what constitutes good work, when a student needs intervention or support.
When teachers build with AI, they need to think about: Privacy: What student data is being collected? Where is it stored? Who has access? Bias: If using training data from previous years, are there biases in that data? Would the system perpetuate those biases? Accuracy: Is the AI good enough for its use case? Feedback on essays has lower accuracy than factual classification, so AI-generated feedback needs more teacher review. Transparency: Should students know they are interacting with AI? This depends on context — a tutoring chatbot might not need disclosure, but assessment should be transparent about how much is AI-generated.
Teachers building with AI are not implementing a technical system. They are designing a learning experience that uses AI as a tool. This means teachers' expertise — their knowledge of their students, their discipline, their pedagogy — is essential. The AI is valuable because it amplifies teacher decisions, not because it replaces them. This is only possible if teachers remain in control and understand what the AI is doing and why.
Create a learning experience that uses AI tools to enhance teaching — keeping teachers in control of pedagogy
You are a teacher designing an AI-enhanced learning activity. Your goal is to use AI to amplify your teaching, not replace it. You will specify what the activity does, how AI fits in, how you remain in control of the pedagogy, and what you are monitoring for.
For your activity: (1) Describe what you are trying to teach and who would benefit. (2) Specify where AI fits in and what it does. (3) Explain how you remain in control — what judgment stays with you. (4) Describe what you would monitor — what could go wrong, and how would you know? (5) Identify any privacy or bias concerns and how you would address them.
What learning science says about AI tutors, designing for transfer not just performance, and avoiding dependency
A school implemented an adaptive math tutoring system that proved remarkably effective at increasing test scores. The system personalized the difficulty of problems to each student, provided immediate feedback, and allowed students to practice indefinitely. Students' performance on the system's own assessment improved dramatically. But the learning science literature suggested concern. The system optimized for performance on specific problems, but did not optimize for transfer — could students apply what they learned to new problems? The system provided scaffolding (making problems easier) but did not train the independence students would need when support was removed. The school implemented the system more carefully, adding explicit transfer tasks, gradually reducing scaffolding, and combining AI tutoring with human instruction. The results were better — not just higher scores, but more genuine learning.
The system had improved performance. Redesigning for learning outcomes had improved actual learning.
Educational AI often optimizes for performance on a specific task. A student gets more problems right, their score goes up. But learning science distinguishes between performance — immediate success on practiced tasks — and learning — the ability to transfer knowledge to new contexts. Transfer: Can a student who learned to solve algebra problems solve word problems that require algebra? Can they recognize when algebra is needed in unfamiliar contexts? Systems that optimize purely for performance may not develop transfer. Maintenance: Will students remember what they learned after a time gap? Systems that provide continuous support may not develop memory that persists without support. Independence: Can students solve problems without the AI system? Systems that provide extensive scaffolding may create dependency rather than independence.
AI tutors often provide scaffolding — breaking problems into smaller steps, providing hints, allowing unlimited attempts. Scaffolding helps students succeed in the moment, but learning science suggests that some struggle is necessary for real learning. If the system makes success too easy, students do not develop the problem-solving strategies they need when support is removed. Effective learning design uses scaffolding strategically — providing support when students are overwhelmed, but withdrawing it gradually as competence develops, so students build independence.
The most effective AI-enhanced learning combines AI tutoring with human instruction. The AI can handle the personalized practice and feedback. The human can provide the big-picture explanation, the motivation, the connection to other learning, the feedback on metacognitive strategies. A system where students use an AI tutor for 20 minutes, then have 30 minutes of human-led discussion or project work, is likely more effective than either alone. This requires careful design — the AI and human instruction need to be integrated, not just added together.
The question educators should ask about educational AI is not "will this improve test scores" but "will this develop the learning outcomes I care about?" Test scores often improve with AI tutoring. But deeper learning — understanding, transfer, independence — requires educational design that attends to learning science, not just optimization of performance metrics. Educators are the experts in these design questions. Technology vendors are not.
Create the assessment and monitoring that will tell you whether an AI-enhanced learning experience actually improves learning
You are implementing an AI-enhanced learning experience and need to evaluate whether it actually improves learning. You will design a comprehensive evaluation plan that goes beyond test scores to measure the outcomes that matter.
For your evaluation: (1) Specify what learning outcomes you care about (beyond test scores — what about understanding, transfer, independence, motivation?). (2) Specify how you will measure each outcome. (3) Design how you will compare with and without AI (what is your comparison group?). (4) Plan for unintended consequences (what could go wrong with learning that would not show up in your main metrics?). (5) Commit to what will cause you to stop using the system if it is not working.
How to assess whether AI-enhanced learning is working, communicate about it with students and families, and improve based on evidence
A school had implemented an AI tutoring system that showed strong average improvements in math performance. The leadership team was proud of the implementation. But a teacher asked about equity: was the system helping all students, or only some? An analysis revealed that the system was helping high-performing students — they used it productively and showed strong gains. But lower-performing students were not using the system productively. They were getting frustrated by the difficulty and giving up. The system had been optimized for the middle, and was leaving behind students who needed support most. The school redesigned the implementation: lowering the starting difficulty for struggling students, adding human support alongside the system, and monitoring gains separately for different student groups. This required more work, but it meant the system was actually helping rather than amplifying existing inequities.
The system had looked like success without the disaggregated evidence revealing the full picture.
Overall improvements in student learning can hide serious equity problems. A system that improves average scores but helps only some students is not successful — it is amplifying inequality. Disaggregated analysis: Break down your data by student groups: gender, race/ethnicity, socioeconomic status, English learner status, students with disabilities. Does the system help all groups or only some? Usage patterns: Do all students use the system or only some? If certain groups use the system more or less, why? Outcomes by group: Do all students show learning gains or only some? If the system helps some groups but not others, it is not equitable.
Educational AI that amplifies existing inequities is worse than no AI. If a system helps high-performing students and leaves behind struggling students, it is making educational inequity worse. Before claiming success, educators must look at disaggregated data. If equity is not there, the system is not working no matter what the average numbers show.
Schools using AI should be transparent with students and families about how AI is being used. Student communication: Students should understand why they are using an AI system, what it is doing, and how it affects them. Family communication: Families should know what data is being collected, how it is used, who has access. Teacher communication: Teachers need to understand what the system does and how to interpret its recommendations. AI should not be a black box — for anyone. Transparency about limitations: Be honest about what the system does well and what it does not do. An AI tutor is not a substitute for a good teacher. It is a tool that works best in combination with human instruction.
Implementation of AI is not a one-time project. It is an ongoing commitment to understanding whether it is working and improving it if it is not. Regular data review: Look at disaggregated data quarterly or monthly, not just annually. Problems that emerge can be addressed quickly rather than affecting whole cohorts. Feedback loops: Ask students and teachers regularly: is this helpful? What is not working? Is the system causing unintended harm? Willingness to stop: If evaluation shows the system is not working or is causing harm, be willing to stop using it. Continuing a system just because you have invested in it is not good policy.
Educational institutions remain responsible for student learning, regardless of whether they use AI systems. An AI tutor cannot be blamed for poor outcomes — educators can only be. This is not a burden; it is the core of educational responsibility. Educators who use AI remain fully responsible for whether it helps students learn or amplifies inequity. This responsibility means evaluating evidence rigorously and being willing to change course when evidence shows the system is not working.
Design how you will communicate about AI use with students and families, and how you will evaluate effectiveness responsibly
You are implementing an AI-enhanced learning experience and need to communicate about it and evaluate it effectively. You will create both a communication plan and an evaluation plan that honor student and family rights to transparency and educators' responsibility for learning outcomes.
For your plan: (1) Write what you will tell students about the AI system — why you are using it, what it does, how it affects them. (2) Write what you will tell families — what data is collected, how it is used, what their options are. (3) Design your evaluation: what data will you disaggregate by? What will indicate the system is not working? (4) Commit to how you will iterate: how frequently will you review data, and what will trigger changes?