panel_llms_in_education.md

LLMs in Education: Opportunities and Challenges

• Opening
  • Personal story: used to be a tutor
  • Learn about different learning styles
  • Communicate the beauty of a topic, mystery, why it's useful
  • The news
    • 2008 study - effect on homework, had a larger effect than 2017 because of cheating
    • 2017 study: cheating became more widespread.
    • in Kenya, 20,000 people are employed to write essays
  • Don't Ban ChatGPT in Schools. Teach with it.
  • Potential
    • personal tutor for every student
    • personal coach for every parent
    • personal teaching assistant for every teacher (and homeschooler)
• Discussion Questions
• Panelists

Shawn Jansepar - LLMs in Education: Opportunities and Challenges

• Self-paced mastery based learning
• Biggest opportunity: A tutor for every student. An assistant for every teacher.
• Benjamin Bloom - 2 sigma problem - if students had a tutor they would be 2 sigmas above average
• Students asking questions of a character in a book
  • Ask questions of Charlotte or Louis (Trumpet of the Swan)
  • 1-1 help without a direct answer
• Provide immediate feedback on math steps
• Writing
  • co-writing with an AI (similar to doing math with a calculator), showing your work along the way
• Programming
• Why should I care about learning this?
• Frameworks
  • Passive, Active, Constructive, Interactive
  • Passive - watching a video
  • Active - highlighting, asking a tutor
  • Constructive - self-explanation, learner giving an explanation for the answer (could do this to the AI)
  • Interactive - debating with fellow students and teachers
• Teaching Assistant for everyone

Madeleine Udell - Assistant Professor of Management Science at Stanford

• What are the promises of LLMs for education?
  • Topic of research: can they do data science, solve machine learning, do optimization better. - Results
    • Speeding up feedback
    • Self-critique is important
  • Translate an idea or answer to a different grade level and given to someone's background
  • Tailored quizzes - we learn through making mistakes
  • Custom textbooks
  • Critique papers to speedup revision process
  • Generate examples for a lesson
• What are the challenges?
  • Don't know whether the LLMs output is true or just "truthy"
  • In some topics you can verify correctness (engineering, math, CS)
  • How do identify falseness when it's presented very coherently. We're used to trusting coherent explanations. This is not true anymore w/ LMs. How to learn from LLMs without absorbing falsehoods.
• How should humans and LLMs work together?
  • LLMs teaching humans
  • is the goal to have more information in human heads?
  • Example of doctors/LLMs - both did better separate, worse together because they didn't trust each other's evidence

Eric Horvitz

• ITS Conference a few years ago - Intelligence Tutoring Systems - vs proceedings in 2013 in Montreal were similar
• Top research challenges
  • developing and evaluating student/teaching facing systems
  • developing machinery with long-term memory & synthesis capabilities
  • understanding long-term impacts
• Long-term dream of ITS
  • provide highly personalized, adaptive guidance to learners
  • people are building very complex machines --> complicated to build & modify with feedback
• Pace, Precision
  • Can you explain this more slowly?
• Building a personalized courses of study [slide]
  • How is this subject / sub-subject important because of God?
• Toward Continual, Cross-Session Learning
  • Project from Microsoft: Semantic Kernel - to help build systems like this

Discussion

• Hallucinations are a problem. Question of evaluation in an educational setting? What kind of benchmarks should be created? If they have super-human intelligence, how can we recognize it?
  • Ed
    • Concepts were not learned by repeated exposure, but by repeated testing/retrieve
  • Shawn
    • Quant/Qual indicators
    • Hard to determine on a data-driven basis whether this is working or not
    • Qual: asking students for their feedback
    • Quant: leading indicators, are they getting more number of skills to proficient within a given session or time-frame
    • Want to do a longitudinal study
  • Eric
    • these models can be good pedagogy
    • have benchmarks for LLM abilities around education: dimensions of positive support, generate analogies
  • Madeleine
    • LLMs are so transformative that we may want to change the goal. Current goal is achieving the right answer or mastery of a subject. Maybe the goal can be using this subject/tool to achieve their goals.
• We're assuming humans in the loop. Do we need to have humans in the loop?
  • Humans in the loop slow things down. Building Teams example.
  • What if we had a model that was a 1st grader.
  • Ed: In development of Bard, we are using the teacher/student model. AI Content loops. In this situation, in the teacher/student loop, we need to know whether we're in a loop. Is the learner outcome the shared goal between teacher/learner goals? I want to challenge that assumption. In Taiwan, corporal punishment was used to ensure there were shared outcomes. The shared goal is not just better test scores and not being punished but being inspired.
  • Eric: Herb - there was hot learning and cold learning. The most memorable teachers made education disappear and inspired students. Do we understand the role of a human in that kind of engagement.
  • Shawn: Our goal at Khan Academy is not to replace teachers. Free up more of their time. Human not in the loop --> helps in the development process --> refine prompts. Using the LLM to speed up the evaluation of prompts on a set of test cases.
• We've been discussing LLMs as an assistant. But what if the student is directly using the LLM for other things?
  • With a graphing calculator - changed the way you learn graphing
  • Eric: mirrors the calculator debate in ~1975. Our grandchildren won't need to deal with this. [Lawrence - how do we get there though?]
  • Ed: Most schools do not allow students bring in mobile phones: cheating, distraction, etc. For adults, it's extremely difficult to do anything without our phone, because we've become augmented humans. When a student is waiting for the teacher, why don't we allow them to look up something on Wikipedia. Isn't the goal to be Will some schools ban LLMs? What is the goal?
  • Madeleine: The distraction issue is huge. Periodically 1/3 of the class erupts in laughter because of an Airdropped picture. Focus/attention --> are huge issues. The mechanisms we get around this are assessments. Timed exams are not matching real-world. We're talking about the happy path/ideal learner who is not going to go on Tiktok and YouTube.
  • Ed: sister is professor rhetorical writing in Ashville, NC. She is extremely concerned. She's been teaching 2 decades, thinking skills, show how you analyze and abstract. Her class are stretching the minds of students. She sees glazing over now.
  • Shawn: This will help us understand that the current education system doesn't work. Lead with project creation that leverages these skills.
  • Madeleine: We have to change assessments because none of the old ones will work in the LLM-era. My only ideas are projects and oral exams. Teach students high-level skills of project management without teaching them the base skills. Last quarter - you don't need to learn Pandas, just use Copilot. But the students wanted to learn Pandas.
  • Host: How we learn and what we learn is going to change. A liberal arts education is supposed to teach you how to learn.

Q&A

• Eric: We're suffering from denial of service of attacks on our attention!
• Thought-inducers vs. Thought-indexers.
  • Shawn: Failure is important. How can we shorten the feedback loop? [Lawrence - I wonder if shortening this feedback loop in an manager/employee setting]
  • Ed: Customization of the approach to teaching individual learners. The reality of our education system is our teachers are limited so we parallelize it 30x. LLMs can take on different styles of conversation (provoking, supportive). Positive reinforcement learning is one way. There's a real emotional component to educational. What would it be to have different styles of Khanmigo?
    • Shawn: A student asked Khanmigo to talk to him like he's a Gen Z. Used the term "Let's get that bread"
• What should kids learn in their educational upbringing in the era of LLMs?
  • Madeleine: Seeing kids take control of their learning, son is starting Kindergarten tomorrow.
  • [Lawrence - reading proficiency and speed to understand LLM output, knowing how to prompt the system and how LLMs work]
  • Ed: We tend to think of education as knowledge acquisition. One of the hats I can wear is a manager at a tech company. What skills do they have? Knowledge / technical skills are important. There are many other factors: relationship management, emotional intelligence. Are there relationship coaches in school helping students learn these things? Probably not. Maybe we shouldn't be teaching coaching syntax. Teach how to collaborate without being territorial, project management.
• My question is about the human experience? I learned about logorithmic table. There's a certain intuition that you get from that experience you don't get from a computer. You can extrapolate that to previous decades. Technology is chipping away at that human experience. Being a human now is different than 30-40 years ago. There's a siege on the human experience. Where do you think goes? [Lawrence - What a great question!]
  • Eric: Understanding the different ways curriculum is designed or why Hungarians and Romanians are good at math, to learn what the optimal way we sequence curriculum. But I agree we may be under siege.
  • Ed: One of the tasks of Bard is to generate poems. Crowd sourced workers, reach the limit of what they are capable. Who defines what a good poem? Part of being human is understanding beauty. So teaching beauty is important.
    • How do you evaluate beauty?
    • Does this cheapen valentines day in the future?
    • I have no idea. We are painting a rosy picture because the technology is so awesome and it cheapens our understanding of beauty.
• As a professor, encourage all students use LLMs. Once you understand more of this tool, educational gap. The students need to learn how to prompt engineering. This technological skill gap is huge. How as educators can we bring these tools to a wide-enough audience?
  • If you only submit the final report, you only get 50%.
  • Submit all of the interactions with the LLM as an appendix.
  • Shawn: There will be a diverse set of UIs like Khanmigo with a base prompt.