Conversational agent · Design & evaluation

BillBot

A course agent for students learning to justify instructional design decisions.

The question

How can an agent help a student connect a design choice to learning theory and explain the logic behind it? I was interested in what a student might do after receiving an explanation, especially whether they would apply or revise an idea.

My role and collaborators

I developed BillBot with William Wisser, starting from a practicum capstone and later integrating it into Canvas for T127. I curated course materials, wrote guidance for different types of questions, reviewed classroom feedback, and drafted an interview guide.

A decision that shaped the work

I wrote separate response guidance for course logistics and open-ended design questions. A logistics answer can point to the relevant course material. A design response can explain a principle and ask the learner to connect it to a choice they have made.

Why add a targeted follow-up?

In the planned comparison, both versions use comparable content and scaffolding. One version adds a targeted follow-up on eligible turns. That keeps the added question distinct from other changes in the response.

Bot walkthrough

I wrote this fictional dialogue to illustrate the design decisions.

Explain a concept

Learner
“What does it mean for my activity to use retrieval practice?”
Sample agent reply
“Give learners a chance to bring an idea to mind from memory. In your activity, where would they answer before they can look back at their notes?”
Design annotation

The response gives a definition the learner can use and asks them to locate a specific step in their design.

Apply it to a design

Learner
“I could ask them to explain a term before we review the reading.”
Sample agent reply
“Ask them to write their explanation with the reading closed, then let them revisit the text. What would you ask them to check when they compare their answer with the reading?”
Design annotation

The follow-up connects an activity choice to how learners will use the comparison.

Handle uncertainty

Learner
“Is this the part I’m meant to submit for the project?”
Sample agent reply
“Check the assignment instructions in Canvas for what to submit. If you are deciding between two formats and the instructions leave that open, ask the teaching team before submitting.”
Design annotation

For a course requirement, the response points to the authoritative source and leaves room for the teaching team to resolve ambiguity.

Artifacts

Draft media spaces for approved, annotated screenshots and a captioned demo.

Video demo pending
BillBot walkthrough. A captioned recording could show where a learner opens the agent and follows a design question.

Evidence, status, and limitations

Completed work
A classroom pilot, later deployment, and review of conversations and survey feedback to guide revisions.
Planned analysis
I drafted a turn-level coding approach linking student requests, agent responses, and the next student turn. A classroom comparison was also planned. Its implementation and results are pending confirmation.
Findings and limitations
Findings are pending confirmation. A visible revision or explanation in a chat can show what the student did in that moment. Durable learning would require separate evidence.

Related materials

Choose the next response lets you compare three authored feedback moves. Research projects includes the evaluation alongside other feedback studies.