20 3 / 2012

08 3 / 2012

Prototype 2 Testing

PROBLEM

We are trying to solve the problem of classroom stigma associated with teacher feedback and question-asking.

USER GROUP

Stanford undergraduates, potentially in large, technical classes, who feel uncomfortable offering open feedback.

We identified this group based on interviews with lecturers, TAs, and especially undergraduates themselves. They helped us to map out the conditions in which they are most and least comfortable raising their hand to ask a question or offer feedback.
OTHER EXISTING WORK

Current work aimed at increasing teacher feedback is primarily focused on increasing student cognitive engagement. The intent of these systems is to keep students engaged by answering questions, with a secondary benefit of giving the teacher feedback on in-the-moment student understanding.
However, these systems fail to address two needs: the needs of students to feel safe and not judged for actually asking a question by raising their hand during class, and the need of the teacher for post-lecture data about student understanding.
EMPATHY / OBSERVATION

Most of our observations come from the CS109 class (“probability for computer scientists”). One of our team member (Bertrand) is currently taking that class. We interviewed several students and TAs to get insights on how to improve the way the class is taught. Our insights are summarized in our previous blog posts.
POVs
  • “I don’t want to be ‘that guy’ in class who is holding everyone back - I want to do something that helps me but is also good for everyone else.”
  • “What if I could give stress-free, anonymous performance feedback to the teacher and have them act on it?”

PROTOTYPE 2

USER

Student in class who is afraid of becoming “that guy” who’s holding everyone back, by asking questions of the lecturer.

NEED

A way to give anonymous, immediate feedback that the lecturer can act upon immediately.

INSIGHT

Our last prototype was investigating how students could monitor the attitude of the whole class, and see if the pace of the teacher was too fast, using a phone-based “clicker” and a laptop facing the class, but the lecturer never saw this data. We wanted to specifically look at the dynamics of using the same clicker system to provide feedback to the presenter on where students were confused.

We tried out our system in a small lecture given by one of David’s labmates. The lecturer could see a plot of when during his talk students were having the most confusion; the audience had no feedback, other than knowing when they themselves pressed “help” on the clicker page.

NEW INSIGHTS

  • Audience felt guilty about using clickers: “if I’m confused, I should just ask a real question!”
  • Attendees did not use clickers very much. A class-facing component might be necessary to keep faith in the system.
  • Presenter did see chart clearly, but they’re so nonspecific that he did not know how to act upon them during the lecture. “I didn’t know what to re-explain.” Presenter though that this would be more useful after lecture, when he would have time to re-assess the specific trouble spots.

ISSUES

  • Prototypes need to be tested in environment that mimics the intimidating environment of a large intro class; we were in a smaller, optional lecture, that was very low stress.
  • Presenter might need training as to how to deal with immediate feedback usefully.
  • The system needs both prototypes simultaneously to be functional; lecture environment can be changed more if there is an audience-facing screen, lecture confusion can be reduced more if there’s a professor-facing screen.

02 3 / 2012

Why Don’t Students Ask Questions?

We wanted to further investigate why students do not give immediate feedback to the teacher in a lecture or class, and Bertrand and Daniel went out and interviewed new students to understand their mindset:

NEW OBSERVATIONS

  • "I never raise my hand."
  • "I’m slow, so it’s difficult to quickly come up with a good question."
  • "I’m okay asking questions 1 on 1, but I don’t do it even in small groups."
  • "Often I have a basic conceptual question and I don’t want to ask it because I should know the answer."
  • "It would be helpful to know what the rest of the class thinks." Repeated by all students!
  • Big lecture < small classroom < office hours if you want to comfortably ask questions.
  • Reasons for not speaking up: embarrassment in front of other students who don’t know them, don’t want to take up other students’ time.

NEW INSIGHTS

  • Students generally aren’t comfortable giving direct performance feedback to the teacher, especially during class; they want to do it privately and anonymously to save face for themselves and the teacher. 
  • Students are more comfortable asking questions if they know that others have the same question.
  • Students are more comfortable asking questions if they know the other students personally.
  • It’s hard to speak up during a lecture for two reasons - classroom culture, and size / anonymity.

FINAL POVs

  • "I don’t want to be ‘that guy’ in class who is holding everyone back - I want to do something that helps me but is also good for everyone else."
  • "What if I could give stress-free, anonymous performance feedback to the teacher and have them act on it?"

NEW PROTOTYPE

We think these new insights give us some ways to improve our previous prototype “clicker” system to include a way to give the professor immediate feedback. Students still are able to press a “help” button on their clicker or phone and a program records when during the class this button was pressed.

We show the professor, in lecture, a graphic that shows “student confusion” as a function of time. They can see how well understood their comments just were, and it allows them to go back after class and look at what parts of their presentation need to be clarified and improved, by looking at what timestamps had the most confusion.

This professor-facing interface is added to the student-facing one that shows the whole class the current level of “student confusion” to still address the insight that students will feel more comfortable being active learners if they can see the status of their peers.

REMAINING ISSUES

  • If students have their phones out, will be tempted to goof off? Prevent students who leave the app from giving feedback?
  • Is a generic indicator of “confusion” enough information to convey to students about their peers?
  • What about students who don’t have smart phones?
  • Students might down-vote difficult sections that are still necessary. They are not the best judges of learning content.
  • Feedback isn’t constructive. This puts a lot of demand on the teacher to be a good performer; will this frustrate teachers who get bad feedback?

TESTING

We have scheduled testing the teacher-facing part of new prototype in two smaller, seminar settings this coming week. We’ll report back.

23 2 / 2012

CS109 Students Want the Teacher to Slow Down!

CS109 Students Want the Teacher to Slow Down!
 
 
What is currently happening in the field? 
General trend of ACTIVE / PARTICIPATORY learning for cognitive engagement:

  • Clicker / Personal Response systems widely used
  • Also some clickers that communicate student understanding to teacher, but not necessarily to other students
  • Nothing addressing the stigma of asking a strange question in class… nothing helping students to understand each other’s collective knowledge or to engage the teacher in cooperation


 
How did you observe during the EMPATHY phase? 
TA office hours / lectures / interviews with students. Large lecture hall with hundreds of students. In situ, totally appropriate location and user groups. We were not intrusive. Students in lecture were watching and taking notes or surfing the web; sometimes students ask overly-advanced questions that aren’t helpful for the class. Students in office hours were working, getting help from the TA, or waiting.


SYNTHESIS: What were your most interesting/ surprising / generative INSIGHTS from your EMPATHY phase?

Student 1:

  • “There is a crucial moment when you learn a new formula, and you have to stop listening to the professor in order to mentally compute each steps.”
  • “Lack of big picture; you’re just flooded with information and you need to recall as much stuff and procedures as possible.”
  • “Video tapping is helpful because you can stop and do some consult other resources when needed.”
  • “I love when the teacher says a joke after a complex equation, because it gives me time to conceptualize it.”
  • “A huge cognitive load comes from the fact that you have to mentally replace the indices and variables in addition to knowing what they represent; writing their name in terms of the problem (concretely) helps a lot.”



TA:

  • “Students usually get it when I explain a concept to them; at least for a moment. The insight often doesn’t last.”
  • “I try to make problems more familiar to students by using concrete examples, or by breaking it down in smaller problems.”



Student 2:

  • “The class is very useful but also painful. I feel like I’m really spending a lot of time on it.”
  • “I need to look at the slides 3-4 times before class and again 3-4 times after class to feel like I have understood the lesson.”
  • “Students have to learn to ask conceptual questions to the TAs; office hours have become sections because there is too much content in a lecture.”
  • “It’s easier for people who are always doing maths; for others, there is a huge learning curve.”



What POVs came out of your EMPATHY and SYNTHESIS work?

POV 1
INSIGHT: People joke about not having time to write stuff down, but they don’t ask the teacher to slow down!
NEED: CS109 student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb.
POV: “I don’t want to be ‘that guy’ in class who is holding everyone back - I want to do something that helps me but is also good for everyone else.”

POV 2
INSIGHT: Teacher doesn’t realize that he’s going too fast.
NEED: Feedback from student. A way to create a classroom culture in which it’s okay to ask the teacher for things.
POV: “I’m doing the best I can for my students, and I don’t want any assumptions to hold us all back from creating a great learning experience.”

BRAINSTORMING SOLUTION:
We want to come up with a way to have students loose their inhibition towards asking the professor to slow down, and this will probably happen if they realize they’re not alone in that desire. Let’s give students a way to discreetly let the rest of the class know that they feel rushed and want the lecture to slow down; clicker systems have been used for similar feedback before, so let’s make one with that goal.

PROTOTYPE 1:
NEED: CS109 student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb.
IDEA: Give students clickers so they can show they want the professor to slow down; then show the dynamic results of this back to all the students to show they’re not alone in that desire.
VARIABLE: Will students ask the professor to slow down if prompted by a passive indicator?

Preliminary feedback from users:

  • students may spam the system; we need a “mature” audience to test it 
  • showing the screen to students only is an indirect way to slow down the teacher
  • (red) coloring is a strong feedback

23 2 / 2012

CS Probability Students Are Overwhelmed by Complex Slides

CS Probability Students Are Overwhelmed by Complex Slides
 

What is currently happening in the field?
Intervention often involves additional activities (e.g. hands-on) or simulation. Most approaches don’t focus on the notation.



LOCATION
We attended TA office hours, lectures, and conducted interviews with students.
Lectures are often where learning first occurs, and we want to solve problems of understanding ASAP.

WHO
We observed students, TAs, and the professor. They were either engaged in group discussions, tutoring sessions or giving a lecture to an auditorium (in the case of the professor). Students interact with each other mainly by drawing equations or diagrams on paper; TAs use whiteboard to go through a problem; the professor of CS109 only uses PowerPoint slides to present content.

WHAT
We observed that TAs have to re-conceptualize a lot of the course content because the teacher goes over the slides pretty quickly. Students don’t have enough time to digest every equation, or they may understand them for a moment but forget their meaning after class. It was also interesting that most questions asked in class were high-level questions (details of a proof, and so on). There were very few “simple” conceptual questions.



SYNTHESIS: What were your most interesting/ surprising / generative INSIGHTS from your EMPATHY phase?
Student 1:

  • “There are too many indices and too many random letters in formulas, which makes them incredibly difficult to understand.”
  • “Lack of big picture; you’re just flooded with information and you need to recall as much stuff and procedures as possible.”
  • “Video tapping is helpful because you can stop and do some consult other resources when needed.”
  • “I like exercises that relates to things I knows (e.g. eye color) as opposed to hash tables.”
  • “A huge cognitive load comes from the fact that you have to mentally replace the indices and variables in addition to knowing what they represent; writing their name in terms of the problem (concretely) helps a lot.”



TA:

  • “Students need to see the concept applied to several domains in order to grasp it; they generalize by seeing a pattern.”
  • “Students usually get it when I explain a concept to them; at least for a moment. The insight often doesn’t last.”
  • “It’s difficult for students to generalize from one example; you change a small thing and they lose it.”
  • “I try to make problems more familiar to students by using concrete examples, or by breaking it down in smaller problems.”



Student 2:

  • “I need to look at the slides 3-4 times before class and again 3-4 times after class to feel like I have understood the lesson.”



Students seem to be having specific trouble digesting equations, specifically in the context of a fast-paced lecture.

What POVs came out of your EMPATHY and SYNTHESIS work?
POV 1 
Like a dirty window, mathematical notation needs to be “cleaned” to reveal the model underneath.

POV 2 
Cute equation has a crush on student, but doesn’t know how to express its feelings in plain English.

POV 3 
Programmers use tools to help them read code more quickly (e.g. syntax highlighters, adding/reading comments using a documentation, playing in a sandbox); similarly, students need cognitive crutches to help them read math more quickly and efficiently

BRAINSTORMING IDEAS:
The professor should add “metadata” to slides so that students can access it during the lecture. The students would be able to pull up these augmented slides, and personally look at the extra information in them, so they don’t clutter the main slides.

PROTOTYPE 1:
INSIGHT: Inability to translate mathematical notation in lecture slides impairs students’ understanding of the content taught; give students immediate explanations of mathematical terms.
NEED: Students need to digest information in slides quicker
VARIABLE: Do students better understand “plain English” descriptions of terms?

Not tested in CS109 yet.

http://chasingflow.com/test/binom.html

PROTOTYPE 2:
INSIGHT: Inability to translate mathematical notation in lecture slides impairs students’ understanding of the content taught; give students immediate explanations of mathematical terms.
NEED: Students need to digest information in slides quicker
VARIABLE: Do students better understand visual descriptions of terms?

Not tested in CS109 yet.

http://stanford.edu/~selassid/vis/slidevis/

08 2 / 2012

We worked on a second-generation sticky note prototype that introduced explicit problem solving and goals into the prototype. We presented a word problem involving the ideal gas law that had a specific goal of having pressure reaching a specific value with some constraints. The goal was visually represented by a dotted line around the P that shows the size the symbol needs to reach to achieve the goal. We also had hidden hints that would be revealed if the user violated constraints of the problem.
Some insights and conclusions from this prototype:
How do we convey that a variable CAN in principle be changed, but that it SHOULDN&#8217;T according to the assumptions of the problem? (Put another way, how do we guide the user to actively pin down a variable that should be pinned?)
How do we effectively relate violation of constraints back to the text of the word problem?
When should hints be revealed?
Having an explicit goal channels users desire to &#8220;break the system&#8221; into solving a problem and might reduce the flexibility expected by the system.

We worked on a second-generation sticky note prototype that introduced explicit problem solving and goals into the prototype. We presented a word problem involving the ideal gas law that had a specific goal of having pressure reaching a specific value with some constraints. The goal was visually represented by a dotted line around the P that shows the size the symbol needs to reach to achieve the goal. We also had hidden hints that would be revealed if the user violated constraints of the problem.

Some insights and conclusions from this prototype:

  • How do we convey that a variable CAN in principle be changed, but that it SHOULDN’T according to the assumptions of the problem? (Put another way, how do we guide the user to actively pin down a variable that should be pinned?)
  • How do we effectively relate violation of constraints back to the text of the word problem?
  • When should hints be revealed?
  • Having an explicit goal channels users desire to “break the system” into solving a problem and might reduce the flexibility expected by the system.

06 2 / 2012

One of our brainstorming ideas for helping students understand equations was to visualize how changes in the relative magnitude of some terms cause changes in others. We wanted to test out how this would function as an interactive system and see if students engaged with the method of interaction.

Some insights from this prototype:

  • There’s not one “intuitive” way that the other, unfixed variables should react to the user changing one. What’s best might depend on the specific problem being used.
  • There are many subtleties of the physical interaction that need to be sorted out. The user expects the equation to stay approximately centered while it’s being manipulated.
  • There needs to be a visual indicator of constants that shows that they can’t be varied, since that’s not evident from the variable letters and is confusing, since they don’t work like the other variables.
  • There will be some difficulty in implementing an arbitrary equation solver if we want to allow the manipulation of the equation in an interactive environment.

03 2 / 2012

Here is our in class equation manipulator prototype. We wanted to see how a user would interact with the general idea of the system of making an equation tangible. We used the ideal gas law and manually swapped out letters of varying sizes to show the desired behavior; if you made a letter bigger, the non-fixed variables would change size to maintain the equality. We used pins to allow the user to fix a variable at its current value.
With a little explanation, other d.science students understood the concept and seemed to find it fun.
The sticky-note prototype revealed design challenges that we didn&#8217;t foresee:
How do we convey that a variable can change, but only as an indirect result of the user manipulating other variables?
Users tried to &#8220;push the system&#8221; and explored interesting combinations in an attempt to make it &#8220;break,&#8221; e.g. pin &#8220;too many&#8221; variables and then try and change others, or move the letters around.
Users expect extreme flexibility from a physically-inspired system; everything needs to be able to move as in real life. Users thought they could algebraically rearrange the equation, even though we didn&#8217;t expect them to.

Here is our in class equation manipulator prototype. We wanted to see how a user would interact with the general idea of the system of making an equation tangible. We used the ideal gas law and manually swapped out letters of varying sizes to show the desired behavior; if you made a letter bigger, the non-fixed variables would change size to maintain the equality. We used pins to allow the user to fix a variable at its current value.

With a little explanation, other d.science students understood the concept and seemed to find it fun.

The sticky-note prototype revealed design challenges that we didn’t foresee:

  • How do we convey that a variable can change, but only as an indirect result of the user manipulating other variables?
  • Users tried to “push the system” and explored interesting combinations in an attempt to make it “break,” e.g. pin “too many” variables and then try and change others, or move the letters around.
  • Users expect extreme flexibility from a physically-inspired system; everything needs to be able to move as in real life. Users thought they could algebraically rearrange the equation, even though we didn’t expect them to.

31 1 / 2012


Here&#8217;s a picture of our results from our final group brainstorming session with Rio, Emilio, and Theresa.
We eventually honed our original POVs. For our &#8220;enhancing group collaboration&#8221; domain:

Students learn better if the classroom is not a coffee shop.

And for our &#8220;making mathematical models understandable&#8221;:

Students understand a model if they have to take it apart and build it back up again, like a lego house.

We explicitly decided to tackle the questions:

How do we encourage students to teach each other?
How might we better communicate the concept of pressure in gasses?


Some of the brainstorming highlights for our &#8220;encouraging collaboration&#8221; were:
Some sort of &#8220;knowledge economy;&#8221; how do we get students to trade explanations with each other?
Grading on explanations; have part of a students grade depend on how well they explained a concept.
Ponzi knowledge; allow people to work together and attribute their results to other people; somehow encourage students to have their explanation used the most.
"Group annotated handouts;" allow all students to take notes on a homework PDF and easily share questions and explanations.

For our &#8220;explaining pressure&#8221; work:

A &#8220;manipulatable equation&#8221; where the symbols can be stretched and squashed and how their value effects the value of other variables is shown by others&#8217; sizes changing.
Using an analogy to buoyancy. Unfortunately it&#8217;s not physically perfect.
Make a game where you need to achieve some physical goal which requires pressure to be a specific value; have a balloon blocking a door, and you have to shrink it to get past.
Analogies work best when students can relate, so let&#8217;s create an experience in the classroom, so everyone can relate, then use the analogy; like bake muffins and have them rise, then relate to gas expanding.
We were really inspired by the &#8220;manipulatable equation&#8221; and are planning on prototyping some of the ideas surrounding the manipulation interactions, goal-driven problems using it, and relating the equation to a physical system. The &#8220;economy of knowledge&#8221; ideas was also very interesting, but we are having difficulty coming up with specific implementations that don&#8217;t drastically mess with the concept of grading and that can be incorporated into a class simply.

Here’s a picture of our results from our final group brainstorming session with Rio, Emilio, and Theresa.

We eventually honed our original POVs. For our “enhancing group collaboration” domain:

Students learn better if the classroom is not a coffee shop.

And for our “making mathematical models understandable”:

Students understand a model if they have to take it apart and build it back up again, like a lego house.

We explicitly decided to tackle the questions:

How do we encourage students to teach each other?

How might we better communicate the concept of pressure in gasses?

Some of the brainstorming highlights for our “encouraging collaboration” were:

  • Some sort of “knowledge economy;” how do we get students to trade explanations with each other?
  • Grading on explanations; have part of a students grade depend on how well they explained a concept.
  • Ponzi knowledge; allow people to work together and attribute their results to other people; somehow encourage students to have their explanation used the most.
  • "Group annotated handouts;" allow all students to take notes on a homework PDF and easily share questions and explanations.
For our “explaining pressure” work:
  • A “manipulatable equation” where the symbols can be stretched and squashed and how their value effects the value of other variables is shown by others’ sizes changing.
  • Using an analogy to buoyancy. Unfortunately it’s not physically perfect.
  • Make a game where you need to achieve some physical goal which requires pressure to be a specific value; have a balloon blocking a door, and you have to shrink it to get past.
  • Analogies work best when students can relate, so let’s create an experience in the classroom, so everyone can relate, then use the analogy; like bake muffins and have them rise, then relate to gas expanding.

We were really inspired by the “manipulatable equation” and are planning on prototyping some of the ideas surrounding the manipulation interactions, goal-driven problems using it, and relating the equation to a physical system. The “economy of knowledge” ideas was also very interesting, but we are having difficulty coming up with specific implementations that don’t drastically mess with the concept of grading and that can be incorporated into a class simply.

    26 1 / 2012

    Our initial point-of-view generation session board. We came up with two main POVs.
One for our domain of helping teachers triage student help:

How can teachers supervising group work take inspiration from battlefield triage?

One for helping students understand equations:

Cute equation has a crush on student, but doesn&#8217;t know how to express its feelings.

    Our initial point-of-view generation session board. We came up with two main POVs.

    One for our domain of helping teachers triage student help:

    How can teachers supervising group work take inspiration from battlefield triage?

    One for helping students understand equations:

    Cute equation has a crush on student, but doesn’t know how to express its feelings.