The Pygmalion Effect

I think I've mentioned in previous posts that I'm earning a second masters (MS) in Instructional Design and Educational Technology, but a new update to that is I'm also earning a certificate in motivation + learning environments through the Educational Psychology department to coincide with my degree. My Ed Psych course for this semester is Seminal Readings in Education and Educational Psychology. So, I might blog about either Ed Tech or Ed Psych as I'm going along.

Image from theinsideouteffect.com

Today we discussed some readings we did on the Pygmalion Effect. This is the notion that preconceived expectations for others impact performance or an outcome, so it's the self-fulfilling prophecy. What's interesting is these preconceived expectations have the same effect whether they are self-generated or imposed by an outside source (though, naturalistic expectations are stronger). So, for example, foremen in a warehouse were told certain employees did good or bad on an exam for the job (regardless of how good or bad they actually did), and the foremen rated those employees who they believed to be smarter as better and more efficient. Another study experimented on mice (I am not a fan of this, but...) where mice were either lesioned through lobotomy or made to look like they were so mice handlers could not tell the difference. Handlers were told mice were either bright or dull regardless of lobotomy. Unsurprising is that lesion-free mice with handlers led to believe they were bright performed the best. What was surprising was that lesion-free mice with handlers led to believe they were dull performed just as poorly as lesioned mice who were determined to be dull.

We are looking at this research more directly related to classrooms and formal education next week, but there are huge implications. Visual cues in all of this are one of the most important factors. There is a study in psychology of "thin-slicing" (person perception based on superficial aspects in a short period of time.. so, first impressions, essentially) for student perceptions of teachers, where students watched 30 second clips of teachers teaching with no sound and were to rate their effectiveness as a teacher based on that video clip alone. The study found that the students watching the clips had nearly the same ratings of teachers as the students who actually completed the class and filled out TCEs.

Ambady, N., & Rosenthal, R. (1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology,64(3), 431-441.

Another (and very recent) study by Chia Jung Tsay did something similar where people would receive short clips with no sound of musicians competing in formal events, and they would need to predict who won based on the videos alone, hearing no sound. Accuracy in guessing was astounding, where visual impressions clearly had a greater impact than actual talent. When participants tried to base their ranking guesses on audio alone, they were not able to distinguish who won. Tsay points out that this "suggests that the visual trumps the audio, even in a setting where audio information should matter much more."

In looking at perception-of-self and perception of librarians by patrons, students, faculty, etc., this is important to think about (and something we are examining in the Librarian Wardrobe book). How we are perceived by others might influence how they evaluate us, and how we perceive others might influence how we evaluate them. If visual cues are especially important, then understanding how we present ourselves, whether in gesturing, other physical movements, or clothing, then studying how we dress and public perceptions would be quite significant.

IRB: Your research isn't Research

Image from Southern Fried Science

I'm writing this post in part to procrastinate getting to my 3rd year review packet candidate statement (overwhelming!) and also to share some interesting facts I learned from meeting with IRB on campus to discuss restrictions of my study on effectiveness of digital badges in an IL course for student success.

There was a lot of confusion at first when filling out IRB because we were so low risk, and then had to deal with so many restrictions. Our IRB rep explained that historically, any research involving human subjects needed to come through them, and this was a ton of work for them to handle even if research studies proposed didn't really need to be under their jurisdiction. More recently, "Research" with a capital R has been defined by the federal government as being generalizable (some of this info might be here, though our rep said there isn't one definitive page or site explaining this yet for the general public). This is wholly different from research that is not meant to be generalizable, for example: program evaluations, quality improvement, case studies, etc.

It was funny because after providing the explanation, our rep wanted to backtrack and say that she wasn't implying we aren't doing "research," we just aren't doing "Research." We in fact are doing program evaluation research because we are measuring a specific program at the UA in order to improve the program and will be showing what was successful/problematic for us. We would not be saying our results clearly apply to all IL programs or credit courses across the country. However, if we did want to try to prove that somehow, we would need to stay under their jurisdiction (we have been approved as Exempt level 2 I believe). If we were to stay under IRB, we have to keep them updated on any changes to methods and the form of consent for students, as well as if we were to make any changes in obtaining data. We would also have a ton of restrictions in what we can and cannot access with student info.

Filling out what is the "309 form" here at the UA to essentially rescind our IRB application (I am thrilled to do this, but it's probably not a good idea for me to think about how much time I spent on this for my own sanity) will then move us to program evaluation and we can essentially do whatever we want so long as it's generally ethical and follows FERPA regulations.

With IRB, we would have had to only do an opt-in to our study for students, anonymize data to potentially miss out on seeing trends, and be cautious about asking certain questions in our survey. As program evaluation, we can really gather data any way we want.

My co-researcher made a good point that the basis for Research vs research is in the eye of the beholder (the reader). We won't be saying our research is generalizable, but obviously if someone is reading the article (if we are able to get published, of course), that means they are considering how they might apply our findings to their own program or credit course. It can be very nuanced.

So anyhow, I just wanted to share this. It sounds like it's getting a big push not just on our campus but all over. This conflicts with LIS anxiety over publishing Research, but program evaluation is not any less important. The only distinction is that it does not go through IRB.

Grading and assessment, water and oil?

As I've been getting ready for ACRL Immersion 2013 Program Track (I leave in 1 week!), I've been finishing up a lot of readings on assessment. I've actually been really glad to read these articles, because as I've better solidified my notion of assessment through applying it to the instruction I am doing, I am finding my ideas are aligning with what I am reading.

The pattern I am finding in these readings is that assessment needs to be more holistic; assessment should be a method for students to learn rather than a focus on evaluation; and assessment should provide ongoing, meaningful feedback for students to practice instead of being judged.

We are wrapping up the summer semester with the badges pilot, and Purdue Passport incorporates assessment within earning a badge. Typically, a badge is given after a skill has been achieved, where assessment is more evaluative and judgmental rather than to provide feedback for improvement. This clashes with how I would prefer to teach and use the badges, so I've been using the feedback/assessment mechanism in Passport differently than it might be intended.

This is good because I think students are getting more out of the class, but also poses some conflicts:

1. If badges being awarded are not based on more rigid judgement of skill acquisition, how valuable are they?
2. On this note, how interoperable are they? Can their qualities be translated or compared to other institutions or libraries offering similar badges if desired evidence isn't as clearly enforced?

Because this is a credit class, grades need to be tied to student work. For this, the badges are essentially pass/fail. You either earn the badge or you don't. If a student is late in finishing badge work an exception is made to give them half off, but this is the only partial credit awarded. There are pros and cons to this as well:

Pros: Students can take risks in their responses and have less fear of failure (this positive aspect is rooted in game mechanics); I can focus more on the quality of my feedback rather than what level of good or bad the student's work falls into

Cons: How is good student work differentiated from bad work? Particularly if bad work is due to sloppiness or disinterest. Shouldn't a student who submitted excellent work (or evidence) for a badge be awarded the badge, where less stellar work would not be awarded the badge? Isn't the purpose of awarding badges to demonstrate that a skill was successfully acquired?

I have such mixed feelings on this. But one feature of Passport is to allow students a re-do. I use this often for sloppy work. I will leave feedback explaining exactly what I'm looking for and give the student a second chance (next semester I will be sharing specific rubrics for each badge with students so they have an even better concept of what level of understanding is desired). 

I am not a stickler on lower-level concepts like formatting a citation perfectly or memorizing exact steps on how to find an article when you only have a citation (these are specific assessments in the class to address more basic skills within learning outcomes). If a student has most of a citation right but forgets to italicize the journal title for MLA style, it's really just busy work for them to make them re-do it or for me to take points off. I leave feedback letting them know they mostly got it and to remember to double check these things for formal papers; and then I give them all the points. I love Barbara Fister's 2013 LOEX Keynote (in fact, my team read it as part of our strategic planning for the new fiscal year). I agree so strongly with her whole presentation, and using a specific example here, "very rarely outside of school are citations needed." I care way more about if students are able to understand what the purpose of a citation is and to incorporate this into their new understanding of "research as conversation" than about styles and how to format.

One assigned article that has been part of this class for a long time is a CQ Researcher article on cheating: why students cheat and how they cheat. It's interesting to see what students agree with in their reflections and a number do say that when a student doesn't feel course material has real application in their lives (or when an instructor provides little to no meaningful feedback), a student has no motivation or investment to put in quality work, and so cheating is easy. Focusing less on grades and more on understanding and a conversation between the students and us as instructors creates a richer experience for all. Their reflections resonate well with what we're doing in the course to make it apply to their lives, to attain better work from them, and in turn to provide more meaningful, continuous feedback. This also allows for continuous improvement on our end; the crux of assessment. 

Feelings and games

I'm currently enrolled in a MOOC (I am unsure how I currently feel about MOOCs overall in theory, but am liking this one so far). It's the Game Elements for Learning (#GE4L) MOOC offered through Canvas.

One of the readings we have been directed to is Karl Kapp's post, The emotional toll of instructional games.

(Kapp also wrote: The gamification of learning and instruction: Game-based methods and strategies for training and education. 2012).

I am finding this very intriguing because I have been having reservations about incorporating some game elements into the badge pilot I have designed for our 1 credit course. I wanted to have a leaderboard so students could feel good about doing extra work, but I did not want people to "shut down," as Kapp says, for not making it onto the leaderboard. The intent is to motivate, but in reality, it could have an opposite, unintended effect.

Kapp says, "If you decide to add game-elements (gamification) or if you decide to create a learning game with winners and losers, you need to find a way to deal with those who do not win. You need to help them avoid some of the negative feelings. You may even decide that a cooperative game is better than placing someone in a losing situation."

In the class, we are not offering enough rewards where certain people would feel like everyone but them is winning, and I am trying to mix it up so that a variety of people are included. I am also incorporating easter egg "mini-badges" for exceptional work, cooperative skills, or etc. be varied so the same people are not winning for excelling at particular skills while ignoring other skills. Students can unlock them without knowing. Still feeling it out as I go, and since this is the pilot it will be great to see what worked and what did not. I'm honestly a little worried that there is the reverse effect, that students might not care at all about the leaderboard. I'm still getting great work from them, but I'm not clear on if it is factoring into their submissions or not. They are very likely more motivated by the grade. And since this is a summer class, people are traveling and might have their thoughts elsewhere. Looking forward to survey data at the end of summer. I think I'm going to need to revisit the game + instructional design to make it more well-defined, include more motivational game elements, and find a way to make it not so heavily grade-focused (if possible since it is a credit course).


Kapp lists 12 ways to "mitigate losing in an instructional game or gamification situation," and I thought it might be useful to comment on each:

1) Forewarn the learners that they might become upset or frustrated if they find themselves losing and that is part of the learning process.
I did not do this, but I have at least tried to address affective learning outcomes and how research can cause frustration; that it's not easy or linear and you have to practice. I hope these skills can be applied to the game mechanics and course content as well, but perhaps it's better to come right out and say it.

2) Inform learners that they might lose the game and that is OK, learning will still occur.
I have done this in a way to allow students multiple attempts at a badge. This really can tie in to taking risks and having a decreased fear of failure. Since assessment should be to improve learning, not just judge work, I try to make the feedback really count.

3) Carefully brief all the learners on the instructional objectives of the game and de-emphasize winning.
Since the badging structure is tied directly to the course, the course objectives are the main focus. In this case, badges are more of a visual way to track progress.

4) Acknowledge the frustration or anger at losing.
Similar to my response for #1.

5) Ask learners to find lessons and reason within the lose. Have them dissect why they lost. Ask “can those insights lead to learning?”
Perhaps I should have students do more cognitive work here, but when they get it wrong when submitting work for a badge, I reiterate and clarify what exactly I'm looking for so they have a better opportunity to "win" the badge when they try again.

6) Don’t spend a great deal of time extolling the winners. Acknowledge winning and move right to the instructional lesson.
Exactly. I post a leaderboard in the news section and just leave it at that.

7) Provide a list of strategies that will help the learner win next time. (After the game.)
I try to do this in the assignment description so students know exactly what I'm looking for. It might even be better to post the actual rubric next semester.

8) Within the curriculum, follow the game activity with an activity where everyone can feel positive.
I'm not sure how this would play out in my scenario, but I think the participation points for discussion might work in this way. Rather than being graded on what is said, general points are given simply for being involved.

9) If in a classroom, allow people who did not win a chance to discuss why they didn’t win. Online, provide chat opportunities.
More chances for reflection would be very beneficial. The badges offered are so incremental, however, larger reflection might not be a good fit.

10) Consider if creating “winning” or “losing” is really what you want in the learning experience. Sometimes it is appropriate. Often it is appropriate but be prepared for unintended consequences and negative feedback if you don’t handle the situation properly.
Yes, definitely reconsidering even having a leaderboard. Maybe these easter egg "mini-badges" could instead be private between instructor and student.

11) Create different levels of winning, can a learner win a round, or one task, can small victories occur throughout the game. This is helpful because if a learner falls behind early, they may mentally drop out early in the learning process. Find ways to keep them engaged.
Love this.

12) Finally, you may want to consider building a cooperative rather than a competitive game. Working together is far more inclusive than competition.
Love this too. Just have to find a way to track each student's own work to tie to badge earning.