Competency-Based Learning & Creating Meaningful Experiences: Mutually Exclusive?

Lately, I have grown to be more skeptical of competency-based learning as used in the contexts it has been generally implemented, despite the fact that I am working to integrate library information literacy badges into our university-wide general education program (see my recent presentation with Andrew Battista about this topic for the 2015 CUNY Games Festival). So I was a little unsure what to expect in the Educause webinar I attended yesterday, Participatory Learning and Assessment in Competency-Based Contexts (ignore the unfortunate abbreviation of assessment in that URL...).

But I was pleasantly surprised with the webinarand also glad to see it was Dan Hickey from Indiana University doing the presentation. I took a BOOC on assessment practices with him a year or two ago and the way that course was developed has influenced my online course design.

I just wanted to reflect on what he talked about during the webinar because I think it's important for info lit instructional design, student engagement in general, and also as a way to think about standards vs the framework as we continue to have ongoing conversations about the ACRL revisions.

So first, if you're not familiar with competency-based learning (CBL), you can get some background here. Granted, that background info might be a bit biased since the Dept of Ed is in favor of implementing CBL. It's essentially the idea of replacing Carnegie seat hours with focus on passing assessments instead. So, if you prove you already have the skills or knowledge, you don't have the spend the time (re)learning the material, or if you learn content more quickly than others, you can spend less time on a unit.. On one hand, there are some great things that could come out of that, especially when we think about making information literacy instruction more appealing for both faculty and students. But there is also the *other* hand, where both Audrey Watters and Tressie McMillan Cottom have discussed the false meritocracy this reinforces, creating more barriers and difficulty for lower-income students in particular. Likewise, when you can just buy your skills through "cheaper" online assessments that have been corporatized, where does that leave social learning and any magic that could happen in the classroom? And how much weight does that really carry for finding a job (particularly for marginalized groups)?

Dan Hickey's presentation seemed to be about bringing the benefits of CBL into the classroom, while avoiding the not-so-great parts. He did mention that CBL is really like an assembly line, and that it's hard to use competencies in this way because teaching is so contextual. We don't want to make competencies a "statement of declarative knowledge." It's impossible to have students all learn the same things in the same way. Different students will have experiences that make them find more importance in one thing over another, and different groups of students will create knowledge that differs based on varying points of view.

Hickey discussed 5 Participatory Learning and Assessment Design Principles in order to make this point and demonstrate how to better incorporate CBL to make it contextual, examples follow:

1. Use public contexts to give meaning to knowledge tools: it's necessary to help students unpack between course concepts and their own context. This is personalized learning, not individualized learning.
2. Reward productive disciplinary engagement: disciplinary engagement involves both declarative knowledge and cultural practices. Be open with comments and engagement, stay away from grades. Let students interact and explore.
3. Grade artifacts through local reflections: save time for interaction, not on nitpicking via grading. Grade reflections instead of posts and comments (and stay away from using discussion boards).
4. Let individuals assess their understanding privately: use re-engagement instead of remediation, and offer open-ended and optional opportunities.
5. Measure achievement discreetly: there is too much teaching to the test, focus on bigger ideas. Withhold item-level feedback for test security and don't let students obsess over item-level answer memorization.

Moving away from teaching to the research paper

Might be sacrilege, but I find I have more engaging instruction sessions when students don't have a research paper attached as an assignment
— Nicole Pagowsky (@pumpedlibrarian) October 10, 2014

As I've been teaching a lot more classes lately that have a big research paper or capstone assignment attached in my new role as a subject liaison, I'm comparing it to my other work focusing on FYE-type instruction and student retention, thinking about engagement. This topic also came up in the Instructional Design Essentials ecourse I'm co-teaching with Erica DeFrain for ALA. Many participants in our course are starting to see the big red flags popping up with demo-based one-shots and student motivation as they have been working through designing their instruction or learning objects. As info lit instruction practice is moving more toward programmatic instruction and ensuring that an assignment is present so that there is more student buy-in and opportunities for assessment, I'm starting to question the assignment and (formal) assessment parts of library instruction... or, at least the research-paper-as-assignment.

The problem with one shots of course is that there is often an expectation to cram a ton of information into a 50-75min session that students will need to just remember for the rest of the semester and be able to complete their research papers "well." Not to mention library instruction becomes an isolated integration into the curriculum, particularly so when this type of instruction is in the form of skill-and-drill. There are many discussions going on--that have been going on for awhile now--pointing out that just teaching students how to use a database via a demo is not effective, and is boring for everyone (agreed!). Once students get to a point where they are writing a huge research paper, I almost feel like we've missed them, that they should have had more incremental, activity-based instruction, because this juncture in their instruction-need winds up being focused more on use of databases and just finding peer-reviewed articles to get the paper "done." I was teaching some undergrad students more context about what a literature review is for their required big paper, talking about their role as creators of knowledge, thinking of research as a conversation and where their research fits in, and crafting a narrative. I also did need to weave in database demos because the students had a certain requirement to fulfill. At the end of the session, I talked to the instructor to see if the session was what he was hoping for, especially since he had another section of the course coming in a few weeks later. He told me I really didn't need to talk about all that other stuff, all I really needed to do was point them to the databases because that's what they need for their paper. Students become so focused on the need to gather x amount of articles that other discussions become irrelevant and inefficient.

This is the issue with huge summative assessments, particularly the research paper. Barbara Fister has written about this problem at length, where she talks about Why the "Research Paper" isn't Working. I don't believe we have problems with student engagement when research papers are not attached to library instruction because our (potential) content isn't interesting, I think it's because traditionally (not everyone and not always, but typically in the past) a library instruction session divorced from an assignment *still* focused on a database demo. A database demo with no purpose, of course, is going to be agony for students (and the librarian). There are so many other things we could be doing, that some of us are doing, that serve as better options.

My perspective is that by the time students are writing huge research papers, they should have already had enough library instruction to where they could benefit from just a review of what they know. We should be scaffolding from the first year up instead of dumping all the boring mechanics of searching on students, with little other context, all at once. Now of course, much of this is out of our control, we get asked to do a one-shot where an assignment is already established, or even with efforts for collaboration, faculty might not want to work with us, or might just not feel they have the time. But when we can have a larger role in collaboration, especially for programmatic instruction, I try to suggest more scaffolding and lower-risk library instruction activities to enable greater discovery and discussion. Some of the best instruction sessions I've had have been with student success courses that don't have a big research paper, working with athletes, and working with a class examining social media that needed less help with "finding" and more so with creating a bigger discussion about information and communication. Unfortunately, I think this problem goes back to faculty not really knowing what we do and assuming we're just there to help students find things, as well as perceptions of librarians tying us to a "helper" role, so I think it just depends on the faculty we are working with and what our collaborative relationships are like. But I do think trying to move away from teaching to the research paper is one step in the right direction.

The Twitter convo continues from above...

More on threshold concepts and #ACRLILRevisions

The three threshold concepts in the new ACRL draft Framework for Information Literacy (Higher Ed) are noted as:

• Scholarship as a conversation
• Research as inquiry
• Format as process

From conversations on Twitter, Andy Burkhardt made a great post about how he has implemented "Research as inquiry" in his instruction. These practical examples are so helpful in understanding such a theoretical framework. Since I have been pushing research as conversation, or "Scholarship as conversation," in my own teaching, I thought I would share what I have done as well (for reference, I wrote about my initial thoughts on the new draft framework in a previous post).

Credit course and scaffolding

We have since paused our for-credit courses at the library, but in the last two sessions, I scaffolded research as conversation throughout the semester. I started off with introducing the concept, then made greater analogies to other modules, and in the end, had students create a short, animated video or comic strip (or script if they were not feeling visuals) illustrating a facet of research as conversation. (And this course is where we initially started using digital badges, as a side-note).

Searching online communities

In the fall, I had two additional opportunities of note to use scholarship as conversation, but also the other two threshold concepts. In a course in the UA's new eSociety program, my colleague, Leslie Sult, and I collaborated with the instructor to develop an in class activity and assignment. Students were researching a current event in a variety of formats/online communities (social media, local news, national/international news, news blogs, etc.), and the instructor wanted her students to do some critical thinking in groups to evaluate information and think about bias and point-of-view. I came up with the worksheet below, and we wound up having some great conversations as each group presented on their resource (YouTube and Twitter were especially interesting):

And additional questions we asked to coincide with the worksheet, after a brief lecture on related issues was delivered, included:

• Open versus closed community: impact? E.g., Facebook closed vs Twitter open – algorithms on Fbook and Google search when signed in (stay in the echo chamber)
• Primary versus secondary sources: what is the difference and when might you use either?
• How are messages changed/altered when they are retweeted or shared? Is anything lost? (like playing telephone), how do you account for this in searching? How do you know what part of the message is accurate? Methods for this
• Search strategies and tools: hashtags, groups, slang, memes, etc.
• Trolling: how does it affect communities and how might it change your search strategy?
• How do you know if someone is trolling a group or a topic discussion? Does trolling have significance in your search? Should you seek it out or ignore it?
• Back to whose voices are heard? What might the effect of being in the “echo chamber” do to whose voices you hear personally? What search strategies could you use to get out of the echo chamber?

The learning outcomes based on the instructor's course learning outcomes in conjunction with Leslie's and my goals for library instruction were the following:

1. Engage in a focused process of inquiry within an assigned online community in order to articulate the ways in which online communities function across contexts in contemporary life
2. Strategically access and evaluate information via search in an assigned online community in order to recognize various perspectives including rhetorical, philosophical, historical, sociological, and psychological viewpoints
3. Develop insight into the ethical aspects of information creation, use, access, and durability in order to be conscious of many group-related issues and practices relative to the use of computing technologies to facilitate group collaboration

Student athletes and avoiding plagiarism

When working with student athletes later in the semester, I more literally included scholarship as a conversation into my instructional design for a session I collaborated on with the Director of the athletes' writing center and my colleague, Niamh Wallace. I started the session talking about the process of research to frame positive uses of citations (how they help a conversation) and the negative effects of plagiarism, accidental or not (how they harm a conversation). To illustrate the concept in their minds first, I read them Burke's Unending Conversation Metaphor that I slightly adapted to more modern language they could relate to. I asked them to close their eyes and....

Imagine that you enter a party. You come late. When you arrive, others have been there long before you, and are engaged in a heated discussion, a discussion too heated for them to pause and fill you in. In fact, the discussion had already begun long before any of them got there, so that no one present is qualified to retrace for you all the steps that had gone before. You listen for a while, until you decide that you get the gist of the argument and join in. Someone answers; you respond; another comes to your defense; another aligns against you, to either the embarrassment or gratification of your opponent, depending on the quality of your ally’s assistance. However, the discussion is endless. It’s getting late, so you have to take off. And you leave, with the discussion still vigorously in progress.

Marisa (Director of writing center) incorporated a short lecture on how to write well when using other people's ideas, and as a hands-on activity had students write about their thoughts based on what we had presented to them (having them cite us). Then, using game design for this session, the theme was the "Citation Olympics," and we had students compete in groups for prizes as they learned content. Our format was introduce concept > practice > compete in the Citation Olympics at the end. Each module was a "sport," essentially. Here is a copy of the PPT we used to guide the session for a better idea (though much detail still gets left out from not including lecture notes).

Workshop on avoiding plagiarism for student athletes from Nicole Pagowsky


Anyhow, thought it might be helpful to share, and I hope to see how others have been teaching these concepts to gain a better understanding of how the new framework can be put into practice.

Reflection on Feminist Pedagogy for Library Instruction (book)

image from powderroom.jezebel.com

I just finished reading Maria T. Accardi's Feminist Pedagogy for Library Instruction (Library Juice Press, edited by Emily Drabinski). Aside from it resonating with me because I do try to employ critical library instruction and feminist pedagogy when I can, a lot of what Accardi discusses in the book also relates to what I'm doing with digital badges and also student retention.

First, for some background, Accardi explains that feminist pedagogy resides within critical pedagogy. Feminist pedagogy might carry the misconception of being instruction about women and feminism. Although it can often be related to that and employed in women's studies courses, it can be integrated in any form of curriculum. It typically exposes students to issues hidden in society, particularly injustices based on race, class, ability, sexual orientation, etc., and of course gender. Accardi quotes bell hooks (1994) for a concise description: "Feminist teaching techniques are anti-hierarchical, student-centered, promote community and collaboration, validate experiential knowledge, discourage passivity, and emphasize well being and self-actualization" (hooks in Accardi, p.31). To explain this further, it's to help students develop a critical consciousness and be able to take action on their learning.

So I wanted to look at some of the work I'm doing through this lens after this book made me think more clearly about what I am trying to accomplish.

Digital Badges: one of the issues I'm really struggling with for our badges are in scalability. There is a conflict between reaching many with limited FTE (meaning having automatic assessments that don't require intervention) versus reaching fewer, but retaining the ability to provide meaningful feedback and interact with students. One thing about badges is that typically they are awarded for rigid criteria. In a sense they need to be because a badge means something specific and ascribes value to a particular skill. So, if you have no concrete way of measuring this skill to determine if a badge was "rightfully earned" or not, what does it even mean if anyone or no one can actually obtain it? On the other hand, I believe students need to create their own learning and be proactive (feminist pedagogy), and I don't believe there should necessarily be an authority figure telling them what is right or wrong in absolute terms. Obviously, I know more about information literacy than they do, so I would need to develop content, etc., but as Accardi explains, feminist pedagogy is about being a guide and a facilitator rather than an all-knowing "sage-on-the-stage." A lot of the badges I have created focus on affective outcomes, students developing their own meaning of content, and opportunities for reflection and relating material to students' own lived experience. It's difficult enough to measure this as it is, let alone within the more rigid confines of a badge rubric. Not all badges need to be this way, but when attempting to design a suite of badges for campus, making as many automatic as possible without intervention on a 40k campus with 10 FTE instruction librarians tends to be more desirable. Using an automatic multiple choice quiz to determine skill acquisition is an easy, yet banking-model-esque method to award badges at scale. So something here I am trying to figure out is how to use feminist pedagogy but be simultaneously efficient? I'm working on some ideas for this, but it's certainly a point for discussion. How do you reconcile this in your teaching, particularly when instruction is for high numbers of students?

Student Retention: another area that I focus on. How conflicting that student retention is measured in rigid, big data and explained ROI, but it turns out some of the most effective methods to retain students include providing opportunities for personalization, social involvement, and affective learning outcomes. A lot of the instruction I do, and particularly for student success courses and "at-risk" groups includes promoting greater awareness and comfort in the library, rather than an explicit focus on content. I think student retention work would benefit greatly from feminist pedagogy, as would library instruction in general based on the high anxiety many students feel when using the library (and as Accardi does touch on).

This is my brief rundown of my most current thoughts from reading this book. I thought it was a great introduction to understanding feminist pedagogy and how it can be applied to library instruction. Accardi talked about her experience with the ACRL Immersion Program and also talked about issues with ACRL Standards, which I'd like to address in another post.

Grading and assessment, water and oil?

As I've been getting ready for ACRL Immersion 2013 Program Track (I leave in 1 week!), I've been finishing up a lot of readings on assessment. I've actually been really glad to read these articles, because as I've better solidified my notion of assessment through applying it to the instruction I am doing, I am finding my ideas are aligning with what I am reading.

The pattern I am finding in these readings is that assessment needs to be more holistic; assessment should be a method for students to learn rather than a focus on evaluation; and assessment should provide ongoing, meaningful feedback for students to practice instead of being judged.

We are wrapping up the summer semester with the badges pilot, and Purdue Passport incorporates assessment within earning a badge. Typically, a badge is given after a skill has been achieved, where assessment is more evaluative and judgmental rather than to provide feedback for improvement. This clashes with how I would prefer to teach and use the badges, so I've been using the feedback/assessment mechanism in Passport differently than it might be intended.

This is good because I think students are getting more out of the class, but also poses some conflicts:

1. If badges being awarded are not based on more rigid judgement of skill acquisition, how valuable are they?
2. On this note, how interoperable are they? Can their qualities be translated or compared to other institutions or libraries offering similar badges if desired evidence isn't as clearly enforced?

Because this is a credit class, grades need to be tied to student work. For this, the badges are essentially pass/fail. You either earn the badge or you don't. If a student is late in finishing badge work an exception is made to give them half off, but this is the only partial credit awarded. There are pros and cons to this as well:

Pros: Students can take risks in their responses and have less fear of failure (this positive aspect is rooted in game mechanics); I can focus more on the quality of my feedback rather than what level of good or bad the student's work falls into

Cons: How is good student work differentiated from bad work? Particularly if bad work is due to sloppiness or disinterest. Shouldn't a student who submitted excellent work (or evidence) for a badge be awarded the badge, where less stellar work would not be awarded the badge? Isn't the purpose of awarding badges to demonstrate that a skill was successfully acquired?

I have such mixed feelings on this. But one feature of Passport is to allow students a re-do. I use this often for sloppy work. I will leave feedback explaining exactly what I'm looking for and give the student a second chance (next semester I will be sharing specific rubrics for each badge with students so they have an even better concept of what level of understanding is desired). 

I am not a stickler on lower-level concepts like formatting a citation perfectly or memorizing exact steps on how to find an article when you only have a citation (these are specific assessments in the class to address more basic skills within learning outcomes). If a student has most of a citation right but forgets to italicize the journal title for MLA style, it's really just busy work for them to make them re-do it or for me to take points off. I leave feedback letting them know they mostly got it and to remember to double check these things for formal papers; and then I give them all the points. I love Barbara Fister's 2013 LOEX Keynote (in fact, my team read it as part of our strategic planning for the new fiscal year). I agree so strongly with her whole presentation, and using a specific example here, "very rarely outside of school are citations needed." I care way more about if students are able to understand what the purpose of a citation is and to incorporate this into their new understanding of "research as conversation" than about styles and how to format.

One assigned article that has been part of this class for a long time is a CQ Researcher article on cheating: why students cheat and how they cheat. It's interesting to see what students agree with in their reflections and a number do say that when a student doesn't feel course material has real application in their lives (or when an instructor provides little to no meaningful feedback), a student has no motivation or investment to put in quality work, and so cheating is easy. Focusing less on grades and more on understanding and a conversation between the students and us as instructors creates a richer experience for all. Their reflections resonate well with what we're doing in the course to make it apply to their lives, to attain better work from them, and in turn to provide more meaningful, continuous feedback. This also allows for continuous improvement on our end; the crux of assessment. 

Feelings and games

I'm currently enrolled in a MOOC (I am unsure how I currently feel about MOOCs overall in theory, but am liking this one so far). It's the Game Elements for Learning (#GE4L) MOOC offered through Canvas.

One of the readings we have been directed to is Karl Kapp's post, The emotional toll of instructional games.

(Kapp also wrote: The gamification of learning and instruction: Game-based methods and strategies for training and education. 2012).

I am finding this very intriguing because I have been having reservations about incorporating some game elements into the badge pilot I have designed for our 1 credit course. I wanted to have a leaderboard so students could feel good about doing extra work, but I did not want people to "shut down," as Kapp says, for not making it onto the leaderboard. The intent is to motivate, but in reality, it could have an opposite, unintended effect.

Kapp says, "If you decide to add game-elements (gamification) or if you decide to create a learning game with winners and losers, you need to find a way to deal with those who do not win. You need to help them avoid some of the negative feelings. You may even decide that a cooperative game is better than placing someone in a losing situation."

In the class, we are not offering enough rewards where certain people would feel like everyone but them is winning, and I am trying to mix it up so that a variety of people are included. I am also incorporating easter egg "mini-badges" for exceptional work, cooperative skills, or etc. be varied so the same people are not winning for excelling at particular skills while ignoring other skills. Students can unlock them without knowing. Still feeling it out as I go, and since this is the pilot it will be great to see what worked and what did not. I'm honestly a little worried that there is the reverse effect, that students might not care at all about the leaderboard. I'm still getting great work from them, but I'm not clear on if it is factoring into their submissions or not. They are very likely more motivated by the grade. And since this is a summer class, people are traveling and might have their thoughts elsewhere. Looking forward to survey data at the end of summer. I think I'm going to need to revisit the game + instructional design to make it more well-defined, include more motivational game elements, and find a way to make it not so heavily grade-focused (if possible since it is a credit course).


Kapp lists 12 ways to "mitigate losing in an instructional game or gamification situation," and I thought it might be useful to comment on each:

1) Forewarn the learners that they might become upset or frustrated if they find themselves losing and that is part of the learning process.
I did not do this, but I have at least tried to address affective learning outcomes and how research can cause frustration; that it's not easy or linear and you have to practice. I hope these skills can be applied to the game mechanics and course content as well, but perhaps it's better to come right out and say it.

2) Inform learners that they might lose the game and that is OK, learning will still occur.
I have done this in a way to allow students multiple attempts at a badge. This really can tie in to taking risks and having a decreased fear of failure. Since assessment should be to improve learning, not just judge work, I try to make the feedback really count.

3) Carefully brief all the learners on the instructional objectives of the game and de-emphasize winning.
Since the badging structure is tied directly to the course, the course objectives are the main focus. In this case, badges are more of a visual way to track progress.

4) Acknowledge the frustration or anger at losing.
Similar to my response for #1.

5) Ask learners to find lessons and reason within the lose. Have them dissect why they lost. Ask “can those insights lead to learning?”
Perhaps I should have students do more cognitive work here, but when they get it wrong when submitting work for a badge, I reiterate and clarify what exactly I'm looking for so they have a better opportunity to "win" the badge when they try again.

6) Don’t spend a great deal of time extolling the winners. Acknowledge winning and move right to the instructional lesson.
Exactly. I post a leaderboard in the news section and just leave it at that.

7) Provide a list of strategies that will help the learner win next time. (After the game.)
I try to do this in the assignment description so students know exactly what I'm looking for. It might even be better to post the actual rubric next semester.

8) Within the curriculum, follow the game activity with an activity where everyone can feel positive.
I'm not sure how this would play out in my scenario, but I think the participation points for discussion might work in this way. Rather than being graded on what is said, general points are given simply for being involved.

9) If in a classroom, allow people who did not win a chance to discuss why they didn’t win. Online, provide chat opportunities.
More chances for reflection would be very beneficial. The badges offered are so incremental, however, larger reflection might not be a good fit.

10) Consider if creating “winning” or “losing” is really what you want in the learning experience. Sometimes it is appropriate. Often it is appropriate but be prepared for unintended consequences and negative feedback if you don’t handle the situation properly.
Yes, definitely reconsidering even having a leaderboard. Maybe these easter egg "mini-badges" could instead be private between instructor and student.

11) Create different levels of winning, can a learner win a round, or one task, can small victories occur throughout the game. This is helpful because if a learner falls behind early, they may mentally drop out early in the learning process. Find ways to keep them engaged.
Love this.

12) Finally, you may want to consider building a cooperative rather than a competitive game. Working together is far more inclusive than competition.
Love this too. Just have to find a way to track each student's own work to tie to badge earning.

Workshop for Faculty: Designing Effective Research Assignments

Designing Effective Research Assignments from Nicole Pagowsky

Today, myself and a colleague presented a workshop to faculty on designing effective research assignments for student success. Since we consult with faculty often and see good/bad assignment design in action through library instruction and feedback from the reference desk, we were proactive and offered this session as part of the Office of Instruction and Assessment teaching academy offered each semester. We thought this would be a great opportunity to work more closely with faculty who might not know about library services or best practices / pitfalls.

If you download the PPT instead of just viewing it on SlideShare, you can see our presenter notes, detailing what was covered in each slide. We started off talking about issues students have with research and research assignments through looking at the ERIAL Project, Project Information Literacy, and Kuhlthau's Information Search Process. We discussed how faculty and librarians overestimate students' skills in research (ERIAL), how students overestimate their own skills as well but are anxious about research (and even dread it, PIL), and then how to understand this affective learning and when/how to intervene (Kuhlthau). Applying this knowledge to ACRL Info Lit standards, we had faculty start to think about current assignments they are using, or assignments they would hope to use, in this new context.

Next, we covered specific design pitfalls and best practices, breaking best practices down into: scaffolding, transparency, context, critical thinking, process over perfection, and embedding academic integrity.

As part of hands-on activities, we, as I mentioned, had instructors use a worksheet to think about their own assignments and evaluating their effectiveness, then we also had them evaluate a sample assignment using criteria related to being specific, transparent, and encouraging critical thinking.

The session went very well, we even received applause at the end with many thank yous. There is really nothing I can think of to modify at this time, other than spending more time hearing about what kinds of assignments instructors are using. We will be getting formal assessment back soon from the Office of Instruction and Assessment, who hosted the workshop series. We will be offering this workshop again in a month or two and am looking forward to working with more faculty.