A short post on #critlib outcomes and assessment

As #critlib is wrapping up for this week, the topic of assessment being prohibitive came up in regards to libraries contributing to social justice initiatives in communities when tragedies like Ferguson happen. I mentioned in the chat that I needed to develop a rubric for a campus committee, where we are working on our equivalent of AAC&U's High Impact Practices. I was able to include critical pedagogy components, and even the new ACRL framework to design it, so I am sharing by request. This is certainly not finalized or widely distributed, so just sharing my work so far:

(^ Click to fully view)

Regarding the evil assessment talk, outcomes and assessment definitely can have #critlib components and work for "good" (vs "evil"). There are also affective learning outcomes (#feelings) that can tie in especially to feminist and critical pedagogy. Lisa Hinchliffe made some great points:

Although we do have institutional constraints in many cases and need to work with/around those, there are still a lot of opportunities to use assessment for more than just measuring required quantification. Perhaps this is a topic that could use more discussion in future #critlib chats!

More on threshold concepts and #ACRLILRevisions

The three threshold concepts in the new ACRL draft Framework for Information Literacy (Higher Ed) are noted as:

• Scholarship as a conversation
• Research as inquiry
• Format as process

From conversations on Twitter, Andy Burkhardt made a great post about how he has implemented "Research as inquiry" in his instruction. These practical examples are so helpful in understanding such a theoretical framework. Since I have been pushing research as conversation, or "Scholarship as conversation," in my own teaching, I thought I would share what I have done as well (for reference, I wrote about my initial thoughts on the new draft framework in a previous post).

Credit course and scaffolding

We have since paused our for-credit courses at the library, but in the last two sessions, I scaffolded research as conversation throughout the semester. I started off with introducing the concept, then made greater analogies to other modules, and in the end, had students create a short, animated video or comic strip (or script if they were not feeling visuals) illustrating a facet of research as conversation. (And this course is where we initially started using digital badges, as a side-note).

Searching online communities

In the fall, I had two additional opportunities of note to use scholarship as conversation, but also the other two threshold concepts. In a course in the UA's new eSociety program, my colleague, Leslie Sult, and I collaborated with the instructor to develop an in class activity and assignment. Students were researching a current event in a variety of formats/online communities (social media, local news, national/international news, news blogs, etc.), and the instructor wanted her students to do some critical thinking in groups to evaluate information and think about bias and point-of-view. I came up with the worksheet below, and we wound up having some great conversations as each group presented on their resource (YouTube and Twitter were especially interesting):

And additional questions we asked to coincide with the worksheet, after a brief lecture on related issues was delivered, included:

• Open versus closed community: impact? E.g., Facebook closed vs Twitter open – algorithms on Fbook and Google search when signed in (stay in the echo chamber)
• Primary versus secondary sources: what is the difference and when might you use either?
• How are messages changed/altered when they are retweeted or shared? Is anything lost? (like playing telephone), how do you account for this in searching? How do you know what part of the message is accurate? Methods for this
• Search strategies and tools: hashtags, groups, slang, memes, etc.
• Trolling: how does it affect communities and how might it change your search strategy?
• How do you know if someone is trolling a group or a topic discussion? Does trolling have significance in your search? Should you seek it out or ignore it?
• Back to whose voices are heard? What might the effect of being in the “echo chamber” do to whose voices you hear personally? What search strategies could you use to get out of the echo chamber?

The learning outcomes based on the instructor's course learning outcomes in conjunction with Leslie's and my goals for library instruction were the following:

1. Engage in a focused process of inquiry within an assigned online community in order to articulate the ways in which online communities function across contexts in contemporary life
2. Strategically access and evaluate information via search in an assigned online community in order to recognize various perspectives including rhetorical, philosophical, historical, sociological, and psychological viewpoints
3. Develop insight into the ethical aspects of information creation, use, access, and durability in order to be conscious of many group-related issues and practices relative to the use of computing technologies to facilitate group collaboration

Student athletes and avoiding plagiarism

When working with student athletes later in the semester, I more literally included scholarship as a conversation into my instructional design for a session I collaborated on with the Director of the athletes' writing center and my colleague, Niamh Wallace. I started the session talking about the process of research to frame positive uses of citations (how they help a conversation) and the negative effects of plagiarism, accidental or not (how they harm a conversation). To illustrate the concept in their minds first, I read them Burke's Unending Conversation Metaphor that I slightly adapted to more modern language they could relate to. I asked them to close their eyes and....

Imagine that you enter a party. You come late. When you arrive, others have been there long before you, and are engaged in a heated discussion, a discussion too heated for them to pause and fill you in. In fact, the discussion had already begun long before any of them got there, so that no one present is qualified to retrace for you all the steps that had gone before. You listen for a while, until you decide that you get the gist of the argument and join in. Someone answers; you respond; another comes to your defense; another aligns against you, to either the embarrassment or gratification of your opponent, depending on the quality of your ally’s assistance. However, the discussion is endless. It’s getting late, so you have to take off. And you leave, with the discussion still vigorously in progress.

Marisa (Director of writing center) incorporated a short lecture on how to write well when using other people's ideas, and as a hands-on activity had students write about their thoughts based on what we had presented to them (having them cite us). Then, using game design for this session, the theme was the "Citation Olympics," and we had students compete in groups for prizes as they learned content. Our format was introduce concept > practice > compete in the Citation Olympics at the end. Each module was a "sport," essentially. Here is a copy of the PPT we used to guide the session for a better idea (though much detail still gets left out from not including lecture notes).

Workshop on avoiding plagiarism for student athletes from Nicole Pagowsky


Anyhow, thought it might be helpful to share, and I hope to see how others have been teaching these concepts to gain a better understanding of how the new framework can be put into practice.

Reflection on Feminist Pedagogy for Library Instruction (book)

image from powderroom.jezebel.com

I just finished reading Maria T. Accardi's Feminist Pedagogy for Library Instruction (Library Juice Press, edited by Emily Drabinski). Aside from it resonating with me because I do try to employ critical library instruction and feminist pedagogy when I can, a lot of what Accardi discusses in the book also relates to what I'm doing with digital badges and also student retention.

First, for some background, Accardi explains that feminist pedagogy resides within critical pedagogy. Feminist pedagogy might carry the misconception of being instruction about women and feminism. Although it can often be related to that and employed in women's studies courses, it can be integrated in any form of curriculum. It typically exposes students to issues hidden in society, particularly injustices based on race, class, ability, sexual orientation, etc., and of course gender. Accardi quotes bell hooks (1994) for a concise description: "Feminist teaching techniques are anti-hierarchical, student-centered, promote community and collaboration, validate experiential knowledge, discourage passivity, and emphasize well being and self-actualization" (hooks in Accardi, p.31). To explain this further, it's to help students develop a critical consciousness and be able to take action on their learning.

So I wanted to look at some of the work I'm doing through this lens after this book made me think more clearly about what I am trying to accomplish.

Digital Badges: one of the issues I'm really struggling with for our badges are in scalability. There is a conflict between reaching many with limited FTE (meaning having automatic assessments that don't require intervention) versus reaching fewer, but retaining the ability to provide meaningful feedback and interact with students. One thing about badges is that typically they are awarded for rigid criteria. In a sense they need to be because a badge means something specific and ascribes value to a particular skill. So, if you have no concrete way of measuring this skill to determine if a badge was "rightfully earned" or not, what does it even mean if anyone or no one can actually obtain it? On the other hand, I believe students need to create their own learning and be proactive (feminist pedagogy), and I don't believe there should necessarily be an authority figure telling them what is right or wrong in absolute terms. Obviously, I know more about information literacy than they do, so I would need to develop content, etc., but as Accardi explains, feminist pedagogy is about being a guide and a facilitator rather than an all-knowing "sage-on-the-stage." A lot of the badges I have created focus on affective outcomes, students developing their own meaning of content, and opportunities for reflection and relating material to students' own lived experience. It's difficult enough to measure this as it is, let alone within the more rigid confines of a badge rubric. Not all badges need to be this way, but when attempting to design a suite of badges for campus, making as many automatic as possible without intervention on a 40k campus with 10 FTE instruction librarians tends to be more desirable. Using an automatic multiple choice quiz to determine skill acquisition is an easy, yet banking-model-esque method to award badges at scale. So something here I am trying to figure out is how to use feminist pedagogy but be simultaneously efficient? I'm working on some ideas for this, but it's certainly a point for discussion. How do you reconcile this in your teaching, particularly when instruction is for high numbers of students?

Student Retention: another area that I focus on. How conflicting that student retention is measured in rigid, big data and explained ROI, but it turns out some of the most effective methods to retain students include providing opportunities for personalization, social involvement, and affective learning outcomes. A lot of the instruction I do, and particularly for student success courses and "at-risk" groups includes promoting greater awareness and comfort in the library, rather than an explicit focus on content. I think student retention work would benefit greatly from feminist pedagogy, as would library instruction in general based on the high anxiety many students feel when using the library (and as Accardi does touch on).

This is my brief rundown of my most current thoughts from reading this book. I thought it was a great introduction to understanding feminist pedagogy and how it can be applied to library instruction. Accardi talked about her experience with the ACRL Immersion Program and also talked about issues with ACRL Standards, which I'd like to address in another post.

Grading and assessment, water and oil?

As I've been getting ready for ACRL Immersion 2013 Program Track (I leave in 1 week!), I've been finishing up a lot of readings on assessment. I've actually been really glad to read these articles, because as I've better solidified my notion of assessment through applying it to the instruction I am doing, I am finding my ideas are aligning with what I am reading.

The pattern I am finding in these readings is that assessment needs to be more holistic; assessment should be a method for students to learn rather than a focus on evaluation; and assessment should provide ongoing, meaningful feedback for students to practice instead of being judged.

We are wrapping up the summer semester with the badges pilot, and Purdue Passport incorporates assessment within earning a badge. Typically, a badge is given after a skill has been achieved, where assessment is more evaluative and judgmental rather than to provide feedback for improvement. This clashes with how I would prefer to teach and use the badges, so I've been using the feedback/assessment mechanism in Passport differently than it might be intended.

This is good because I think students are getting more out of the class, but also poses some conflicts:

1. If badges being awarded are not based on more rigid judgement of skill acquisition, how valuable are they?
2. On this note, how interoperable are they? Can their qualities be translated or compared to other institutions or libraries offering similar badges if desired evidence isn't as clearly enforced?

Because this is a credit class, grades need to be tied to student work. For this, the badges are essentially pass/fail. You either earn the badge or you don't. If a student is late in finishing badge work an exception is made to give them half off, but this is the only partial credit awarded. There are pros and cons to this as well:

Pros: Students can take risks in their responses and have less fear of failure (this positive aspect is rooted in game mechanics); I can focus more on the quality of my feedback rather than what level of good or bad the student's work falls into

Cons: How is good student work differentiated from bad work? Particularly if bad work is due to sloppiness or disinterest. Shouldn't a student who submitted excellent work (or evidence) for a badge be awarded the badge, where less stellar work would not be awarded the badge? Isn't the purpose of awarding badges to demonstrate that a skill was successfully acquired?

I have such mixed feelings on this. But one feature of Passport is to allow students a re-do. I use this often for sloppy work. I will leave feedback explaining exactly what I'm looking for and give the student a second chance (next semester I will be sharing specific rubrics for each badge with students so they have an even better concept of what level of understanding is desired). 

I am not a stickler on lower-level concepts like formatting a citation perfectly or memorizing exact steps on how to find an article when you only have a citation (these are specific assessments in the class to address more basic skills within learning outcomes). If a student has most of a citation right but forgets to italicize the journal title for MLA style, it's really just busy work for them to make them re-do it or for me to take points off. I leave feedback letting them know they mostly got it and to remember to double check these things for formal papers; and then I give them all the points. I love Barbara Fister's 2013 LOEX Keynote (in fact, my team read it as part of our strategic planning for the new fiscal year). I agree so strongly with her whole presentation, and using a specific example here, "very rarely outside of school are citations needed." I care way more about if students are able to understand what the purpose of a citation is and to incorporate this into their new understanding of "research as conversation" than about styles and how to format.

One assigned article that has been part of this class for a long time is a CQ Researcher article on cheating: why students cheat and how they cheat. It's interesting to see what students agree with in their reflections and a number do say that when a student doesn't feel course material has real application in their lives (or when an instructor provides little to no meaningful feedback), a student has no motivation or investment to put in quality work, and so cheating is easy. Focusing less on grades and more on understanding and a conversation between the students and us as instructors creates a richer experience for all. Their reflections resonate well with what we're doing in the course to make it apply to their lives, to attain better work from them, and in turn to provide more meaningful, continuous feedback. This also allows for continuous improvement on our end; the crux of assessment. 

Meme-themed CRAAP Test

As part of my work in identifying and supporting student retention/success efforts on campus, I have been providing workshops to athletes as part of their mandatory study table hours. This semester, I am working with the football players and just finished up my second visit with them. We are really only given a half hour since they are busy getting homework done, and oftentimes, attention spans are short. This means I have to plan out lessons that are quick and to the point, and preferably engaging and hands-on as well.

I plotted out the three sessions to cover topics at point-of-need in the semester. First, a colleague and I went over library resources available to them and how to get started with research (catalog and databases). This session was directly tied to assignments in classes the majority of the athletes take in their first semester. The session I covered this week was evaluating sources, and the final session next month will be on citations and avoiding plagiarism. The latter two sessions are not directly tied to a particular assignment, so the challenge is to get their attention and make the workshops relevant.

Since over the summer, the competition was fierce when we used a BINGO-like game to teach the athletes about library services available to them, I figured using a game again would appeal to the football players. The plan was to review search tips from the first session briefly, cover the importance of evaluating sources, show the students the CRAAP Test from CSU Chico, and then with these skills, let them loose on the game.

Students were broken up into groups of 3 (out of a 24-person class), and each team was given a different meme. The athletes would need to use their newly-acquired searching skills to find a source that explains what their meme means, and then use the CRAAP Test to determine if the source is credible or not. The first team(s) to finish, and then accurately explain their meme and credibility of their source would win candy (bribes do help).

From http://knowyourmeme.com/photos/212976

They seemed to perk up a bit when I mentioned memes. Some of them weren't sure what they were at first, but once they saw an example, they knew. Because it was such a short session, it was acceptable for them to use web sources. This made it easier for them to explain the meme but seemed more difficult for them to assess credibility. Also, in assigning different memes to each team, some of the ones chosen were more difficult to explain than others, so this gave an unfair advantage to certain groups. I tried to choose ones with cultural or historical significance that would have more content available when researching, but this wasn't always successful. Another issue was students simply locating pieces of information on the site and then regurgitating this on their worksheet, rather than using critical thinking skills to examine credibility. After noticing this, I made an announcement to think critically about what they're finding: instead of just writing down the year of publication or author name, think about if having more recent information on the topic is essential, or what affiliations and expertise the author has.

Using this lesson a second time with pre-business freshmen in an MIS class, the colleague I delivered the instruction with to the athletes and I instead required using the CRAAP Test on library resources only, requiring the students to search the databases. A problem with memes and researching pop culture of course is that it's brand new, so it's not likely there are scholarly sources. We permitted use of news articles instead because of this. Because of the class and revision of the lesson plan, it seemed to be more successful the second time around.

And today was the third trial with this lesson, on the second section of the MIS class. Instead of assigning a different meme to each group, I chose the two memes most conducive to research, divided the class in half, and then broke each half into teams of three. The first team in group 1 to explain the meme and source credibility would win, as would the first team in group 2 to do so. This went much more smoothly and it generated more possibility for discussion when groups explained their results.

I have to say, after working with the athletes especially, my classroom management skills are really taking off.... dealing with students not paying attention, talking, doing other homework.... I'm finding a silent pause with direct eye contact, asking discussion questions so students have to pay attention to each other if not me, and saying stuff like "come on guys, really? this is to help you finish your homework faster" seems to help a little. I think the library instruction sessions will always be perceived as boring and irrelevant by a number of students, unfortunately (I certainly thought so for a good portion of my undergrad years), but if we can mix things up with games and interactivity, it definitely can start to change those perceptions.

Lesson plan - Original - added in more discussion of aspects of CRAAP Test: students seemed unsure about certain terminology, like currency, relevance, and authority

Individual handouts

Group activity - memes - Original before revising to only include library resources and handing out only the memes: casually pepper spray everything cop and Neil deGrasse Tyson reaction

Library mystery as outreach and instruction

We do outreach to the community, particularly over the summer, and tomorrow we will have high school students visiting the library for a summer workshop on research skills. Since it's more of a summer camp and these are younger students, we wanted to make sure they would have some fun and be engaged... so we are using murder mysteries as our hands-on activity after a short instruction session to prepare the students for detective work.

I think the mystery I created is fun and it works; I'm sure it would be much better if I had more background in game design (working on that), but this at least will hit all the learning outcomes in a cohesive way:

1. Students will understand how databases work, and what the difference is between library databases and Google.
2. Students will be able to construct a basic search using synonyms for a broader search strategy.
3. Students will be able to locate a book using the library catalog.
4. Students will be able to evaluate websites using the CRAAP test.
5. Students will be able to use information appropriately by citing sources in APA style.

Assessment will be done by seeing if they solve the mystery, and since they have to write down answers along the way, we can see some of their search process to get a sense of how much they learned during the instruction portion of the session.

The mystery takes them through using different types of resources in the library, including (hopefully) getting value and comfort in asking a librarian for help. In the end, they wind up in Special Collections where they are spending the afternoon, and will solve the case at the end of the day. We decided to tie our instruction to Special Collections so the students get a more holistic picture of the research process.

I am sure I will notice some snags along the way as this is the first time we are doing this, so I hope to do a follow up post about what went wrong and what could be improved. This would be a great way to gamify orientations to the library for UA freshmen, especially for the smaller student success courses, and could then be tied to retention efforts.

See the mystery with answer key here.
(The narrative makes more sense and is more engaging if you read the full mystery here, below is a synopsis.)

The students start off with information that Wilbur Wildcat (the UA mascot) has been found in the library by one of the exhibits. They need to use the library website to figure out which one and where; they are given a clue that the exhibit features two types of music that were influential in Tucson's culture.

Here they get information the police have collected as well as stats from the medical examiner. They find out Wilbur died from exsiccion, which when they are prompted to look up in Stedman's Medical Dictionary from our health subject guide, they realize that this is actually a synonym for dehydration. From that, they are given a riddle to figure out that a five-letter word for a liquid that can cure dehydration is water. They then need to search the library catalog for a book about water and border issues. Once they find a particular book, they need to go to the stacks to get their next clue.

In this next clue, the students realize an important fact was left off the police report: the suspect left a copy of the Oxford English Dictionary at the scene of the crime, open to the page on aliens. Since the physical copy is locked up at police headquarters, they can luckily search the OED online through the library. They must write down the first use of the term alien in science fiction to realize that the suspect is extra-terrestrial. With this info, they then go to the reference desk and are required to ask a librarian for help in locating an article on UFO sightings in Arizona in the last 50 years. Once they find an article, they must write down the citation in APA style; if the librarian approves that the citation is correct, s/he will hand the team their next clue.

Going to the police with the hypothesis that the killer is an alien would probably get the detectives laughed at, so it is suggested in the next clue to get background information first. A great place to start for background info is CQ Researcher. They must look up UFOs in this database and click on the most recent entry (which, unfortunately, is 1996). They are prompted to read about the University of Arizona professor, James E McDonald, who was a pro-UFO meteorologist. He happened to collect dirt samples from UFO sightings, which are housed in Special Collections (I think this is awesome). They locate his name in CQ Researcher, then must search the catalog to find any works by him as an author in the stacks. They will find the McDonald papers which are housed in Special Collections, along with the dirt, and it is there they will apprehend the killer... who in fact isn't really a killer since the medical examiner made a small mistake in pronouncing Wilbur dead: he was simply in a coma from dehydration and just needs to drink some water (keeping it PG).

I'm excited to see how this goes, and how well my portion of the mystery ties into what Special Collections will be covering. More next time...

SOPA on a ropa

I initially wrote this blog post on 1/21/12 during #alamw12 but didn't get a chance to post it. With the boycotting of Elsevier, I thought it would be a good time to pull this back out:

There has been a lot of talk about SOPA and PIPA leading up to the conference, and now during. One of my conference roommates, Lauren Bradley, pointed out this Tweet that is pretty hilarious:

http://twitter.com/#!/danwho/status/160800863298916353

Clearly, there is some inner turmoil in dealing with these vendors professionally, and having good relationships with them for our libraries and in general, yet if they are supporting something you (or I, I do) oppose such as SOPA, then what is our professional obligation versus personal ethics? This last Wednesday was a blackout day in solidarity of protest for SOPA. ALA made commentary via the website, and librarian projects such as Radical Reference and In the Library with the Lead Pipe went black for the day. I even blacked out Librarian Wardrobe.

Now that we’re physically at the conference, though (or, those of us who are here), what can we do to not have the cognitive dissonance of being so vocal on the internet battleground, but feeling politely silent at the conference? For starters, Andy Woodworth at Agnostic Maybe made an *amazing* color-coded guide to the exhibit hall. Amazing, really! So you know which vendors to complain to and/or avoid. I’d say this extends to the parties, too. I had RSVP’d to the Elsevier Dessert Reception but now decided I’m not going to go. Some of the ALA Think Tankers are going to go and protest while there. I guess it goes either way it’s kind of like if you don’t go and they see there are significantly fewer people there then maybe they’ll realize our collective voice is pretty strong. At the same time, if we don’t go to these things and actually verbalize our opposition, what will actually come of it? They could just think we aren’t showing up because of unrelated reasons. 

So what vendors have you spoken to, who support SOPA? What are you doing to fight against this crap? Do you think going to the party and protesting or not going by silent protest is more effective?

Fanzines book has no fans

I'm a little slow in catching up to post this, but there are some interesting discussions surrounding Fanzines, a new title by academic professor, Teal Triggs. The main discussion I'd like to direct you to is a blog post by Jerianne of Zine World, where she explains the problem with this new text and its implications.

This new book documents zines, with visuals and historical information. Although it is true an academic book about zines and zine history could stand to further legitimize the medium in research and academia, there are some problems here, which I learned from Jerianne's post. These problems center on copyright and accurate factual information. I won't repeat everything Jerianne discusses in her blog post, but do want to point out some key information:

1. Zine images were used without notifying zinesters prior to publication (in most cases, if a zine creator even was contacted, it was an afterthought).
2. Perhaps related to this snafu (lightly-stated), there is information about the zines and zinesters covered in the book that is incorrect.

This translates to: well, zinesters being pissed, and rightfully so due to someone profiting financially and profesionally off their DIY/not-doing-it-for-profit ethos; as well as now having a text covering a topic that is not highly documented -- that will be imprinting incorrect information into history (with not much else in formal print to contest it).

Jerianne quotes Tobi Vail:

As Tobi Vail pointed out in her blog entry about the book: “I think there should be a way to contest ‘false information’ in published works. Because once it’s in a book, it’s a ‘fact.’ People will use this book as a source for further writing on the subject matter. … Because once something is in print, it becomes an authority.”

This is true, and I see it often in student research. If information is printed, it will often be repeated without question. So, what good could come of this situation? It looks like there is no effort or interest on the author/editor or publisher's part to rectify the situation, so what to do? Using this as an example in teaching critical thinking skills in information literacy is a start; I'm not suggesting to purchase the book at all, but explaining the issue to students and showing them how misinformation can appear in authoritative-seeming sources, even by a credentialed professor, is possible.

This example could also be used to show that research is truly a discussion. Look at Wikipedia, with all the arguments that spawn out of incorrect information in entries, or even just differences of opinion. Of course, the book in question is not a matter of opinion, there is simply factual information that is incorrect, but it could be a good segue into also showing Wikipedia can be useful for the zeitgeist and basic information, but it's still necessary to double check, consider motivations, and question "authority." Wikipedia is not the only source that can have faulty information, but could serve as familiar grounds for comparison. And, hopefully, someone with more accurate information might consider publishing a book to rival Fanzines.