Rethinking Rubrics in Writing Assessment

[Maja Wilson]

When I started teaching writing, I expected (and was expected) to use rubrics. I had misgivings about them, but figured that the problem was me: I wasn't using the right rubric, or I was using it wrong, or I hadn't included all the right categories in my own. This book is my attempt to figure out the cause and implications of my misgivings, and an attempt to figure out how to assess writing in a different way, a way that honors what Chris Anson would call "our deepest convictions about the complexities of the writing process."

[Lisa Park]

ahahaha!!! What a great reply, Ken. Very clever! Just had to say. :) Thanks for the laugh.

[Daniel Sharkovitz]

Maja,

1. This is such an important question. Rubrics can be helpful and dangerous. Improperly used, they can give the impression that the proverbial emperor wears raiments of gold, when in fact he is actually naked. Properly used, they can be clarifying and inspiring.

2. The rubric concept landed in our school in earnest several years ago. What I have found is that, in general, if my students and I create rubrics on the fly and that we actually connect them to what we are trying to study and learn, that the efficacy of the rubrics improves. (Now don't laugh, but I'm finishing up the first draft of a hypertext poetry writing rubric that my students and I started writing in my Creative Writing class. I'll post it on my website next month. Maybe I'll post it here, too, for comment. I do have some other rubrics posted already that I've written with my students--though frankly I am not happy with any of them.)

Of course, does what I'm saying mean that rubrics need to be re-invented with each new group of students and for each new lesson? I don't know yet. But I think that part of learning how to write something means learning how to reflect upon the qualities it should have when finished not just when it is done, but along the way. Sometimes I think that an invisible rubric, that is to say, the understandings that emerge between writer and teacher during whatever sorts of conversations they have about a piece, can be more helpful to the writer because they are always and forever grafted onto the writing itself.

3. There is an interesting article, "A Mania for Rubrics," by Thomas Newkirk, at http://www.edweek.org/ew/articles/2000/09/13/02newkirk.h20.html.

Dan

[Maja Wilson]

Daniel,

Interesting to hear your thoughts about the ability of a rubric to reflect a conversation. Conversation is the heart of assessment, to me. I can't figure out a way to make rubrics responsive enough to reflect that conversation, though I'm really glad that you're trying. I've written more on this issue of responsiveness, subjectivity, and conversation in assessment in Educational Leadership--if you can forgive the self promotion, the link is here, and I'd be interested in your thoughts on the issue. http://www.ascd.org/publications/educational_leadership/dec07/vol65...

I like Newkirk's article very much, although strangely, I didn't read it until after the book came out, and it was virtually the only critique of rubrics written for a K-12 audience at the time. I'm actually studying with Newkirk now at UNH.


Daniel Sharkovitz said:

Maja,

1. This is such an important question. Rubrics can be helpful and dangerous. Improperly used, they can give the impression that the proverbial emperor wears raiments of gold, when in fact he is actually naked. Properly used, they can be clarifying and inspiring.

2. The rubric concept landed in our school in earnest several years ago. What I have found is that, in general, if my students and I create rubrics on the fly and that we actually connect them to what we are trying to study and learn, that the efficacy of the rubrics improves. (Now don't laugh, but I'm finishing up the first draft of a hypertext poetry writing rubric that my students and I started writing in my Creative Writing class. I'll post it on my website next month. Maybe I'll post it here, too, for comment. I do have some other rubrics posted already that I've written with my students--though frankly I am not happy with any of them.)

Of course, does what I'm saying mean that rubrics need to be re-invented with each new group of students and for each new lesson? I don't know yet. But I think that part of learning how to write something means learning how to reflect upon the qualities it should have when finished not just when it is done, but along the way. Sometimes I think that an invisible rubric, that is to say, the understandings that emerge between writer and teacher during whatever sorts of conversations they have about a piece, can be more helpful to the writer because they are always and forever grafted onto the writing itself.

3. There is an interesting article, "A Mania for Rubrics," by Thomas Newkirk, at http://www.edweek.org/ew/articles/2000/09/13/02newkirk.h20.html.

Dan

[Maja Wilson]

Michael,

You raise an important issue of transparency when you argue that rubrics allow students to see what the person in a position of power (the teacher) wants them to accomplish. I grapple with the same issue in my practice. Here's how I've dealt with it. I give students opportunities to revise every paper, and in the process of responding to drafts, I make it clear what I expect. They can revise until we're both happy with the paper. There is an important difference between the transparency of this process and the transparency provided by a rubric: the transparency that I am able to have with a student during the responding/revision process is flexible. I can take into account a student's emerging intention. I can have my mind changed by our conversation. And believe me, this happens ALL the time. Often, I don't know what I value until I bump into it, so how could I possibly articulate all my values before-hand, even on a self or student generated rubric? I heard Tibetan throat singers on the radio years ago for the first time and had to pull my car to the side of the road because I was weeping. I'd never heard the sound before, and it moved me deeply. I wouldn't have been able to tell you two minutes before I heard it that I valued anything about that unearthly sound. Imagine if that throat singer had read my set of "values" about singing, and decided not to do his gravelly resonant thing because my rubric indicated I wouldn't reward it? Just as I want students to write toward surprise--that thing they didn't know they had to say, I want to be surprised by their writing. By allowing unending revision, and talking clearly and plainly about what I see as we go, I'm able to be both transparent and make room for that surprise. Again, I'm not saying that my way should be mandated. But again, unless I articulate why I do what I do, I'll be forced to use rubrics.

I think you put your finger on quite a bit of the reasoning behind mandating rubrics: "However. I have also dealt with lots of teachers who are frightfully unclear about what they are teaching or why they are teaching it. I think of rubrics mostly as a way of getting clear."

In other words, rubrics are a kind of lowest common denominator. But since they're so often mandated, we enforce the lowest common denominator. And unless we can convincingly articulate that there is different way--or many different ways--to approach assessment, we'll essentially be stuck at the bottom.

[Maja Wilson]

Ken,
I'm loving your pun, and can't get the tune out of my head.
And I actually use that phrase, "rubrics aren't all they're cracked up to be..." in the book. Maybe we were both disgruntled workers in rubric factories in former lives...
Maja

Ken C. said:

To the wizard!

Er, no. E-mail Jim, maybe. This is his Ningdom, after all.

And your book is on my list, Maja. I've been told to follow the Rue Brick Road, too, and frankly I've yet to find one that gets me to the promised land. Maybe it's because rubrics are not all they're cracked up to be? I would say "novel thought," but apparently you've already thought of it (and documented it, too).

[Daniel Sharkovitz]

Hi Maja,

Thank you for suggesting the article in EL. Reading it inspired me to take a look at your book online, where the intro and a chapter were available to read. Now, I'll have to go get the book. Important and necessary thinking about rubrics. Did anyone in the Rubric World come after you? Have you been the recipient of any diatribes? I ask because not too long ago a teacher named Kim Chase published a commentary in Ed Week where she critiqued a professional development session in her district. A lot of folks went online to attack her "audacity" for questioning the system. It was strange.

I plan to give copies of your EL article to my English department colleagues to read and discuss at a future meeting. It will be a good follow up to the Newkirk article I sent around a few years back.

Your statement that, "I was surprised by what I'd written; I'd never thought about the interaction between abstractions and details in quite this way before. In negotiating the conflict between my response and that of Justin's group, I had come to a new insight about writing" was particularly interesting for me. By refusing to depend upon the reductionist pedagogy of the rubric, not only did you help to engage your student in an authentic writing experience, you also open yourself to the opportunity to learn as well. What an extraordinary gift to give to your student and yourself. He must have enjoyed seeing how the conversation he had with you taught you something! I love walking out of school each day with some new understanding of a story or poem.

I am teaching a professional development session on writing this coming Friday and would like to use your EL article as a discussion starter if that is okay w/ you.

Do you have any more info about the Personal Narrative Writing class you taught at the alternative school? The notion about starting writers out with explorations about their own lives can be a helpful way to get students to feel the power and value of their words. I got my start as a teacher teaching a poetry workshop in Boston one evening a week for about 10 years. About a dozen persons would show up on any given night. Some came for a week. Some came for years. There were never any rubrics, only conversations. Writing was not something we did instead of living. It was simply a part of the journey each of us had chosen.

What I like so far about this Ning is that I have enjoyed writing on it. I don't have to. No grades. For those writing on this site, what does this whole process suggest about teaching writing in the classroom? Right now I should be grading papers, but here I am instead.

As a little thought experiment, if students gave us a rubric about how we teach them to write, I wonder what it would look like? What sorts of questions or topics would be on it?

More later, my 11-yr-old needs the computer.

Thank you for your courage. Your students are indeed fortunate.

Dan

[Maja Wilson]

Dan,
I love the connection you make between how excited I felt about my new insight about Justin's paper and how excited you feel when you walk out of the door with a new insight about a poem. You've gotten to the heart of one of my core beliefs: that teachers need to look at students' writing as living texts worthy of interpretation and discussion, rather than variations of formulaic writing that simply need to be responded to by slapping down a template and marking through all the edges that seep over the lines of the rubric or form. In other words, if I ask for a five-paragraph essay, I might as well use a rubric. I don't need to have much "insight"--I just have to identify if the thesis is in the right place or if the intro paragraph is properly funneled. If I encourage a different kind of writing--the kind of writing where form follows function and students are engaged in a constant search for how their emerging intention can best manifest through language and form--then I have to be willing to respond (which is assessment, I think!) in kind--by engaging in that search for meaning with them. Again, I don't think that this process can be reflected (or inspired!) by a rubric.

You ask if I've experienced any push-back. Absolutely! It stuns me, because I'm not telling everyone to abandon their rubrics; I'm trying to argue that there are valid reasons why a teacher would never want to use one. If we can't make this argument, we can't pursue other practices, because rubrics are becoming increasingly mandated. Here's an illustrative story...When I was doing research for the book, I was actually on a county-wide committee charged with creating a county-wide curriculum (there are 5 districts in my county, I think) for ninth grade english that included common, mandated assessments, which, of course, included rubrics. On this committee, I started talking about the ideas I was reading about: writing assessment Brian Huot's discussion of the importance of context for assessment; Patricia Lynne's critique of positivist testing principles (on which rubrics are based); and Bob Broad's contention that rubrics provide a rough and primitive "map" of our values in writing assessment, and that since "we live here," we need a much more detailed (and responsive and contextual) map of our values. Everything going on in this push for common assessment violated what I was reading (and my own instincts), so I began speaking my mind. Ultimately, I was asked to leave the committee. Fortunately, my colleagues in the english department are refreshingly independent-minded humanists, so I made my appeal to them. We organized, wrote a letter, called a meeting with administrators, and asked to pull out of the county's common curriculum and assessment movement. It was an incredibly political move, but it worked. And in the end, I believe it worked because my colleagues and I were united in the belief that we were professionals and didn't need to be told how to assess and teach; we were NOT united in a common abhorrence of rubrics. Some of them love them some rubrics. That wasn't important to me, though. It was simply important to me that thoughtful, talented teachers be given the freedom to make good decisions based on their experience, knowledge of their students, and unique teaching contexts. If we hadn't been united and fought against common mandated assessments, I wouldn't have been free to write the rest of my book, which described my search for a different kind of assessment practice. And certainly, I wouldn't have been as well-prepared for this fight if I hadn't been doing research. Unfortunately, I found that if I spoke my mind, there was less chance of being listened to then if I slapped a published book on the table. So, it is my hope that like-minded teachers will find my book worthy of slapping on the table between them and the forces-that-be.

I love the connection you make between how excited I felt about my new insight about Justin's paper and how excited you feel when you walk out of the door with a new insight about a poem. You've gotten to the heart of one of my core beliefs: that teachers need to look at students' writing as living texts worthy of interpretation and discussion, rather than variations of formulaic writing that simply need to be responded to by slapping down a template and marking through all the edges that seep over the lines of the rubric or form. In other words, if I ask for a five-paragraph essay, I might as well use a rubric. I don't need to have much "insight"--I just have to identify if the thesis is in the right place or if the funnel paragraph is properly funneled. If I encourage a different kind of writing--the kind of writing where form follows function and students are engaged in a constant search for how their emerging intention can best manifest through language and form--then I have to be willing to respond (which is assessment, I think!) in kind--by engaging in that search for meaning with them. Again, I don't think that this process can be reflected (or inspired!) by a rubric.

[Daniel Sharkovitz]

Maja,

Whether we are interrogating the efficacy of rubrics, or other educational practices on this Ning, one of the elements I have very much appreciated is the deep spirit of inquiry that seems to underscore what many people write here.

As Hillocks and others have suggested, student writing improves quite a bit when it is done in the service of authentic inquiry.

From my perspective, one element of our collective inquiry should explore the processes by which ideas become truths, or best practices.

For example, the New England Association of Schools and Colleges is revising their guidelines for accreditation. The new policies will be in effect in 2011. Public high schools in their region will be required to do what this organization says they must in order to become accredited. Now I have nothing against schools being accredited. I know that many years ago the organization's recommendation got our School Committee to find money to buy a thousand new books to get our library up to standard. I thought that was a grand idea.

But the new policy (see attached NEASC PPT) mandates that standards will be based upon research and best practices. Rubrics, essential questions and concepts must be used, according to the policy. The implication of course is that a certain way of developing curriculum materials and teaching known as the Understanding by Design system developed by Wiggins & McTighe will gain hegemony over other systems. Is that a good thing? I am not against anything in the UBD program, so far as I know. I use some of their ideas--to date because I want to.

I was there at Brown University in the early 1980s when Grant Wiggins was developing many of his ideas including the concept of the public exhibition of student work.

Clearly there are many systems available for teachers to use to help them plan lessons, from UBD to Madeline Hunter’s Lesson Plan Format, to whatever. I think, however, that problems begin when one or another system gains hegemony over others. Development, inquiry, improvement can slow or stop altogether.

How many of us are teaching in schools where the regional Accrediting Agency holds enormous power over decisions that should be left to the local schools? Requiring a library to have books is one thing, but requiring thousands of teachers in a particular region to march in lockstep to a particular curriculum drumbeat is anathema to educational innovation and teacher creativity. No Accrediting Agency should become a Ministry of Culture.

I think some questions worth exploring are:

How do accrediting agencies decide what is best practice?
What role if any do publishers and curriculum businesses play in determining what the accrediting standards will be?
How do accrediting agencies in other parts of the US & in other countries develop their standards?

Dan

[David B. Cohen]

I find myself coming to this discussion at a time when, after close to fifteen years in teaching, I've embraced the use of rubrics much more this year than in the past. In much of the discussion here, I find myself curious about the teaching context of everyone here - what age, what type of classes, what type of school? I'm also curious to know if we're all talking about the same type of rubrics. At some point (maybe in a comment on the video), I was disappointed to see an anecdote about a student graded down because of a picky formatting idea. When rubrics correspond to arbitrary quantities of fictional points and formulaic grade calculations, then I'm not interested either. My rubrics are more qualitative/descriptive, and there's a generic version that I modify according to the type of assessment. The consistency helps students internalize certain concepts like audience, purpose, evidence, and writing conventions, while the variations in rubrics help them understand the idea of adopting different approaches to writing, rather than always attempting the same thing. There are no formulas or set percentages for anything, but I wouldn't feel comfortable suggesting that I'm not looking for certain characteristics in their writing, or that I'm equally open to whatever they produce; I think it serves students well to know that most readers, most of the time, are looking for certain traits as well. I wouldn't write a work of fiction with a thesis statement at the end of my intro paragraph, and I wouldn't dream of submitting a thoroughly colloquial text with loose adherence to conventions for consideration in an academic publication. Why not have rubrics that condense this information and help students see that my response to their academic work is likely to be grounded in something predictable and consistent, rather than variable or personal?

I do like to tell students the story of a time when I took a risk in my own writing. I was struggling with an essay about My Antonia, for my American Lit. survey in college, and I was also studying acting at the same time. At some point, I put myself in the place of a minor character, the nameless friend of Jim Burden who is the recipient of Jim's manuscript. It's sort of a frame novel, except Cather doesn't close the frame - that character never reappears. In a moment of inspiration, I decided to write my essay from the point of view of that character - what did he think of Jim and Antonia after reading this manuscript? I went all out, using first person throughout, inventing my own fondness for the landscape, etc. However, I assumed that my work would be graded according to its success at presenting a clear argument and supporting it effectively, rather than on simply avoiding the word "I." In a way, I inferred a rubric where none was provided. The T.A. who was supposed to grade the essay decided it was "too different" and passed it up. The professor loved it and I earned an A. So, if there were an overly strict rubric, I would have been denied one of my all-time favorite writing experiences. If there were any kind of rubric at all, it might not have seemed so risky and more students might have enjoyed a similar experience.

So what do I do as a teacher? I tell them this story. I tell my students that I don't want to see "I think..." randomly floating around in their analytical writing. I also encourage personal response essays sometimes, (and rewrites any time). So, if "I" will be part of the essay, it should be an integral part, not a lapse in attention to certain academic stylistics that are worth practicing, though not worth worshipping.

[Maja Wilson]

David,
I love your story! It is excellent that you keep it in mind as you encourage your own students to take risks. I'm not convinced that this risk HAS to be encouraged through a rubric, but I think it is so critical that you keep this experience in mind as you use your own rubrics.

Like you, I think that a writer's interaction with an audience (real or imagined--see Walter Ong for an interesting description of how EVERY audience is imagined, in a sense) is critical to hir development. Audience and intention, to me, frame every other decision we make as writers. In fact, a real reader's response is the best kind of assessment, in my mind. Because of this, I try to articulate what is going on inside the "black box" of my mind and being as I read. In my experience, what goes on inside my mind when I read a piece of writing is too nuanced and detailed to reduce to a chart that applies to all pieces of writing, even within a specific genre, so I basically do it from scratch each time. Mind you, I LOVE starting fresh each time, because it reminds me not to approach assessment automatically, on autopilot. Some people want a way to streamline the process. I would be bored out of my mind if I did so (and I HAVE been bored out of my mind when I've done so!) I try to only assign papers I want to read. (And I don't "only" teach personal writing--I've an extended post on the teaching research paper discussion that outlines how I teach research writing.) Once I articulate my response, I try to talk with students about what kind of responses they'd LIKE to create, and help them see how different choices and approaches will change my response. This is the way I negotiate my expectations and the students' intentions; the alternate to rubrics is not a free-for-all, in my mind, though in my own practice I do make a concerted effort to make room for, develop, revise, and in all other ways focus on a student's intention. Yes, I'm have quite a bit of experience writing, and I don't hesitate to USE my experience as I talk with students about their work, but in the end, it is a student's motivation that will carry hir through the tough work of thinking/writing/revising, and as a teacher, I simply can't teach well if the student isn't invested. Focusing on the student's intent (and helping hir develop that intent!) is crucial to my teaching.

[Helen MacDonell]

I like what you say about intention. Getting the students to understand that they need to have something they want to say before they get into the writing is key. Unfortunately many school assignments do not really allow the students to search for their own topics and voices. I have found that personal writing is a joy to read because students have a much greater sense of intention. I have no idea how to make analytical literary essays really engaging.

I think that by now, many students have seen so many rubrics that they have internalized some of the typical criteria found in them and I think many of them know how to get a good mark; however I have not really seen students use rubrics to help them grow as writers. I agree that growth happens in the conversation with the teacher or the writing circle, ideally with peers.

I look forward to reading the articles you refer to. I have always dreaded the minefield of returning students' work with a vague number and a rubric full of circled criteria that cannot possibly name the variety of qualities we find in a set of papers, especially--as you have mentioned--for the students who attempt to take risks.

[Tracie Weisz]

Maja - I have to agree with a comment made by Michael earlier, in that it's vital that when we give students writing assignments, we need to be clear about our expectations. Rubrics can take all kinds of forms (I have probably used them all and created just as many on my own!). I think the most useful ones are not for the purpose of a grade or assigning points, but they are, to a degree, a measure of quality.

We know that quality writing comes in many shapes and sizes, but if we are asking our students to write a literary analysis for example, they need a very clear picture of what a quality literary analysis should look like in terms of discussion of content, making connections to themes, citing supporting passage from the text or elsewhere, and the inclusion of their own thoughts or feelings about these ideas.

They should also be aware that although conventions won't sink their paper, it's very important that it is readable and and is written with a level of formality that suits both the subject and the audience. I will add to this that when we are doing creative writing, my rubrics only address the structures and specific stylings we are studying that I expect them to bring to a particular genre. When writing poetry, I usually only employ a rubric for studying a specific form or structure, and the rubric applies to the students' demonstrated understanding of that structure.

However, these days I am thinking about writing far beyond the standard essays, short stories, literary analyses or any of the "traditional" forms of writing we have used in English classrooms. Do we make rubrics for blogs? How about comments to blogs? There are all kinds of collaborative communications that more and more classrooms and students use for writing that they did not a few years ago. They have to be assessed, because those are the rules we must live by. I would like to hear more about how people are addressing writing assessment for these "21st century" communications.