Standardized Testing

The LA Times had a piece over the weekend about Value Added Assessment as a way of measuring teacher success or failure.

And this has sparked new posts (or links to older posts) debunking and critiquing the use of Value Added Assessment.

And the same things happens around discussions of teacher merit pay, school closings, and how - under NCLB - schools are identified as failing or making progress.

And while these are all discussions that need to happen, the process of engaging the issue provides a tacit acceptance of the underlying argument: that standardized tests measure something that is important, and that these measurements are sufficiently valuable and accurate as to be a worthwhile mechanism upon which to base life-changing decisions.

The discussions around merit pay, value added assessment, school closing, teacher effectiveness and value added assessment - these conversations slip in the legitimacy of standardized tests through the back door, Trojan-horse style.

And the point of this piece is not to say that standardized tests do not have a role to play. But they should not be seen as the primary measure of the success or failure of any program. Really, any single measure used exclusively will not provide an adequately detailed view.

I want to see standardized test scores that norm for the child that comes to school that day without breakfast - and without knowing whether dinner is coming - due to food insecurity.

I want to see standardized test scores normed for hours of art instruction, and rates of obesity.

I want to see standardized test results that norm for the quality of that child's kindergarten teacher (which, of course, sets up the quandary of how we would discern good from bad).

I want to see some level of consensus - free from political gamesmanship - that standardized tests correlate to some definition of success besides the ability to do well on other standardized tests.

In the ongoing conversations about what makes for meaningful change, we should all be looking at the following questions, either before - or as part of - ongoing discussions around new education policy:

• What do standardized tests measure? To what does success (or failure) on a standardized test correlate?
• What are the costs and benefits of a fully standardized curriculum? What does it mean when the same companies that profit from selling materials in support of a standardized curriculum play a guiding role in writing the standards?

One way of changing education is to change how we assess learning.

This isn't going to be a post about standards, but we need to start with them to get into the center of the discussion (this is not to say that standards are not a subject worthy of close consideration; rather, they are just not the main focus here, today).

• Standards define curricular goals and objectives.
• Textbook companies prepare packaged materials that are "aligned to the standards." These textbooks, in theory, are designed to address the curricular goals and objectives as defined by the standards (and for fun, ask a textbook rep to demonstrate how their texts "align to the standards." Ask them to define the process by which the texts are "aligned to standards." Then, get out the boots, and enjoy the hijinks that will ensue).
• Student learning is measured by a standardized test that claims to assess a student's base of knowledge as measured against the standard.
• The "quality" of a school is determined (in part or in whole) by how students have done on the test. Test results can be a key factor in closing down schools.
• The "quality" of a teacher is determined (and in many of the merit pay schemes, teachers are rewarded or punished) based on student scores on these tests.

So, let's take an enormous, completely unjustifiable leap of faith and assume that the standards actually define something meaningful, for one reason and one reason only: this post is not about standards, it's about assessment.

When a curriculum is defined by a pre-packaged text, teachers and students are relegated to content consumers. Teachers get the text; they deliver the text; they test on the text, and teacher effectiveness is tied to how students perform on the test that purportedly measures how well students "know" the content that has been delivered to them. Any process used to "learn" the material is overshadowed by the means of assessment that defines the experience, and defines one's success or failure within that experience.

It's also worth noting that in lower performing schools, there is more motivation to stick with the "proven" or "traditional" route of using a standards-aligned text, as this provides a level of cover and plausible deniability should a school not meet growth goals. In an environment where sanctions accompany low test scores, using alternative means of working with kids is equated with gambling with kid's futures - unless, of course it's happening under the auspices of TFA, KIPP, or a charter school. Higher performing schools - where socioeconomic level appears to play a role - tend to have more freedom to experiment, largely because the threat of sanctions for "failure" is missing.

This is why serious discussions about assessment are a necessary part of the dialogue around improving education. What would an educational environment look like where, in addition to or instead of a standardized test, students had the opportunity to show their mastery via two portfolios: one defined by the school, and the second defined by the student?

The process of building a portfolio (ie, of crafting the assessment) is also a learning process. Selecting and justifying elements in a portfolio requires a level of critical, reflective thought that is not present in either preparing for or taking current standardized tests. It's a more efficient means of mastering both material and life skills than the assessments that currently claim to measure those skills.

What would teacher professional development look like if a teacher was assessed on how they provided feedback on student work? What if teachers developed professional portfolios that included curriculum they developed, modified, collaborated on, and/or shared? Most teachers create curriculum on a regular basis as workarounds for sections of the text that are weak or not suited for their classroom; what if creating and sharing these units was made an explicit requirement for growth and development as a teacher? What if this ongoing creativity and collaboration was a factor in assessing an educator's professional growth?

These shifts are possible now; they require a change in how we look at assessment, which potentially could inform changes in what and how we teach.

Changing assessment is hard. Generally, more individualized assessment takes more time. From a business place, it's hard to plan a "disruptive" business around this because you can't really streamline the time required for good feedback. The challenge (and therefore the opportunity here) is to make tools that simplify and streamline creating portfolios of work that demonstrate learning. The benefit - especially when compared to other forms of evaluation, and certainly to standardized testing - is that the process of creating and justifying the artifacts that demonstrate learning is also a process that supports and reinforces learning.

But this is a subtle point, and one that is often buried beneath the time required to assess portfolio-based projects versus the time required to process a standardized test. Ironically, the quest for efficiency in assessment has occurred at the expense of efficiency in learning.

In an Op-Ed written by Ok Go member Damian Kulash Jr, he describes how his record company, with the help of YouTube, has made it impossible to embed their new videos in external sites. Since the change, there has been a marked drop in viewings of the Here It Goes Again video:

When EMI disabled the embedding feature, views of our treadmill video dropped 90 percent, from about 10,000 per day to just over 1,000. Our last royalty statement from the label, which covered six months of streams, shows a whopping $27.77 credit to our account.

(Seriously, take some time and click on the above link to the video. $27.77 for 6 months? Wow)

Leaving aside the incredible shortsightedness of a policy that eliminates free marketing, I was struck by the deadweight that musicians carry with them as part of a record deal. Industry lawyers needed to vet this change in policy, industry PR and Marketing people needed to articulate this new strategy, and these people's salaries are paid by the talent of the musicians "represented" by the label.

(Photo credit: betseyweber, original available here)

Except that, as this situation clearly demonstrates, the business needs of the recording industry often run counter to the creative needs and processes of musicians. The recording industry is bloated with administrative deadweight that contributes little to nothing to the people creating music.

And the more I think about it, the more I see parallels between what is happening with music and what is happening with education.

In education, as in music, there is an Education Industry that has differing goals and interests of the people upon whom they depend for their continued existence. While the goals of the education industry do not need to diverge from the actual needs of students and teachers, in practice the needs of the education industry require that organizations spend money that has little or nothing to do with student performance.

A broad range of companies and organizations lobby Congress about Education (at the national level, as well as at the state and local levels); several of the larger textbook companies spend money for lobbyists. It's almost like they lobby on issues that dovetail with the products they sell, to the point where it's difficult to tell if the product line follows the legal requirements, or the legal requirements follow the product line. But in any case, it's probably easier to tease out the relationship between lobbyist influence and business needs than the relationship that any of this has to making schools a better place.

It's also worth noting the money spent on lobbyists spent by the College Board -- you know, the people that brings us the SAT and Friends, and who create a need for a test-preparation industry that rakes in millions from anxious teens and their parents. These players occupy space in the educational market, and generate an incredible amount of revenue, yet I doubt you will find many people making the argument that learning how to take the SAT brings greater intellectual growth than creating a portfolio. What could happen if the resources poured into test preparation were instead directed towards Arts education, or -- to help counter increased rates of obesity -- Outdoor Education?

The money spent on testing becomes even more questionable when one considers the apparent correlation between family income and student performance on tests.

The existing conference model of professional development provides another example where industry needs have diverged from the needs of teachers and learners. Virtually all of these conferences have a vendor driven model (about the only exception I know of is Educon). For vendors, these conferences are an opportunity to connect with (aka market to) the people they view as their customers. For attendees, however, this means that a portion of their professional development experience is cluttered with sessions devoted to salespeople extolling the virtues of their product in the educational process. There doesn't need to be a disconnect between the needs of companies and their customers, but the vendor-driven model tends to put the product as the focus, as opposed to the needs of the learner. A more authentic approach -- and the approach taken by many classroom teachers -- is to assess what students need, and assemble resources based on that need. However, using technology in response to a need -- rather than buying up technology because vendors offer district-wide rollouts -- doesn't help people make sales quotas, and doesn't convince vendors that their sponsorship dollars are well spent.

The next time you are at a conference, wander onto the vendor floor. Do a quick count of the number of booths, and the number of people at each booth, and what the vendors are giving away. The cost of this marketing is reflected in the cost of products sold to teachers, schools, and districts. Given that schools are increasingly being held accountable for the cost of supplies (how many teachers have had to supply paper for your classroom?), it would be nice to see something that resembles restraint from organizations that profit directly from schools.

The process of reworking how we teach and learn requires more than just changing how the interpersonal exchanges between teacher and student take place; it will also require a critical rethinking of how we assess learning, and a critical rethinking of how we equip ourselves to teach and learn. Many companies make a tidy profit within the current supply chain, and they will not passively go away simply because there are better ways of doing things that renders them obsolete. And in case you're wondering, a teacher's lobby doesn't feel like the answer; the American Federation of Teachers has been pretty active, and they haven't done much. I don't know whether we need to get on or off the treadmill, but we certainly need to get moving. The educator's voice -- the principals and teachers working in schools -- has been largely absent from the conversation. How can the people doing the actual work become more involved in shaping the policy?

From an article in the New York Times, it appears that the Princeton Review "published the personal data and standardized test scores of tens of thousands of Florida students on its Web site, where they were available for seven weeks."

According to the article, the breach -- likely caused by human error -- exposed some very sensitive data:

"One file on the site contained information on about 34,000 students in the public schools in Sarasota, Fla., where the Princeton Review was hired to build an online tool to help the county measure students’ academic progress. The file included the students’ birthdays and ethnicities, whether they had learning disabilities, whether English was their second language, and their level of performance on the Florida Comprehensive Assessment Test, which is given to students in grades 3 to 11."

In another folder on the same server, several files containing the names and birthdays of 74,000 students from Fairfax County, VA, were exposed.

On a related note, a prominent test preparation company will soon be looking for an experienced web security expert. The qualified applicant should be able to meet or exceed the skillset laid out in this document.

From an Op-Ed in the June 1 online edition of the NY Times by Brian Greene: Put a Little Science in Your Life