I can easily show that standardized tests measure (at a minimum) the ability to read the test.
This alone was a huge jump in the average literacy in my state when they first implemented a functional literacy exam.
There are a number of problems with that claim. The first is causality — there's a fine post hoc fallacy here. Where's the evidence that the standardized tests caused the improvement in literacy? The improvement in literacy and the standardized tests could be the result of a broad, renewed focus on literacy. It's a beautiful irony, that a comment in support of standardized testing contains a serious error that would not be caught by that testing! Personally, I'd much prefer someone who can't spell 'work' over someone who can't recognize what is probably the most common logical fallacy. The first is an annoyance, while the second is an outright danger to society. Errors like this one make it possible for alties to flourish, preying on the ignorant, as Orac will tell you — often in excruciating detail.
(There's also a certain amount of irony in the grammar of the post.)
Second, the measure of literacy is the standardized test. The improvement in literacy could just as well be an artefact of the test itself. sgent claims that literacy improved when the literacy exam was instituted — what was the measure of literacy before and after the exam was begun?
I should also note that if the functional literacy exam noted above was this one, it contains a subjective component. The students must actually compose a letter, among other tasks. I have fewer problems with this kind of test than the much more common guess-and-bubble tests.
Frankly, since we've been making testing more and more important, I trust the results less and less. When people's jobs, prestige and power depend on the results of tests like these, they're going to do their best to game the system so that they come out ahead. This gaming happens to the detriment of the students. I've watched good teachers struggle with this question: "Do I teach the subject, or do I teach the test?" I've also watched some of those same teachers hate themselves and the system for the answer that they were forced into, bemoaning the loss of instructional time to test preparation.
In the previous post, I posed the question of "what are our goals?" The follow-on question is "once we have our goals, how do we measure them?" I have a good idea of what my answer to the first question would be. Anyone who has watched a five year old knows that young children are little scientists, musicians, inventors, story tellers and artists. I'd like to see them come out the other end of the education system just the same. Not all scientists in the the mold of, say PZ Myers, but curious about the world (which is what is at the core of every branch of science.) Not musicians in the sense of Isaac Stern, but able to appreciate music outside of the genre that defines their generation, and able to make music for themselves and their friends. Not inventors like Thomas Edison, but able to apply their curiosity and knowlege to solving problems. I'm not looking for another Shakespere, but someone who can tell a story clearly from beginning to end. Nor artists with the vision and skill of Van Gogh, but able to make and appreciate art for themselves.
I think that sgent's comment sets an extremely low bar. He/she seems happy with the fact that a base literacy rate has risen. I don't know if s/he is content with that, but I know that I am not. That doesn't begin to come close to the goals that I've set above.
I also know what my idealist's answer to the second question would be. In an ideal world, assessment of a student's progress would be made at a broad level, not at a detailed level. "How well does someone write," not "do they know how to spell these 15 words." There are variations in people's abilities — I'm far more interested in knowing that someone is working to their ability, than I am in knowing how well they do on a somewhat arbitrary scale. Of course, assessing potential is even harder than assessing someone's current state.
In a practical world, I know that this ideal is impossible. Subjective assessments are always difficult, and even more difficult to standardize. They're also resource-consuming, and we're already wasting huge amounts of effort with standardized testing.
So, I really don't have an answer to my own challenge. I truly wish that I did.