The Cost of Field Testing

http://student.collegeboard.org/?student

     This week our Freshmen students took the first two tests in the latest version of Texas' standardized testing program STAAR (State of Texas Assessment of Academic Readiness).  At the same time our Juniors were being tested in a field test on the same subject areas.  I had the opportunity to be one of the proctors for the Junior tests.

     I want to make sure that I say, up front, that the staff here tried to take these field tests very seriously.  We tried to get the students "excited" (if that's even possible) about taking them.  We went through all of the steps you do when you are administering the test as it is required by law.  The one thing we could not control was how the students would act during the testing.

     Student conduct on field testing drives me crazy and makes me do things like write this post.  As someone with a degree in mathematics (specifically statistics), I question the validity of data obtained from this "random sample" of students taking the tests.  Yes, you can randomly select schools to do the field testing.  But the data will be incredibly skewed by the attitudes of the test takers at those schools.  And, high schoolers WILL skew data if they decide to do that.

    Anyone who has seen Freshmen take a standardized test knows that there will be a certain percentage of students who will "christmas tree" the test.  Christmas treeing is quite fascinating.  Some students will select a letter to bubble in, say an A, and the entire test will be that letter.  Most students just bubble in A,B,C,D,A,... so there is a beautiful pattern of diagonal dots.  Others who are more imaginative will turn the answer sheet sideways and will try to spell out a message such as THIS TEST SUCKS with the dots.

     In Texas students get it in their head that the only test that "matters" is the tests they take as Juniors.  That allows for a huge (still less than 10% ?)  percentage of students christmas treeing as Freshmen.  That number, from experience not official data, is cut in half during the sophomore tests.  But there are a LOT of students doing this and it effects the score schools receive for the tests.  When this happens throughout the state the end results from school to school are statistically relevant and so we can justify the scoring that is used by the state to grade the school.

     Now when you place the fact that a percentage of high schoolers WILL not take a test seriously if it "doesn't count."  And, you then make that test a four hour test where they have to read passages and write essays and the "give-a-darn" (GAD) factor drops to a minimum level.   When the GAD reaches a critically low level some interesting things can happen.

    Take, for example, the student who christmas treed the entire test answer book even though the book was for two different tests and the test had a line that said "STOP and turn your test into the administrator" after the first test.  What was he thinking?  He was thinking "I want to finish this test as soon as possible so I can get out of here."  Or the students who were discussing (after the test had ended) what they had written.  As I was telling them that they can not discuss the test I heard one say that they had written Taylor Swift lyrics as their essay.  What they were doing was imaginatively writing essays that meant something to them - no matter what the prompt was about. (But it DOES make you wonder what the prompt was - was it really that bad or did they never even read it?)

     We have pretty darn good students.  They perform at a high level when they are asked and when the result of their efforts is meaningful to them.  If our school could be graded by the results of tests on every subject except math we would have received the highest rating given to schools in Texas in 2011.  These kids know how to take standardized tests and they know how to do well on them.

     So, how much did it "cost" to take the field test?  Well there were tests and answer booklets printed for our entire Junior class.  The Juniors lost about 5 or 6 hours of instructional time over two separate days.  Teachers were out of their classrooms monitoring the testing and administrators were involved with testing instead of being available for students, parents, and teachers.  Now that the tests have been given they will have to be graded - how much of the contract is taken up by that cost?

     The entirety of the cost can only be estimated. But the bottom line is that whatever dollar amount you want to assign to this field test has to be evaluated towards the validity of the test results.  I would estimate that between 20 and 40% of our students ended up not completing the test in a serious fashion. I would also be bold enough to say that another 10 to 20% wanted to just get it over as fast as possible. The end result is I can safely say that at least 50% of the students gave less than their full effort on that field test.  So how relevant will the data be from this test?  Not relevant at all and that is money that has been totally wasted.

Manor Plays the State Assessment Game

http://www.flickr.com/photos/califmom/

     Today I attended our district's meeting to lock in details for how the state's new End of Course (EOC) exams will effect our students.  We met to discuss key issues concerning how we would convert the scale score to a grade and how (or if) these exams would effect the Grade Point Average (GPA) and Class Rank.

     Before I say anything further I want to say that I was very impressed with how we really attempted to interpret "suggestions" by the Texas Education Association (TEA) regarding these topics.  This was really made evident when we heard anecdotal accounts of what other districts in the area were considering for their answers to the same questions.

     What was frustrating, for all, was that fine line we kept treading between what is best for the district and what is best for our students.  We looked at how tests would be perceived by students.  How students AND parents would view the results of the tests.  And we discussed whether we needed to have the results effect class rank (which lead to thoughts of parental and district law suits over said results).

     The first item of business was to decide how we wanted to convert the state scale score to a graded score to be averaged into the student's grade.  This seems minor but in previous meetings we had, pretty much, decided to no longer award 0.5 credits for completion of each semester and just award 1.0 credit for completion of the entire year.  As for the grading score we had to start with a fictitious scale score because the state hasn't created one yet.  And, each test will have a sliding score along that scale for passing and failing.  By the end of the meeting we had agreed to a set score for the categories of  minimum, satisfactory, and advanced.  What we all agreed to not do is have a graduated scale but one set score for each of the  categories.

     What we ended the meeting with was discussing what the minimum score would be.  We never locked in a value but we are definitely willing to hold our students more accountable to the results of the test than other districts in the area.  One district, it is said, is considering having a score of 100 for a satisfactory score and a 70 for anything below that score.  So, no matter how their students do on the test, they will pass it.  Sign your name?  You passed.  Are you kidding me?  We never came to consensus but let's say we are willing to go below a 70 for scores below that mark on the state scale score.

     An interesting phenomenon occurred as we were in this discussion.  We, as educators, are willing to fight for the rights of those kids in "the danger zone," (just at passing going into the EOC) but we don't always think about how our top tier of students will do on these tests.   We were able to have some good thoughts on how these scores would effect GPA and Class Rank.  We came to consensus that the tests would be included in these two areas but we didn't just gloss it over.  We really took our time considering all of the angles.

     It is worth noting that a definite feel of affluent districts versus less-affluent districts came out of our discussions.  It really seemed to most of us that some districts are doing things in anticipation of lawsuits and are almost daring the state to create mandates that are worthy of bringing forth litigation.  Many of the other districts, like ours, are trying to give a good faith effort to impose changes that will reflect the intentions of the TEA.  We will be ready in January to present our decisions to the school board for their blessings and, come May, our students will have a system in place to convert their score on the EOC exams to a grade that will be averaged in with their other grades for the year in that subject.

     Once more I am pleasantly surprised at how our school district is able to come to consensus while letting all sides present opinions.  I think we can all agree that TEA really hasn't been able to think through with this albatross they've strung around our necks.  But we don't plan on drowning.  We're treading water and moving towards the high ground.