Joan Herman is the director of the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) at the University of California, Los Angeles. Noted for her ability to bridge research and practice, Herman is also the author of several books, including Tracking Your School’s Success: A Guide to Sensible School-Based Evaluation. She is a past president of the California Educational Research Association and has served on committees for the National Academies, the American Educational Research Association, and the National Organization of Research Centers. Herman spoke with Education Northwest’s Edith Gummer by phone. Their conversation has been edited for brevity and clarity.

Edith Gummer: How has data use, at the school level, changed over the past decade?

Joan Herman: There’s more attention at every level—state, district, school—to using data. Part of that is attributable to the explosion of available data, due to all the testing that’s going on now in schools. And new technology has also fueled greater interest in data use, by making the data more easily available. Teachers and administrators are certainly talking the talk now. They know they’re supposed to be using data, whether it’s state data to align their curriculum and identify students who are struggling or interim data to monitor progress or—more and more in the last five years—classroom formative assessment data. I think teachers understand that they’re supposed to be looking at how students are doing on those assessments and making instructional decisions accordingly. They get it.

Do you think we’re approaching a situation in which there is too much data for teachers to handle?

I think, typically, there is more data than teachers know what to do with, yes. And I think this is true for both teachers and administrators. They see it as a lot of numbers, and they don’t necessarily know how to make sense of it. They don’t always approach the data with questions that they want the data to help answer. If you don’t approach it that way, it can become overwhelming.

Is it too often a case of, “I have all this data and I know I’m supposed to be using it,” as opposed to, “I have these questions that I need data to answer, now let’s find the appropriate data?”

Yes, exactly. From my perspective you start with the questions and then you see what data are available that can help you answer those questions. In most cases you have multiple sources of data that can help you. In approaching that, it should be more about digging down into the specific data you need, rather than looking at all the state data and all the interim data—regardless of the subject area—and trying to apply it to what’s going on in the classroom. Instead of that piecemeal approach, you need to take a coordinated look across those data sources with regard to the specific question or questions you’re trying to address.

Do you think teachers have the right data they need in order to make effective decisions in the classroom?

The simplest way to answer that is: The data that teachers need to make decisions in the classroom are classroom data. Data from an annual test can only go so far. That can help you understand the strengths and weaknesses of your curriculum—assuming the test assesses the right things—and it can help you determine which students are struggling and the depth of their struggle. Basically, the annual data help you set some general parameters. Similarly, good benchmark tests can give you snapshots of how students are doing at particular points in the year. But again, the best that you can do is to give feedback on major subskills. The data that are most powerful in the classroom, as research suggests, are the data that teachers acquire minute by minute, day by day as they’re teaching.

In that vein, do you think classroom-level formative assessment is something that’s becoming widespread now? Is it being done effectively?

I think the available research suggests that teachers—in their preservice and inservice training—haven’t had enough attention given to formative assessment. Their backgrounds don’t always give them the sophisticated content and technological knowledge they need to do it well. And the curriculum materials they’re typically using don’t give them good guidance on solid formative assessments to use either, whether that’s formal assessments that come with textbooks or ideas for substantive prompts they can use in classroom discussions to see how students are acquiring understanding and what misconceptions may exist.

Are professional learning communities, among teachers, one mechanism to move teachers’ knowledge about formative assessment forward?

I think professional learning communities can be really important in the whole saga of data use, starting with getting teachers together to analyze the annual state assessment and to consider, on a regular basis, how kids are doing with regard to important standards for learning. I would take those professional learning communities down to every level of data use. And I think that raises the important issue of how to create a culture of data use within a school, as well as a culture of support for professional learning and collaboration in general. That is essential to having effective data use that actually informs better teaching and learning.

Are there any programs that you know of that are currently providing teachers with the professional development they need to use data effectively?

Margaret Heritage has done good work with the New York Comprehensive Center. There are examples in that work of a professional development program that has helped teachers implement formative assessment. Similarly, I think Margaret Heritage has worked in Arkansas and in the Pacific Northwest as well. So yes, I think there are some states that are moving ahead and trying to provide their teachers, schools, and districts with resources to help build teachers’ formative assessment capacity. That’s one area of professional development that is taking place.

Can you point to those examples and say: Here’s something that’s working?

You can say: Here’s one approach that seem to be working. From my perspective, this isn’t just about formative assessment, it’s about bringing a formative assessment perspective together with the right instructional and assessment resources that teachers can use to fuel those formative assessments. And it’s about having a supportive professional learning community. All of this has to be combined and well orchestrated to be effective. I think it’s reasonable to assume that the first time a teacher teaches something they’re giving it their best shot. If that doesn’t work or if some kids are still struggling after that initial approach, teachers don’t necessarily have ready-made strategies for what to do next—they’ve already tried the best thing knew to try. That’s what makes the collaboration with other teachers so important. And that collaboration and support shouldn’t be limited to their own school; it’s also about having access to resources and to experts and networks outside the school that can help bring in new options. It’s about opening new windows for teachers and, most importantly, for students.

What’s the single most important thing that a principal or school administrative team can do to improve how data are used in the classroom?

I think the most important thing a school can do is have a vision of what it is trying to accomplish. What are the most central or essential learning goals? They need to have a performance orientation and they need to build a solid community and commitment to that vision. And that vision and commitment should be one of accountability. That means: We are responsible for our kids’ learning, and we’re going to keep our eye on the data to see how we’re doing, and we’re going to use that data to improve our instruction. Everyone needs to agree on that and be fully committed to it.

And yet, that’s something that schools have supposedly been doing for years—creating vision statements. In my own teaching experience it was always very frustrating at the beginning of each school year, trying to lay out the vision of what we wanted students to know and be able to do.

OK, but it’s not only about what we want kids to know and be able to do. It’s also about keeping an eye on that effort, continually, across the year. I think that is new. Now, it’s about saying: We’re here and we want to be here, what are the steps we’re going to take to get from point A to point B? And how are we going to know whether that’s working? And the answer is, by keeping our attention focused on what the data are telling us about how kids are doing and how they’re performing and whether they’re learning. It’s not about getting together once a year and holding hands and singing “Kumbaya.” It’s about continually being accountable for every kid’s learning, every minute, every day.

Do you think that school administrators, in general, have the knowledge and skills to be able to do that with their staff?

Some do, some don’t. Just as there is a teacher professional development problem, there is also an administrator professional development problem.

What is a common mistake schools and teachers make in terms of data use?

Schools often look to the data to provide answers. The data really don’t provide answers. Instead, the data raise questions and they can stimulate discussion: What’s going on here? What’s the source of these strengths and weaknesses? The important link in formative assessment is not “What’s the problem?” That’s part of the process, but the most important step is: “What are we going to do about it? What are the implications for next steps?” That’s not only a critical step but a critical challenge in the process, and it’s an additional reason why professional learning communities are so important.

What should happen at the state and district levels to facilitate better data use?

With so many states involved in the development of longitudinal databases there has been a lot of attention to amassing the data. There needs to be equal attention paid to providing those data in digestible and usable forms to teachers and administrators. Conveying the data in ways that are usable is the number one priority for states. And certainly, districts can set the tone for a culture that is committed to and accountable for student performance and to using data to improve that performance. I think that commitment sometimes gets sidetracked by having a narrow focus on formal data. For example, people don’t always think of teacher observations and professional judgment as a valid part of the data.

Is that because it’s more difficult to quantify or because it’s seen as less reliable?

Both, probably. But I also think there’s the instinct, in education, to look for the magic Band-Aid. And the use of assessment data has become one of those current magic Band-Aids. But to reiterate: The data raise questions; they don’t provide answers. It’s the professional—the teacher or the grade-level team—that needs to be able to analyze and infer from the data: What’s going on? And the formal data only tell part of the story. The other part of the story comes from professional insight and the professional experience to be able to diagnose the situation and –most importantly—to know what to do about it.

So what can the state do? Realistically, what kind of guidance can come from the state that would really have an impact?

The states can strengthen teachers’ capacity to engage in data use. And by that I mean the whole range from understanding how to use state data to inform instruction, all the way down to the daily, minute-by-minute data from specific lessons: how to integrate assessment into ongoing instruction. I think the state can do that through credentialing requirements and through certification procedures—certification of both teacher preparation institutions and certification of the people who become teachers. It’s a continuing problem. It never seems to get solved, but it could be solved, and state regulations and state policies could go a long way toward solving it.

Another big leverage point is the state longitudinal databases. It’s not only a matter of making the data available, but also of providing training—or at the very least templates—for schools and teachers to use that can help them ask some of the questions that ought to be foremost in their minds. How are our students doing? Are things getting better or worse? Where are the largest gaps? Where are the strengths? How can we build on the strengths and address the weaknesses? States can provide systems that at least give educators a starting set of questions and analyses to help fuel their thinking. Down the line, I think you want data systems that enable schools to query the data in ways that are sensitive to their needs.

Are there some states you can think of that are good examples of an effective system?

Arkansas. I’m not familiar with what every state is doing, but I am familiar with what Arkansas is doing, and I can point to them as a good example.

Is that based on Neil Gibson’s work?

Yes, exactly. There may be other states that are also very advanced in this; I just happen to know about Arkansas. In Arkansas there has been attention paid to reporting the data out at different levels for different users, which I think is really important. Secondly, they have emphasized professional development. They’re customizing the data use resources to respond to user perspectives and they’re providing training and professional development to enable users to access those resources.

I can think of some other states: Florida and Louisiana, for example. There are a number of states that are beginning to develop this. In the Pacific Northwest, there is the work Oregon is doing with its longitudinal data system and the training materials it’s developed.

Yes, you’re right about those states. I think, to the extent that the data can be connected to instructional resources, that is also very important. So, if you discover that students’ math skills are low in rational number equivalents, for example, it’s easy to connect to resources that can help teachers deal with that very specific challenge.

Do you think most instructional materials are sufficiently connected to the particular data instruments that are being used?

In some systems it’s easy to link the data—the analysis of student performance—to instructional resources. It’s not always clear that those systems have done their homework and that there is any reason to believe those resources will be effective, but that’s another point. The point I’d like to make is: Assessment is a process—you assess, you analyze the data to tell you strengths and weaknesses (depths of understanding, misconceptions, and the like), and then you take action to remediate or close the gap between where kids are and where they need to be. In general, we have tests that are administered and we have systems that can analyze the data. But we often do not have anything in place to help teachers take that next step. Good systems are connected to those next-step actions that could be taken to close the gap.

Do you think the Common Core State Standards and the common assessments that are being developed will dramatically alter how data are being use at the classroom level?

I think they have the potential to alter, dramatically, how data are used at the classroom level, but only if we’re serious that the Common Core State Standards will be fewer and clearer. By fewer I mean that it’s actually possible to prepare kids to be proficient—at a given grade level—on all the standards that are laid out for that grade level. And by clearer I mean that teachers actually understand what each standard means, in terms of what students need to know and be able to do. To the extent that the Common Core standards provide a clear target and to the extent that the common assessment consortia create assessments that are well aligned with those targets and with significant learning goals, then I think they can do a lot toward improving the use of assessment data at the classroom level.

In your view, how does data use align with other school reform efforts?

I think, for schools and districts, the orchestration of the whole is the most important thing. In educational reform we have a history of trying things in isolation. For example, considering data use independently from the quality of the curriculum and from the available assessments. Currently, we have all sorts of technology and sophisticated data-management systems and software that enable us to analyze data up, down, and backwards. But not all schools have access to the expertise they need to do what I would consider effective analyses of that data—analyses that have some validity. There is a lot of expertise that needs to be in the system in order to have good data use come to fruition, whether that system is a classroom, a school, a district, or a state. There are leadership skills, there are measurement skills, and there is the ability to differentiate between the data we ought to be paying attention to versus what is garbage in/garbage out. It takes leadership. And as with other reform efforts, you have to see it in context. You can’t just have one piece of the puzzle. You have to have all the pieces.

Do you think we’re heading in that direction?

I certainly think that’s the intent. No one has a crystal ball. The assessment consortia—both of them—have laid out very ambitious plans. For example, including end-of-year assessments and during-the-year, performance-oriented assessments that should do a better job of aligning with what’s important for kids to know and be able to do. There may end up being variations in how states combine their assessments, but [the work of the assessment consortia] should still provide more-or-less common, end-of-year targets that can be linked to curriculum and instruction and assessment.

Currently, there is a lack of research that could connect these pieces. Do you think we’re sophisticated enough, in terms of our research base, that we can inform this process effectively?

I think the lack of a significant research base should not stop us, in the sense that we have to move forward, not back. I think the very data systems and the sophisticated technology we have for both specifying tests and tracking student performance on individual items and being able to potentially tag items to both content and cognitive demand at various levels of detail—we can learn as we go. And we can use the data, as it were, that we’re collecting to refine what we’re doing. Not only to refine the assessments, but to refine our expectations for student learning. We both know that learning progressions are a very hot topic in today’s world—formative assessment and the like. That’s an area where the research base is not very strong, and yet it seems to be a critical foundation for good formative practice. So, at the moment we essentially have hypotheses about what the learning progressions are or what we think they are. If we use those hypothesized progressions to generate assessments, and if we’re able to aggregate data across a whole state or across districts or a large number of students and different instructional approaches, then we can potentially start to answer: What did we get right about the learning progressions and what did we get wrong? Similarly, what did we get right about the standards and what did we get wrong? Are there some that stand out as absolutely essential to future success? Are there others that maybe don’t matter as much as we initially thought they would?

Is there a danger of being hyper-rational in this? The danger of seeing kids as a collection of data—as numbers—rather than as individuals?

I think good teachers want to use every piece of evidence at their disposal to make the best decisions for kids, and those are decisions for kids. And so, that would be counter to that argument. At the same time, we know that the pressure of test scores—for example as part of No Child Left Behind—have led some schools and some teachers and some principals to focus on the data and not on the kids. To me, when we hear anecdotes about “bubble kids,” that’s exactly what they’re doing. That’s about teaching and learning based on what will make us look best on an end-of-year report and not about what’s best for kids or about identifying who are the kids who are in the most need.

What other issues do you think are important for improving how data are used?

I think the orchestration of the whole is the most important thing. I think in educational reform we have a history of trying things in isolation. For example, considering data use independently from the quality of the curriculum and the available assessments. Or, considering data use independent of teachers’ capacity to use the data or teachers’ content pedagogical knowledge to be able to really inform their teaching. Or, considering data use independent of a school culture that is really focused on student learning and where there is sufficient trust and collegiality that teachers can work together and support one another in making forward progress. We have all sorts of technology and sophisticated data-management systems and software that enable us to fiddle with data every which way. Not all schools have access to the expertise they need to do what I would call the right kind of analyses of those data—analyses that have some validity. My favorite example is: I had the occasion to be on a discussion panel in which a paper was presented about schools’ data use and teachers were using pre- and posttests to look at the effectiveness of their instruction. That sounds like a good thing, on one hand, and certainly it’s a good thing that they’re focused on student learning, but the pretest and posttest were not composed of the same items—they were not parallel tests. As a result, they were subtracting apples from oranges to get an indicator of how effective their instruction was. That was happening because they didn’t have sufficient expertise within that particular school system to say, “Hey, wait! If we’re going to look for gains in learning we can’t subtract apples from oranges! We either have to give the same test twice, which has one set of problems, or we need parallel measures, which has another set of challenges.” That’s just one example. There is a lot of expertise that needs to be in the system in order to have good data use come to fruition, whether that system is a school, a classroom, a district, or a state. There are leadership skills and measurement skills and the ability to differentiate between the data we ought to be paying attention to versus what is garbage in/garbage out. One of my pet peeves is a school or teacher that agonizes about how students are responding on single items. They think they can infer from a student’s response to a single item what that student understands or doesn’t understand. As opposed to looking for patterns or saying, “Oh, that’s interesting. How does that compare to what we’re seeing in classrooms?” Looking at a single item is putting too much trust in data that are not, in fact, reliable.

That’s one way of saying: It takes leadership. It takes measurement knowledge. It takes technological knowledge, data analysis knowledge. It certainly takes content pedagogical knowledge and curriculum knowledge. You can’t just have one piece of the puzzle. You have to have all the pieces.

And you have to have the resources—particularly in terms of time—to be able to analyze the data effectively. It certainly is not at all simple. I think that, to the extent that we can get a message out about districts and schools that are doing it well, and make those part of the culture of teacher preparation, then we can begin to move this forward more easily.

Yes, I definitely agree with that. And I think you start, in addition, with not trying to do everything at once. For example: You’re a school. You need to improve student learning. Start with one priority. Or at least give the lion’s share of the attention to one priority. Build commitment around that. Take it down to, “What does that look like in classroom practice? How do we know how we’re doing? How do we know how we’re doing day-by-day? What are teachers going to be looking for?” Use that one priority to build capacity, as you’re helping a school attack an important problem.

For those of us who are kind of standing on the outside, providing technical assistance, figuring out what that one priority should be and helping districts stick to it is tough, because there are so many issues they want to address all at once.

Yes. But in trying to address everything all at once I think they often end up doing nothing. They need to focus on one main goal to start with and build from there.