How can we measure student understandings in science?

In this paper, I explore what the literature has to say about measuring or assessing student understandings, particularly understandings in science. I begin by defining the construct "understanding" and give examples of different understanding frameworks and research that may be applied to that framework. I then provide a brief overview of assessment before exploring a specific assessment approach that shows particular promise in probing student understandings in science classrooms.

Understandings can be demonstrated because understanding involves action more than possession or the accumulation of cued knowledge (Perkins, 1991; Wiggins, 1993). Understanding implies being able to do something effective, transformative, or novel with a problem or complex situation (Wiggins, 1989). Even our vocabulary of understanding implies action, to understand something is to grasp it, penetrate it, comprehend or apprehend it, to see the point of it (Webster's, 1985) The word itself, meaning to stand under something, implies an inside view or another perspective to the knowledge in question.

When people show certain actions, then we see evidence that they understand something. Understanding goes beyond knowing or retrieving information through involving readiness for a wider range of characteristic performances (Perkins, 1991). For example, suppose that a learner can explain a concept in their own words (not just reciting a canned definition), can exemplify its use in fresh contexts, can make analogies to novel situations, can generalize the law, recognizing other laws or principles with the same form, etc. When learners go conspicuously beyond the information given (in reasonable ways a la Bruner, 1973), then we recognize that they understand.

Figure 1 is an excerpt from a conversation between two students as they begin to build a computer model that represents their understandings of a local creek. In this example, they take a novel situation and apply it to their study site, making hypotheses on the possible effects a large culvert might have on the creek. In this short excerpt, we can observe the application of several concepts and relationships, the two mains ones being: 1). sunlight-algae-photosynthesis-oxygen production and indicator species (Taxa 1's and 3's); and 2). erosion-sedimentation-changing habitat-and community structure (more Taxa 3's). We can also identify less robust understandings, for example, Kreg's statement that Taxa 3's "like silty habitats." In this case, a more complete understanding would be represented by an application of the concept of "adaptation" instead of "liking." Juliann's earlier statement that Taxa 3's "can survive" in low dissolved oxygen environments shows a more developed understanding of adaptation than Kreg's statement about "liking."

Figure 1: An example of student understandings in a ninth grade science classroom.

---

Kreg: Juliann, did you see this article in last night's paper... the one about Allen Creek? This apartment complex wants to bury almost 300 meters of it in a huge culvert where it runs through their property and have asked the city for permission. Maybe we can use this stuff as the basis for our science project.

Juliann What do you mean, I thought we were suppose to create a computer model of Traver Creek?

Kreg: But there are several apartment complexes on Traver Creek above our study site. What do you think would happen if one of them decided to do the same thing to our creek?

Juliann: Oh, I see... we could model some of the changes that might occur in that case... like... like... I know! The sun wouldn't reach the creek inside the culvert and that would affect the photosynthesis rate... it would make it go down... because the algae need the sun to carry out photosynthesis. And that means that maybe the DO [dissolved oxygen level] would go down because algae produce oxygen when they do photosynthesis. And that might affect the kinds of benthics [macroinvertebrates] that can live in the creek. We might find fewer Taxa 1's [pollution intolerant organisms]and more Taxa 3's [pollution tolerant organisms] because Taxa 3's can survive in areas with low DO.

Kreg: Well I guess so... but I was thinking more about when they were building the culvert... because then they bring in all this big equipment... backhoes and bulldozers and stuff. And they would have to do a lot of digging and stuff to make room for the culvert. And that might cause a lot of erosion...

Juliann: Yeah... and the erosion might cause sedimentation further down stream...

Kreg: Uh-huh... and that will affect the benthics habitats, you know, like the rocks and stuff will get covered up by all this erosion stuff. So we might get more Taxa 3's because they like silty habitats.

Juliann: I think this is a good idea... let's do it!

Understanding science requires that an individual integrate a complex structure of many types of knowledge, including the ideas of science, relationships between ideas, reasons for these relationships, ways to use the ideas to explain and predict other natural phenomena, and ways to apply them to many events (NRC, 1996). Therefore, at one level, understanding is a function of the integration and structuring of accumulated knowledge (Figure 2). Understanding in science also encompasses the ability to use scientific knowledge, and it entails the ability to distinguish between what is, and what is not, a scientific idea. Developing understanding presupposes that students are actively engaged with the ideas of science and have many experiences with the natural world where they can develop and apply their understandings (NRC, 1996).

Figure 2. Understanding is partially a function of knowledge accretion and knowledge integration. Theoretically, it is possible to accumulate a large number of discrete bits of knowledge without creating relationships between those pieces of knowledge (cells a and c). It is also possible to have a limited amount of knowledge but each piece of knowledge is well integrated or linked to other knowledge, perhaps in multiple ways (cell b). Rich, robust understandings are achieved as knowledge becomes both structured and integrated (cell d).

The development of knowledge and understandings is now generally considered to be a process in which individuals receive environmental stimuli and construct meaning and understanding from it by connecting the new information with what they already know. Early studies in knowledge construction include that of Piaget, but grew beyond his work as researchers became interested in the content-specific ideas that student hold, and the relationship of those ideas to their subsequent knowledge construction. Based on cognitive and constructivist theories of learning, it is believed that understandings are built as learners construct their own meanings for the knowledge that they acquire (White & Gunstone, 1992). New construction of new understandings might come from: 1). reflecting on knowledge such that the learner is able to perceive new links or deduces new propositions or creates new images; 2) incidental learning where a learner forms a new episode from a situation that was not deliberately designed to promote learning, and sees that that episode illuminates some knowledge; or 3) meanings that are constructed under the guidance of a teacher (White & Gunstone, 1992).

Posner, et al. (1982), in their model of conceptual change, emphasize certain features of the learner's "conceptual ecology" or understandings that influence learners movements from naive concepts to more scientific understandings of phenomena. For instance, epistemological commitments about the nature of evidence and the importance of parsimony in a theory and metaphysical beliefs such as faith in the orderliness of nature influence the willingness of a learner to give up their naive conceptions in favor of more scientific views.

Schoenfeld (1985) discusses four factors that figure in mathematical understanding and problem solving: resources (knowledge base of facts and concepts), heuristics, control (metacognitive monitoring and control of problem solving) and belief systems (broad beliefs about the nature of mathematics and mathematical inquiry). These are similar to the four kinds of knowledge important in cognitive apprenticeships as emphasized by Brown, Collins, & Duguid (1989): domain knowledge (facts and concepts), heuristic strategies, monitoring strategies, and learning strategies. Brown, et al., (1989) claim that people who use strategies or tools within an apprenticeship actively build an increasingly rich understanding of the world in which they use the tools and of the tools themselves. The understanding, both of the world and of the tool, continually changes as a result of their interaction.

White & Gunstone(1992) apply their concept of "understanding" to a range of targets: understanding of concepts, whole disciplines, single elements of knowledge, extensive communications, situations, and people. Perkins & Simmons (1988) characterize deep understanding as involving four interlocked levels of knowledge: the content frame, the problem-solving frame, the epistemic frame, and the inquiry frame. In a revised version of his framework Perkins, et al., (1995) explicates just three kinds of knowledge necessary for understanding: content knowledge, problem-solving knowledge and epistemic knowledge.

Obviously, there is considerable overlap among some of the constructs in these different frameworks. For instance, all of them include a component of concepts or facts. Schoenfeld (1985), Brown, et al., (1989), and Perkins & Simmons (1988) include non-domain specific strategies or metacognitive elements. The Schoenfeld (1985), Brown, et al. (1989), White & Gunstone (1992), and Perkins & Simmons (1988) frameworks all include epistemological elements. For purposes of science education, it is probably most useful to use some of these areas of overlap, collapse some of the others and to consider three main dimensions of understanding: concepts/content, discipline/epistemic, and problem solving/inquiry. In the following sections, I examine each of these frames more closely.

In Perkins' & Simmons' (1988) scheme, the content frame of understanding contains the facts, definitions, and algorithms associated with the "content" of a subject matter, as well as, and even more importantly, the mapping schemes that associate concepts with referents, and the content-oriented metacognitive knowledge, such as strategies for monitoring the execution of an algorithm, or for memorization and recall. The earlier exchange between Kreg and Juliann (figure 1) is an example of understandings demonstrated within the content frame.

According to White & Gunstone (1992) a person's understanding of a concept is the set of propositions (facts, opinions, or beliefs), strings (fixed or unvarying forms - like speeches), images (mental representation of sensory perceptions), episodes (memories of events), intellectual skills (capacities to carry out classes of tasks, memories of procedures) and motor skills (capacity to perform classes of physical tasks) that the person associates with the label. Spoehr (1994) refers to this set or cluster of knowledge representation as a conceptual neighborhood and expands on the structural nature with the elements in a neighborhood most often organized in a roughly hierarchical way ( figure 2 - cell d), and neighborhoods combined hierarchically to form even larger neighborhoods.

In sum, an understanding of a concept is a function of the number of elements of knowledge a learner possesses about the concept and the organization of that knowledge into a conceptual neighborhood. Conceptual understanding can be described as the richness of interconnections and relationships made between concepts and the structure which organizes those concepts (Novak & Gowin, 1984). Expert conceptual neighborhoods are likely to be organized using deep, abstract concepts; that is, more sophisticated knowledge representations tend to abandon neighborhoods built out of exemplars and information that share surface characteristics, and move in favor of structures based on deeper, abstract principles.(Spoehr, 1994, p 80). Implicit in this definition is the idea that understanding is a dynamic rather than a static state, for new knowledge can be added to the set , new links can be formed between things already known, and the knowledge set can be restructured based on more abstract principles.

Associated with the content frame are characteristic performances, including recall of facts and correct description of instances using the vocabulary of the domain in question - performances that are familiar in a typical classroom environment. Yet the most valid measure of a learner's understanding of a concept involves eliciting the full set of elements the person has in memory about that concept since understanding a concept is not a simple dichotomous state (either you get it or you don't), but a multi-dimensional continuum (White & Gunstone, 1992). A learner can understand a little about something (displaying a few understanding performances) or a lot more about something (displaying many varied understanding performances) (Perkins, 1992). Therefore, characteristic performances that require a learner to reveal more of the integration and structure of their content knowledge, e.g. generalization, contextualization, comparison and contrast, justification, application, exemplification, and explanation (Perkins, 1992) will come closer to establishing their conceptual understandings.

In the excerpted conversation in figure 1 we can find several examples of these characteristic performances of conceptual understanding. Contextualization is probably the most obvious example as both Kreg and Juliann think about what might happen if part of their creek was placed in a culvert. They compare and contrast the state of the creek with and without a culvert. Juliann explains the relationship between sunlight and dissolved oxygen mediated by photosynthesis. Kreg exemplifies by using the new example of construction in the creek bed as a cause for erosion. As they go about creating their model, they will have the opportunity to justify their assertions and test the model to see if it works.

Too often, even when "understanding" is purportedly the target of instruction, there continues to be a focus on teachers teaching and students practicing decomposed and decontextualized skills (Campion, 1991), a focus that researchers need to attend to when trying to measure student understandings. In most such classrooms, students do not get to practice their understandings but instead end up practicing remembering (Perkins, 1992). Students are often not afforded opportunities to learn the critical thinking skills that permeate the cognitive repertories of accomplished learner (Campione, 1991) and that help develop understandings. This situation is compounded by the nature of instruction in the higher grades, where the emphasis is on breadth of coverage. For example, science courses allow students, even encourage them, to proceed without building strong relationships among the various concepts to which they are exposed (Campione, 1991). Students are not required to explore a subject in depth, and as a consequence, it is not easy for them to learn to evaluate new information critically and build the multiple links between concepts that are the hallmark of rich, robust understanding.

In the last 20 years a substantial body of research has been built up that explores and documents students' specific ideas and levels of understandings on topics within the physical, earth and life sciences. A number of these studies, working within the conceptual change paradigm, have focused on designing and carrying out instruction to promote a change in student understandings of concepts because many students were found to have ideas that are inconsistent with accepted scientific views (see reviews such as Confrey, 1990; Driver & Easley, 1978; Driver, Guesne, & Tiberghien, 1985; Gilbert & Watts, 1983). The conceptual change paradigm generally considers concepts as mental structures that correspond in some degree to scientific views about the world (Gilbert & Watts, 1983). While there is some recognition that these structures may change or evolve , the primary focus of conceptual change research has been on overthrowing prior conceptions and replacing them with scientific conceptions, thus suggesting a rather "static" view of concepts. This static viewpoint is counter to the current dynamic perspective of understanding.

In a constructivist paradigm, concepts are viewed much more "dynamically:" A concept is something that is continuously constructed and acted upon, given the experiences of its owner (Gilbert & Watts, 1983). Linn, et al. (1990) propose a progression of scientific understandings, organized in what they call "action knowledge," "intuitive conceptions," and "scientific principles", or, as Wiggins (1993) might describe it, a movement from a crude grasp of the whole to a more sophisticated grasp of the whole. Magnusson, Boyle, & Templin (1994) argue that it is more useful to follow the constructivist perspective of understanding and focus on how we can help students develop understanding regardless of where they begin or their prior understandings. Each mind in a classroom is a different receptor - the patterns of episodes, images, and other elements will differ even if the learners share extensive common experiences. Hence, what they make of the instruction will differ and their understandings will not be the same.

Scientific thinking has been shown to be grounded in the particulars of its domain. Thus, declarative or domain-specific knowledge related to principles, laws, theories, and generalizations of science must be learned, alongside the procedural/generic strategic knowledge of the domain (Duschl & Gitomer, 1991). Archbald & Newmann (1988) claim that to understand scientific theories, we must ultimately consider them as wholes, not as collections of knowledge fragments. White & Gunstone (1992) argue that there is no central core of knowledge which is essential to the understanding of a discipline, what is essential is an understanding of how the discipline works.

White's and Gunstone's argument is echoed by Driver, et al.'s (1994) claim that there are some core commitments associated with scientific practices and knowledge claims. Driver, et. al. (1994) go on to explain that scientific knowledge in many domains consists of formally specified entities and the relationships posited as existing between them. These are constructs that have been invented and imposed on phenomena in attempts to interpret and explain them, often as the result of considerable intellectual struggles.(Driver, et al., 1994).

Recent developments in the epistemology and philosophy of science (e.g. Bleier, 1986; Matthews, 1989) show that there is no absolute line to be drawn between fact and theory, data and interpretation. Instead, what is counted as fact depends on a complex of tools and instruments that have theories built into them and on communally accepted methods for deciding among competing assertions. Thus science, along with history and literature, must be understood as an interpretive domain in which knowledge and skill cannot be detached from the contexts of practice and use. This view of scientific knowledge as socially constructed and validated has important implications for science education. It means that learning science involves being initiated into the ideas and practices of the scientific community and making these ideas and practices meaningful at an individual level (Driver, et al., 1994).

The context in which knowledge is constructed plays an important role in developing an epistemological understanding of a discipline. Science has a culture with certain norms, values, beliefs, expectations, and actions. Science classrooms have their own distinct cultures that may more or less reflect that of the larger enterprise. School science often fails to convey the wonder and excitement of doing science. Instead, it tends to present science as a series of known concepts and ideas, a body of knowledge to be mastered. For example, in an observational study of 11 junior high school science classes, only a very small proportion of tasks required higher-level creative or expressive skills; the most frequent activity involved copying answers from the board or textbook onto worksheets (Mitman, et al., 1987). Students are likely to conclude that science is static rather than active, and that science proceeds in a linear trial-and-add-new-information approach rather than as a series of conjectures which may or may not be supported (Linn, et al., 1990). Viewing science as a "storehouse of knowledge" ignores the process by which such knowledge is generated and doesn't support strong epistemological understandings of the discipline.

For knowledge to be meaningful to students and for students to develop the attitude and manner of inquiry in a discipline, a culture of inquiry into the subject matter must be created (Lave, 1991; Lave & Wenger, 1991). The context is a culture of "doing" science, not receiving scientific truths. Students learn not only a set of cognitive tools (for example, facts and procedures) but, more important, a set of beliefs about the subject and about valuing and caring for other students' contributions to knowledge constructions. These cognitive and social tools, perpetuated in everyday practices and rituals of the culture, define the context in which knowledge and epistemic understandings are constructed.

People's ideas about the nature of knowledge and how knowledge is justified develop through stages in which knowledge is initially perceived in terms of "right/wrong," then as a matter of "mere opinion," and finally as "informed" and supported with reasons (Kitchener, 1983; Perry, 1970). Carey, et al. (1989) have described three levels of epistemological understanding in science that students might exhibit:

Level 1 - no clear distinction between ideas and activities. A scientist 'tries it to see if it works'. the nature of 'it' remains unspecified or ambiguous; 'it' could be an idea, a thing, an invention, or an experiment. The motivation for an activities is the achievement of the activity itself, rather than the construction of ideas. The goal of science is to discover facts and answers about the world and to invent things.

Level 2 - a clear distinction between ideas and experiments. The motivation for experimentation is to test an idea to see if it is right. There is an understanding that the results of an experiment may lead to the abandonment or revision of an idea; however, there is yet no appreciation that the revised idea must now encompass all the data -- the new and the old. The goal of science is understanding natural phenomena -- how things in the world work.

Level 3 -a clear distinction between ideas and experiments. The motivation for experiments is verification or exploration. An appreciation of the relation between the results of an experiment (esp. unexpected results) and the idea being tested. Recognizes the cyclic, cumulative nature of science, and identifies the goal of science as the construction of ever-deeper explanations of the natural world.

The "culture of science" cannot be taken for granted... and students typically have had little chance to assimilate it (Aikenhead, 1982; Perkins & Simmons, 1988). Posner, et al.,(1982) have discussed the difficulties inherent in achieving student conceptual change when the epistemological commitments of the student differ from those of the scientific community. They note that most fields maintain explanatory ideals; specific views concerning what counts as a good explanation in a particular field. In addition, they describe what appears to be a consensus across scientific fields for the character of successful knowledge: elegance, economy, parsimony, and structure.

The research (Carey, et al., 1989; Schauble, Klopfer, & Raghaven, 1991; Strike & Posner, 1992) suggests that a learner's epistemological framework is a factor toward effecting changes in knowledge representation. At the classroom level, this translates into what learners consider to be evidence for or against an emerging scientific explanation. Learners use of exemplars and anomalous data are counted as evidence in examining epistemic understandings.

Learners that have discipline or epistemic understanding evidence this by giving evidence, explaining rationales, and proposing tests of claims (Perkins & Simmons, 1988). By gaining a lot of experience doing science, becoming more sophisticated in conducting investigations, and explaining their findings, learners will accumulate a set of concrete experiences on which they can draw to reflect on the process. Learners that have an epistemic understanding of science can distinguish between scientific explanations and non-scientific explanations based upon criteria (NRC, 1996). They know how scientists go about their work and reach scientific conclusions, and what the limitations of such conclusions are, they are more likely to react thoughtfully to scientific claims and less likely to reject them out of hand or accept them uncritically (AAAS , 1993). They ask over and over, "How do we know that's true?"

Research on students' understanding of the nature of science has been conducted since the Sputnik era. The earlier part of the research investigated students' understanding about scientists and the scientific enterprise and about the general methods and aims of science (Klopfer & Cooley, 1963; Mackey, 1971; Mead & Metraux, 1957; Welch & Pella, 1967). More recent studies have added students' understanding of the notion of "experimentation," the development of students' experimentation skills, students' understanding of the notions of "theory" and "evidence," and their conceptions of the nature of knowledge (Aikenhead, 1987; Carey et al, 1989; Carey & Smith, 1993; Lederman & O'Malley, 1990; Waterman, 1983). The research in this area was reviewed by Lederman (1992).

Lederman (1992) argues that the development of adequate student conceptions of the nature of science has been a perennial objective of science instruction regardless of the currently advocated pedagogical or curricular emphases. Consequently, it has been an area of prolific research characterized by several parallel, but distinct, lines of investigation. Research in the 1960's and 70's used multiple-choice questionnaires. Recent studies using clinical interviews reveal discrepancies between researchers' and students' understanding of the questions and the proposed answers in those questionnaires. This finding raises doubt about the earlier studies' findings because almost none of them used the clinical interview to corroborate the questionnaires (Lederman, 1992).

An epistemological understanding research agenda has important implications for science learning. Because cognitive research has shown that the learning environments that do not engage children's prior understandings of a topic are likely to fail, we must include an understanding of the nature of science as a goal for science education (e.g. AAAS , 1993; NRC 1996) and understand the way children think about science. Furthermore, children's beliefs and or knowledge about what science "is" influence their interest in science (Krapp, et al., 1992 ; Renninger, 1992; Brandes, 1996) which has consequences for the value they place on a specific scientific activity or task.

In addition, an epistemological view of science as a collective effort to more deeply understand the workings of the world may prove more appealing to children than the stereotypic picture of the solitary scientist in the laboratory discovering eternal truths. The impact of classroom activities coupled with the influence of today's media (Soloman, 1993; Ford, 1997) and literature (Evans, 1992) shape students' attitudes toward science and role of scientists in our society. Although many students would say that they have never met a scientist (Rampal, 1992), they are able to describe their own images of scientists' appearance, personalities, and work in many ways (Hill & Wheeler, 1991; Chiang, 1996).

Still another area to be explored is the relation between students' epistemological beliefs and conceptual change in science content. Although many have speculated that students' epistemological understandings interfere with successful learning of science and mathematics, Carey & Smith (1993) found little empirical evidence supporting this. Also, since science is a way of knowing or looking at the world, epistemological understandings will influence and in turn by influenced by understandings of scientific inquiry.

Some strategies for problem solving are specific to a discipline, while others cut across several disciplines or have general utility, regardless of the subject. In science, asking questions, constructing hypothesis, building models, predicting, observing, describing, and analyzing are examples of specific strategies while metacognition strategies are more general and help students to monitor progress and stay on task (Brown, 1978). For example, in reciprocal teaching elementary students learn to routinely employ four metacognitive strategies: questioning, clarifying, summarizing, and predicting, in order to improve reading comprehension (Palincsar & Brown, 1984). Such strategies could also be employed in other domains, including scientific inquiry.

Often, instructional approaches that emphasize science process skills (e.g. simple "hands-on-science" and "verification" labs) fail to convey the motivation for the use of such skills in an ongoing cycle of theory generation and verification in scientific culture (Duschl & Gitomer, 1991). Scientific inquiry is more complex than popularly conceived. It is, for instance, a more subtle and demanding process than the naive idea of "making a great many careful observations and then organizing them." It is far more flexible than the rigid sequence of steps commonly depicted in textbooks as "THE Scientific Method." This formally presented scientific method does not fully reflect the actual "messier" practice of scientists. Science is much more than just "doing experiments," and it is not confined to laboratories. There is more imagination and inventiveness involved in scientific inquiry than many people realize, yet strict logic and empirical evidence also critically important. Individual investigators working alone sometimes make great discoveries, but the steady advancement of science depends on the collective efforts of the enterprise as a whole. Covering up the "messiness" of real science further perpetuates the image of science as a linear trial-and-add-new-information process rather than an enterprise where people engaged in science construct understandings based on the norms of evidence and inquiry that make up the culture of science.

Current models in science education predict that if students participate in scientific investigations that progressively approximate good science, then the understanding of scientific inquiry that they develop will likely be reasonably accurate. Models such as project-based science go on to specify that such inquiry needs to be ground by a driving question that has relevance to the learners' everyday world (Blumenfeld, et al., 1991). Schauble, et al., (1995) claim that the primary objective of school science experimentation should be meaning-making. They define meaning making in science as becoming reflective about one's own ideas concerning how things work, evaluating those ideas in light of evidence, and working over an extended period to construct a more coherent and connected understanding.

Unfortunately, the typical high-school science "laboratory experiment" bears little resemblance to authentic scientific investigations and gives little opportunity for constructing understandings. For example, often in school laboratories the question to be investigated is decided not by the student investigator but by the teacher or the textbook; who also decides what apparatus to use, what data to collect, and how to organize the data; time is not made available for replications or, when things are not working out, for revising the experiment; the results are not presented to other investigators for criticism; and, to further "school" the activity, often the correct answer is known before the experiment is carried out (AAAS, 1993).

Upper elementary- and middle-school students may not understand experimentation as a method of testing ideas, but rather as a method of trying things out or producing a desired outcome (Carey et al., 1989; Schauble et al., 1991; Solomon, 1992). With instruction and careful scaffolding, it is possible for middle-school students to understand that experimentation is guided by particular ideas and questions and that experiments are tests of ideas (Carey et al., 1989; Solomon et al., 1992). Whether it is possible for younger students to achieve this understanding needs further investigation although there are inidications of the possiblities (e.g. Metz, 1995 ).

Novice science learners of all ages may overlook the need to hold all but one variable constant, although elementary students have been shown to understand the notion of fair comparisons, a precursor to the idea of "controlled experiments" (Wollman, 1977a, 1977b; Wollman & Lawson, 1977). Another example of students' lack of understanding-of-inquiry comes with the interpretation of experimental data. When engaged in experimentation, students have difficulty interpreting covariation and noncovariation evidence (Kuhn, et al., 1988). For example, students tend to make a causal inference based on a single concurrence of antecedent and outcome or have difficulty understanding the distinction between a variable having no effect and a variable having an opposite effect. Furthermore, students tend to look for or accept evidence that is consistent with their prior beliefs and either distort or fail to generate evidence that is inconsistent with these beliefs (Schauble, 1990). Learners also find it difficult to distinguish between a theory (an explanation) and the evidence for it, or between description of evidence and interpretation of evidence (Kuhn, 1993).

Evidence of problem solving understanding includes student actions such as breaking problems down into manageable parts, regulating time spent on any one solution path, and seeking alternative paths. Weaknesses in this type of understanding are evidenced by undisciplined trial and error methods, persisting in an unproductive approach or quitting if no obvious path to a solution presents itself and using ritual or formulaic approaches (Perkins & Simmons, 1988).. In the latter case, a template-like response to problem solving shows up when, faced with a somewhat new situation, the learner responds in a stereotyped way. At first glance, the student seems to have a respectable understanding of problem solving; specifically, a high degree of technical problem-solving skill in dealing with textbook problems. Yet further analysis establishes that in fact the student applies knowledge in a somewhat ritualistic fashion, and proves unable to deal with novel situations even when the knowledge base should be more than adequate to the task (Perkins & Simmons, 1988).

Characteristic demonstrations of inquiry understanding occur when learners undertake critical and creative thinking that questions the boundaries of their knowledge. Ideally it should be easy to provoke the naive theorist to adopt a venturesome attitude and explore a variety of formulations in constructing new understandings. However, most exhibit more of a spirit of conviction than a spirit of exploration in their theorizing (Perkins & Simmons, 1988). Weaknesses in this understanding frame are revealed when student fail to recognize and explore variations that contrast with their own naive theories, perhaps because of a mental separation of school knowledge from their everyday knowledge.

The National Science Standards (NRC, 1996) identify important understanding of inquiry and problem solving as the ability to design and carry out an investigation from the framing of the question to the design of the inquiry approach, carrying out the investigation, make sense of their findings, and construct, communicate and defend a scientific argument that answers their question. The standards also include understandings about inquiry conducted by scientists and to some extent overlaps the epistemic framework.

In summary, the rich, deep, robust understanding requires not only a substantial amount of content knowledge but also epistemic and problem solving knowledge and understandings. In science education this means understanding not only the corpus of scientific knowledge, but also how that knowledge came to be accepted and how new scientific knowledge is constructed. That is, students also must understand something about the nature of science and how to carry out scientific investigations. Using the three frames of concepts/content, discipline/epistemic, and problem solving/inquiry understandings allow us to both differentiate between the three dimensions of understanding and also to see how they are related. This can be helpful as we try to measure scientific understandings in learners.

Measurements of understandings should integrate knowledge in two ways. Not only must students be challenged to understand integrated forms of knowledge, they must also be involved in the production, not simply the reproduction, of knowledge, because this requires knowledge integration and thus, understandings. Too often assessments of student understandings ask the student only to show comprehension of unrelated knowledge fragments: definitions of terms; short descriptive identifications of people, things, events; or numerical solutions to problems (Wiggins, 1993). Students demonstrate proficiency by giving short responses where answers bear little relation to one another. In such a case, knowing the correct answer may contribute to a more integrated understanding of the topic, but cannot be considered an indicator of understanding.

In measuring student understanding we need to make better distinctions among what students know, what they can do, and what they are capable of knowing and doing. To assess understanding is to see if knowledge can be thoughtfully adapted. In the next section, I explore the notion of assessment and how it might be applied to measuring student understandings.

The tools and techniques of measurement must be consistent with our theories of learning. Current assumptions of learning in the constructivist framework include: Learning is an active, mindful process (Resnick, 1987) and the important role played by the context in which knowledge is constructed (Brown, et al., 1989; Lave, 1991; Lave & Wenger, 1991). Many current science achievement tests measure "inert" knowledge - discrete isolated bits of knowledge - rather than "active" knowledge or understandings - knowledge that is rich and well-structured (NRC, 1996). Rather than checking whether students have memorized certain items of information, assessment needs to probe for students' understanding, reasoning, and the utilization of knowledge (NRC, 1996, p 82).

Shavelson & Baxter (1992) have identified three implications of designing assessments to be consistent with constructivist theory:

1. socially constructed solutions (cooperative groups), not just individual answers to assessment tasks should be encouraged.

2. assessment should contain concrete, meaningful tasks. These tasks should respond to students' actions, providing feedback as they test ideas about problems solutions.

3. assessments should contain tasks for which there are alternative solutions.

A valid assessment system provides information about the particular tasks on which students succeed or fail, but more important, it also presents tasks that are worthwhile, significant, and meaningful -- in short, authentic (Archbald & Newmann, 1988; Newmann & Wehlage, 1993). A true test of intellectual ability requires the performance of exemplary tasks. Authentic assessments replicate the challenges and standards of performance that typically face writers, business people, scientists, community leaders, designers, or historians (Wiggins, 1989). These include writing essays and reports, conducting individual and group research, designing proposals and mock-ups, assembling portfolios, and so on. The equivalent school measures (e.g. open-ended problems, essays, hands-on science problems, computer simulations of real-world problems, and portfolios of student work) are frequently referred to as "authentic" assessments because they involve the performance of tasks that are valued in their own right (Linn, Baker, & Dunbar, 1991). In contrast, paper-and-pencil, multiple-choice tests derive their value primarily as indicators or correlates of other valued performances (Linn, et al., 1991).

In science education, authentic assessment tasks require students to apply scientific knowledge and reasoning in real world contexts as well as in situations that approximate how scientists do their work. The National Science Standards' Assessment Standard C describes authentic assessments as "assessment tasks that are similar in form to tasks in which students will engage in their lives outside of the classroom or are similar to the activities of scientists" (NRC, 1996, p 83).

Good teachers have historically used alternative assessments such as performance and dynamic assessments, diagrams and drawings, reports, presentations, exhibitions, and portfolios, to monitor the progress of their students. Now these approaches are being extended beyond individual classrooms to provide useful tools for conducting educational research and to pose a challenge to traditional ways of mass testing. Various reviews examine the psychometric issues and issues of scaling from classroom to district, state, and national levels. (e.g. Maeroff, 1991; Mehrens, 1992) which are beyond the scope of this paper. This paper reviews some of the alternatives and evaluates their usefulness in revealing students' understandings in science. In particular, I will examine the usefulness of various learner created artifacts to assess student understandings in terms of the evidence they can provide.

There is nothing particularly new or innovative about having students construct artifacts in the classroom. Students have done extended research or lab write-ups, term papers, and other projects to fulfill class requirements for as long as there has been schooling (Madaus & Tan, 1993). Science fairs and art contests are project type activities with a long history in schools (Fitzpatrick & Morrison, 1971). They are frequently well organized and carefully judged. However, the evaluation of the products is made difficult by the fact that each student is usually doing a quite different project. Not only is it hard to compare the relative merits of the art or science projects in general, but it is difficult to decide what the general bases for evaluation should be (Fitzpatrick & Morrison, 1971).

Constructing artifacts provides learners with an opportunity to develop understanding as well as a context in which they may demonstrate their understanding (Papert, 1991; Perkins, 1986; Perkins, 1991; Perkins & Blythe, 1994). The theoretical rational comes from a synthesis of design theory and constructivist learning theory where the "construction of meaning" is viewed as a core process (Kafai & Resnick, 1996; Papert, 1991). The process of creating an artifact requires learners to engage in many elements of design, for example: formulating questions, gathering data from multiple sources, organizing diverse and contradictory information, and presenting their findings (Lehrer, 1993; Perkins, 1986).

Blumenfeld, et. al. (1991) claim that students' freedom to create artifacts is critical, because it is through the process of generation that students construct their knowledge, the doing and the learning are inextricable. As learners engage in artifact development, they enhance their conceptual understanding as they integrate new information and build connections between concepts. They construct and reconstruct their understandings as they synthesis information and work with ideas, forming them into a coherent structure (Papert, 1993; Perkins, 1986). Constructing new relationships with knowledge is seen as important as forming new representations of knowledge (Kafai & Resnick, 1996).

Students also construct understandings as they design their artifacts if they can do so collaboratively so that during the process students communicate and defend their ideas (Brown, et al., 1989). In addition, because artifacts are tangible, concrete and explicit (e.g., a model, report, videotape, hypermedia document, or computer program) they exist within some kind of social space where they can be shared and critiqued. Artifacts are drawn from student understandings, but these understandings are reshaped, reexamined, and selected for the purpose of communicating with a certain audience(Bos, Krajcik, & Soloway, 1997). This allows others to provide feedback and permits learners to reflect on and extend their emergent knowledge and revise their artifacts (Blumenfeld, et al., 1991).

The degree to which students make connections and draw relationships between concepts within their artifacts provides insight into students' understanding of concepts (Spitulnik, 1995; Spitulnik, et al., in press ; Wisnudel, 1994). The next sections of this paper will consider the types of understandings that might be revealed in such student constructed artifacts as concept maps, creative representations, hypermedia, models and portfolios.

The facilitating effects of concept mapping on learning are thought to arise from the fact that it assists students in understanding concepts and the hierarchical relationships between them (White & Gunstone, 1992). As such, it helps learners to organize and reflect upon their conceptual understanding. Concept mapping is consistent with a constructivist perspective of learning when it is used to examine changes in the content and organization of students' knowledge by emphasizing the process of the construction and the uniqueness of the individual products. Learners who collaboratively construct concept maps may increase their understanding because collaborative concept mapping is one form of getting students to talk science and to practice its language; the students establish means to negotiate the meanings of the concepts under study by using language, diagrams and gestures (Roth & Roychoudhury, 1992).

Concept maps focus specifically on the structure and linking of concepts that the student perceives. They can be used to explore the understanding of a limited aspect of a topic and the relationships between key terms by choosing terms to direct the focus of the probe or by limiting terms to two to three key concepts and asking learners to make multiple links between these terms (White & Gunstone, 1992). Concept maps maybe used to check whether students understand the reasons for an activity, understanding how an activity relates cognitively to the content being learned and understanding why a particular activity was used instead of some alternative. Concept maps may also be used to see whether or not students relate distinct topics and whether they can draw links between two concept maps on different topics (White & Gunstone, 1992).

For teachers and researchers, concept maps provide periodic "windows into the minds" of students as they construct their understanding of science concepts (Pearsall, Skipper, & Mintzes, 1996; Roth, 1992). As summative assessment tools, concept maps provide teachers and researchers with a richer view of students knowledge than is possible using conventional tests (Dana, et al., 1991). By using pre/post concept maps, and/or periodic revisions during the project, teachers and researchers can identify changes in relations that students perceive between concepts (Novak & Gowin, 1984). Students' developing understandings can be further probed by having students compare their maps, analyzing and writing about the changes.

Since concept maps are idiosyncratic representations of domain-specific knowledge, various schemes have been developed for analyzing and coding these artifacts. Many of these schemes use such dimensions as hierarchy (the number of levels represented), relationships between concepts, branching between a concept at one level and two or more concepts at the next hierarchical level and crosslinks, the connections that demonstrate integration or parallelism between two or more relationships (Novak & Gowin, 1984; Roth, 1992; Vargas & Alvarez, 1992; Wallace & Mintzes, 1990). Wallace & Mintzes (1990) also included a measure they called "critical concepts and propositions" to gauge the extent of biologically meaningful knowledge revealed in the concept maps in their study.

Much of the literature on concept mapping has examined the use of concept maps in science learning. Relatively few studies have investigated the use of concepts maps as a research tool for documenting and exploring students understandings (see Wallace & Mintzes, 1990 for a review of this literature). Thus, the issue of validity of concept mapping as a measurement tool has not been fully explored. Wallace & Mintzes(1990) claim that they found concept maps to complement other measurement techniques and are useful for revealing not only what students know but how they organize their knowledge. They also add that, from a practical standpoint, concept mapping is quickly taught to experimental subjects, that it can be "administered" to large groups, and that the product is readily interpreted (Wallace & Mintzes, 1990, p 1050). These are important consideration when attempting to probe understandings of complex, scientific concepts.

In addition to learner created concept maps, some researchers use a concept mapping technique to graphically represent student understandings expressed in interviews (Chi, Feltovich, & Glaser, 1981; Rye & Rubba, 1996), student writing (Fellows, 1994); to organize field observations (e.g. Barden, 1997) or to analyze a curriculum or document (Wandersee & Fisher, 1995). Mapping out understandings may also be a useful technique when analyzing other learner created artifacts such as non-linear hypermedia documents and computer models (e.g. Jackson, et al., in press ). Hypermedia artifacts, and the understandings they represent, will be discussed more fully below.

Relational diagrams are closely related to concept maps in that they are a way of graphical representing knowledge and understandings. In a relational diagram a person draws closed figures (Venn diagrams) to show the pattern of overlap between classes of objects, events, or abstractions. Such diagrams show the meaning that people give to terms that stand for classes of objects (White & Gunstone, 1992). They may also be used to probe understandings of single elements of knowledge.

Goldberg (1992) asserts that since individuals share their knowledge with others or make their knowledge explicit via expression; expression is an integral aspect of knowing. Creative assessment techniques are used by students to represent what they have learned during a particular "unit" in a creative manner. Rather than just recalling unrelated facts, the students show what they have learned in a context that makes sense to them. They might use scrap books, comic books, songs, artistic performances or student produced videos, for example, to demonstrate particular concepts. Dana, et al., (1991) claim that these assessment techniques emphasizes the application of higher-level-thinking skills in a context that many students find highly enjoyable.

Karen Gallas (1991) describes how she uses artistic expression in an elementary classroom and how that has informed her determination of student understandings of science content. "What we understood from our experiences with the arts as subject matter and inspiration was that knowing wasn't just telling something back as we had received it. Knowing meant transformation and change, and a gradual awareness of what we had learned (Gallas, 1991)." In reflecting on student artwork, Gallas (1991) found that the deep involvement in representing the form of an insect expanded the child's basic knowledge of that organism and his or her ability to represent it both in thought and form.

Drawings are often used by researchers to probe children's understandings and feelings. The purpose and utility of drawings as a probe of understanding flows from their extreme positions on the word-diagram and closed-open dimensions of assessment (White & Gunstone, 1992). They allow the student to reveal qualities of understanding that are hidden from other procedures. They can also draw out an affective component of understanding. By tapping a holistic understanding, they allow expression of attitudes or feelings as well as cognition (White & Gunstone, 1992).

One area in which drawings have been extensively used to probe students'epistemological understanding in the sciences have been in the Draw-A-Scientist-Test (DAST). The DAST was originally developed by David W. Chambers (1983) using the research of the Draw-a-Man and Draw-A-Person tests (Goodenough, 1926; Harris, 1963; Goodenow, 1977). Chambers' purpose was to learn the person's image of a scientist and to determine the age in which distinctive images of scientists first develop. For his test, Chambers used the simple prompt, "Draw a scientist." Several studies have incorporated the scoring key of DAST, as developed by Chambers, to quantify students' images of scientists (Mason, et al. 1991, Symington &Spurling, 1990). Repeatedly the results represent a stereotypical Caucasian male wearing a lab coat, usually balding and bearded with eyeglasses. Other researchers have used the DAST to gauge various factors in students, including career goals (Warren, 1990), perceptions of scientists at the elementary through high school level (Schibeci & Sorensen, 1983; Flick, 1990) and perception of technology (Hill & Wheeler, 1991). Using drawings such as these are one method of probing students' epistemological understandings.
 
 

Hypermedia artifact designing is thought to promote meta-representational thinking and meta-relational thinking which evolves as a product of the active participation of the student (Carver, Lehrer, Connell, & Erickson, 1992). Hypermedia artifacts appear to facilitate the construction of multiple representations of the understanding of science concepts (Spitulnik, 1995; Spitulnik, et al., in press). In her analysis, (Spitulnik, 1995) found that during the process of incorporating and integrating multiple representations, students were challenged in their own conceptions and often presented non-linked or conflicting representations during early versions of the artifact. She asserts that as students made connections between the pictorial particulate representations, the graphical representations, and textual explanations, they constructed understanding of both the phenomena (phase changes) and the theory which explains the phenomena (kinetic molecular theory) (Spitulnik, 1995).

World wide web publishing is a new kind of technological artifact. Bos, et al. (1997) describe three types of authentic web artifacts constructed by students: publishing of students scientific data and research, multimedia resources on specialized topics, and "value-added" reviews of existing resources. The student constructed WWW Resource reviews described by Bos (1997), although not especially rich in conceptual understanding, may shed insights on students science epistemological understandings. The students engaged in producing these WWW artifacts were asked to review web resources for scientific claims, evidence for claims (empirical data), citations (references to published studies), and source credibility (author or group supporting the web resource)(Bos, 1997).

The construction and manipulation of models of different kinds are fundamental to scientific inquiry and can be invaluable in scientific pedagogy. Assessment of these activities/products is important in order to determine their overall educational value, identify general misconceptions or gaps in both conceptual and epistemological understandings at the classroom level, and assess students' individual achievement levels (Haertel, 1991). The various reform efforts call for an increase emphasis on modeling in science learning. For example, Benchmarks (AAAS, 1993) calls for considerable emphasis to be placed on mathematical modeling in the upper grades because it epitomizes the nature and power of models and provides a context for integrating knowledge from many different domains. The main goal, according to the reform efforts, should be getting students to learn how to create and use models in many different contexts (AAAS , 1993 - emphasis added).

Student use of models has been the primary area of inquiry into student scientific understandings (e.g., Smith et al., 1987; White, 1990; Brand, Pulver, & Rader, 1997) until the recent advent of age-appropriate modeling tools (e.g. Cocoa, Stella, Model-It, StarLogo). The development of these tools came about in part because model development teams learned a great deal from the activities of researching, implementing, evaluating and rejecting numerous models. These experiences led to the belief that students too, would learn more or understand better if they researched and developed their own computer models (Riley, 1990).

Hestenes (1992) proposes that students will learn the principles of a theory by using, manipulating, and building models. He also suggests that by building models, students will gain a better understanding of the purposes and limits of models, as well as the structure of scientific knowledge. The construction of dynamic models encourages analyzing, synthesizing, reasoning and explaining (Spitulnik, et al., in press). These processes help students develop both problem-solving and epistemological understandings as they create and revise their models, and conceptual understandings as they attempt to explicate relationships between the different parts of their models. Building a dynamic model is a concrete way to help students construct mental connections between real-world concepts by providing an environment in which they can formulate and test their mental model of a phenomenon (Spitulnik, et al., in press).

Most models and modeling tools being developed emphasis qualitative modeling for two reasons. Because the prior knowledge and conceptual models students bring to their science instruction are themselves usually qualitative, qualitative reasoning is closely connected to that prior knowledge. Moreover, problem-solving studies have shown that qualitative reasoning is not engaged if students move too quickly into memorizing and applying formal laws.

Grosslight, Unger, Jay, & Smith (1991) describe three levels of understanding about models, reflecting different epistemological views about models and their use in science.

Level 1 - Models are simple copies of reality. Models are useful because they can provide copies of actual objects or actions. If students acknowledge that some aspects or parts of the real thing can be left out of the model, they do not express a reason for doing so beyond the fact that one might want or need to.

Level 2 - There is a specific, explicit purpose that mediates the way the model is constructed. The model no longer must exactly correspond with the real-world object being modeled. The main focus is still on the model and the reality modeled, not the ideas portrayed per se. Tests of the model are tests of the workability of the model, not tests of underlying ideas.

Level 3 - The model is constructed in the service of developing and testing ideas rather than as a copy of reality. The modeler takes an active role in constructing the model, evaluating which of several designs could be used to serve the model's purpose. Models can be manipulated and subjected to tests in the service of informing ideas. Thus, they provide information within a cyclic constructive process.

Many high-school students think models help them understand nature but also believe that models do not duplicate reality. This is chiefly because they think that models have always changed and not because they are aware of the metaphorical status of scientific models (Aikenhead, 1987; Ryan & Aikenhead, 1992).

Portfolios with an emphasis on reflective inclusion fit in with the National Science Standards call for developing self-directed learners (NRC, 1996, p 88). The standards state that students need the opportunity to evaluate and reflect on their own scientific understanding and ability. The ability to self-assess understanding is an essential tool for self-directed learning. Through self-reflection, students clarify ideas of what they are supposed to learn. They begin to internalize the expectation that they can learn science (NRC, 1996). The Standards claim that students demonstrate this kind of understanding when they can:

- Select a piece of their own work to provide evidence of understanding of a scientific concept, principle or law - or their ability to conduct scientific inquiry.

- Explain orally, in writing, or through illustration how a work sample provides evidence of understanding

- Critique a sample of their own work using the teacher's standards and criteria for quality

- critique the work of other students in constructive ways. (NRC & National Research Council, 1996) p 89.

Some artifacts, such as concept maps, may be limited to assessing understanding within a single dimension. Others, such as the construction and testing of dynamic models, may reveal insights into student understandings within all three dimensions: content/conceptual, discipline/epistemic and problem-solving/inquiry. By using a range of artifacts, it is possible to probe student understandings within each of the understanding dimensions.

Bibliography

AAAS, The American Association for the Advancement of Science (1989). Project 2061: Science for All Americans. Washington, D.C.:

AAAS, The American Association for the Advancement of Science (1993). Benchmarks for Science Literacy. New York: Oxford University Press.

Adams, D. M., & Hamm, M. E. (1992). Portfolio assessment and social studies: Collecting, selecting, and reflecting on what is significant. Social Education, 56(Feb.), 103-105.

Aikenhead, G. S. (1982). Science: A Way of Knowing. In V. N. Wanchoo (Ed.), World Views on Science Education (pp. 206-215). New Delhi, India: Oxford & IBH Publishing Co.

Aikenhead, G. S. (1987). High school graduates' beliefs about science-technology-society III. Characteristics and limitations of scientific knowledge. Science Education, 71, 458-487.

Archbald, D. A., & Newmann, F. M. (1988). Beyond Standardized Testing: Assessing Authentic Academic Achievement in the Secondary School (1 ed.). Reston: National Association of Secondary School Principals.

Barden, L. M. (1997). Strategies exhibited by high school biology students during laboratories. In A paper presented at the Annual Meeting of the National Association for Research in Science Teaching, . Oakbrook, IL:

Blank, H. (1993). Alternative Assessment and Technology (ERIC Digest No. ED365312). ERIC Document Reproduction Service.

Bleier, R. (Ed.). (1986). Feminist Approaches to Science. New York, NY: Pergamon Press.

Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning. Educational Psychologist, 26(3/4), 369-398.

Bos, N. (1997). Student publishing of value-added contributions to a digital library. In A paper presented at the annual meeting of the American Educational Research Association (AERA), . Chicago, IL: In E. Soloway (Symposium Chair) Using online digital resources to support sustained inquiry learning in K-12 science.

Bos, N., Krajcik, J., & Soloway, E. (1997). Student publishing in a WWW digital library: Goals and instructional support. In P. Bell (Chair) Artifact-building in computer learning environments: supporting students' scientific inquiry. (Ed.), A paper presented at the annual meeting of the American Educational Research Association (AERA), . Chicago, IL:

Brand, C., Pulver, P., & Rader, C. (1997). Using visual models in an elementary classroom. In A paper presented at the Annual Meeting of the National Association for Research in Science Teaching, . Oakbrook, IL:

Brandes, A. A. (1996). Elementary School Children's Images of Science. In Y. Kafai & M. Resnick (Eds.), Constructionism in Practice: Designing, thinking and learning in a digital world (pp. 37-69). Mahwah, NJ: Lawrence Erlbaum Assoc.

Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in Instructional Psychology Hillsdale, NJ: Erlbaum.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated Cognition and the Culture of learning. Educational Researcher, 18(1 (Jan-Feb)), 32-42.

Bruner, J. S. (1973). Going beyond the information given. In J. M. Anglin (Ed.), Beyond the Information Given: Studies in the psychology of knowing (pp. 218-238). New York, NY: W.W. Norton & Company, Inc.

Campione, J. C. (1991). Dynamic Assessment: Potential for change as a metric of individual readiness. In G. Kulm & S. M. Malcolm (Eds.), Science Assessment in the Service of Reform (pp. 301-312). Washington, D.C.: American Association for the Advancement of Science.

Carey, S., & Smith, C. (1993). On understanding the nature of scientific knowledge. Educational Psychologist, 28(3), 235-251.

Carey, S., Evans, R., Honda, M., Jay, E., & Unger, C. (1989). 'An experiment is when you try it and see if it works': A study of grade 7 students' understanding of the construction of scientific knowledge. International Journal of Science Education, 11(Special Issue), 514-529.

Carver, S. M., Lehrer, R., Connell, T., & Erickson, J. (1992). Learning by Hypermedia Design: Issues of assessment and implementation. Educational Psychologist, 27(3), 385-404.

Chambers, D. W. (1983). Stereotypic Images of the Scientist: The Draw-A-Scientist Test. Science Education, 67(2), 255-265.

Champagne, A. B., & Newell, S. T. (1992). Directions for Research and Development: Alternative methods of assessing scientific literacy. Journal of Research in Science Teaching, 29(8), 841-860.

Chi, M., Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.

Chiang, C.-L., & Guo, C.-J. (1996). A Study of the Images of the Scientist for Elementary School Children. In Thomas R. Koballa , Jr. (Ed.), A Paper presented at National Association for Research in Science Teaching Annual Meeting, . St. Louis, MO: University of Georgia.

Collins, A. (1991). Portfolios for Assessing Student Learning in Science: A new name for a familiar idea? In G. Kulm & S. M. Malcolm (Eds.), Science Assessment in the Service of Reform (pp. 291-300). Washington, D.C.: American Association for the Advancement of Science.

Confrey, J. (1990). A review of the research on student conceptions in mathematics, science and programming. In C. Cazden (Ed.), Review of Research in Education (pp. 3-56). Washington, D.C.: American Educational Research Association.

Dana, T. M., Lorsbach, A. W., Hook, K., & Briscoe, C. (1991). Students showing what they know: A look at alternative assessments. In G. Kulm & S. M. Malcolm (Eds.), Science Assessment in the Service of Reform (pp. 331-337). Washington, D.C.: American Association for the Advancement of Science.

Driver, R., & Easley, J. (1978). Pupils and Paradigms: A review of literature related to concept development in adolescent science students. Studies in Science Education, 5, 61-84.

Driver, R., Asoko, H., Leach, J., Mortimer, E., & Scott, P. (1994). Successful enculturation: Constructing scientific knowledge in the classroom. Educational Researcher, 23(7), 5-12.

Driver, R., Guesne, E., & Tiberghien, A. (Eds.). (1985). Children's ideas in science. UK: Milton Keynes, UK: Open University Press.

Duschl, R. A., & Gitomer, D. H. (1991). Epistemological Perspectives on Conceptual Change: Implications for Educational Practice. Journal of Research in Science Teaching, 28(9), 839-858.

Evans, A. (1992). A look at the scientist as portrayed in children's literature. Science and Children, 29(3), 35-37.

Fellows, N. J. (1994). A Window into Thinking: Using student writing to understand conceptual change in science learning. Journal of Research in Science Teaching, 31(9), 981-1001.

Fitzpatrick, R., & Morrison, E. J. (1971). Performance and Product Evaluation. In R. L. Thorndike (Ed.), Educational Measurement (pp. 237-270). Washington, D.C.: American Council on Education.

Flick, L. (1990). Scientist in Residence Program Improving Children's Image of Science and Scientists. School Science and Mathematics, 90, 204-214.

Ford, D. J. (1997). Science programs on television: Issues of authenticity and inclusiveness. In A paper presented at the Annual Meeting of the National Association for Research in Science Teaching (NARST), . Oakbrook, IL:

Gallas, K. (1991). Arts as Epistemology: Enabling children to know what they know. Harvard Educational Review, 61(1), 40-50.

Gilbert, J. K., & Watts, D. M. (1983). Concepts, Misconceptions and Alternative Conceptions: Changing perspectives in science education. Studies in Science Education, 10, 61-98.

Gilbert, S. W. (1991). Model building and a definition of science. Journal of Research in Science Teaching, 28(1), 73-79.

Glaser, R., & Silver, E. (1993). Assessment, Testing, and Instruction: Retrospect and Prospect. In L. Darling-Hammond (Ed.), Review of Research in Education Washington, D.C.: American Educational Research Association.

Goldberg, M. R. (1992). Expressing and Assessing Understanding Through the Arts. Phi Delta Kappan, 73(Apr), 619-623.

Goodenough, F. L. (1926). Measurement of intelligence by drawings. New York, NY: Harcourt, Brace, and World.

Goodenow, J. (1977). Children's Drawings. London, UK: Open Books.

Grosslight, L., Unger, C., Jay, E., & Smith, C. L. (1991). Understanding models and their use in science: Conceptions of middle and high school students and experts. Journal of Research in Science Teaching, 28(9), 799-822.

Haertel, E. H. (1991). Form and function in assessing science education. In G. Kulm & S. M. Malcolm (Eds.), Science Assessment in the Service of Reform (pp. 133-245). Washington, D.C.: American Association for the Advancement of Science.

Harris, D. B. (1963). Children's drawings as measures of intellectual maturity. New York, NY: Harcourt, Brace, and World.

Hestenes, D. (1992). Modeling games in the Newtonian world. American Journal of Physics, 60(8), 732-748.

Hill, D., & Wheeler, A. (1991). Towards a clearer understanding of students' ideas about science and technology: An exploratory study. Research in Science & Technology Education, 9(2), 125-137.

Horwitz, P. (1996). Linking Models to Data: Hypermodels for Science Education. http://copernicus.bbn.com/genscope/Hypermodel-Paper.html.

Jackson, S. L., Stratford, S. J., Krajcik, J. S., & Soloway, E. (in press (1996)). Model-It: A Case Study of Learner-Centered Design Software for Supporting Model Building. .

Jonassen, D. H. (1992). Evaluating Constructivist Learning. Chapter 12. In D. H. Jonassen & T. M. Duffy (Eds.), Constructivism and the Technology of Instruction: A Conversation (pp. 137-147). Hillsdale, NJ: Lawrence Erlbaum.

Kafai, Y. B. (1996). Learning Design by Making Games: Children's development of design strategies in the creation of a complex computational artifact. In Y. Kafai & M. Resnick (Eds.), Constructionism in Practice: Designing, thinking and learning in a digital world (pp. 71-96). Mahwah, NJ: Lawrence Erlbaum Assoc.

Kafai, Y., & Resnick, M. (Eds.). (1996). Constructionism in Practice: Designing, thinking and learning in a digital world. Mahwah, NJ: Lawrence Erlbaum Assoc.

Kitchener, K. (1983, Fall). Educational goals and reflective thinking. The Educational Forum, 75-95.

Kitchener, K., & King, P. (1981). Reflective judgment: Concepts of justification and their relationship to age and education. Journal of Applied Developmental Psychology, 2, 89-116.

Klopfer, L., & Cooley, W. (1963). Effectiveness of the history of science cases for high schools in the development of student understanding of science and scientists. Journal of Research in Science Teaching, 1, 35-47.

Krapp, A., Hidi, S., & Renninger, K. A. (1992). Interest, learning and development. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.), The Role of Interest in Learning and Development Hillsdale, NJ: Lawrence Erlbaum, Assoc.

Kuhn, D. (1993). Science as Argument: Implications for Teaching and Learning Scientific Thinking. Science Education, 77(3), 319-337.

Kuhn, D., Amsel, E., & O'Loughlin, M. (1988). The development of scientific thinking skills. San Diego, CA: Academic Press.

Lave, J. (1991). Situating learning in communities of practice. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition (pp. 63-82). Washington, D.C.: American Psychological Association.

Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate peripheral participation. New York, NY: Cambridge University Press.

Lederman, N. G. (1992). Students' and Teachers' Conceptions of the Nature of Science: A review of the research. Journal of Research in Science Teaching, 29(4), 331-359.

Lederman, N. G. (1992). Students' and Teachers' Conceptions of the Nature of Science: A review of the research. Journal of Research in Science Teaching, 29(4), 331-359.

Lederman, N., & O'Malley, M. (1990). Students' perceptions of the tentativeness in science: Development, use, and sources of change. Science Education, 74, 225-239.

Lehrer, R. (1993). Authors of Knowledge: Patterns of hypermedia design. Chapter 7. In S. P. Lajole & S. J. Derry (Eds.), Computers as Cognitive Tools (pp. 197-227). Hillsdale, NJ: Lawrence Erlbaum Associates.

Linn, M. C., Songer, N. B., Lewis, E. L., & Stern, J. (1990). Using technology to teach thermodynamics: Achieving integrated understanding. In D. L. Ferguson (Ed.), Advanced Technologies in the Teaching of Mathematics and Science Berlin: Springer-Verlag.

Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991). Complex, Performance-Based Assessment: Expectations and validation criteria. Educational Researcher, 20(8), 15-21.

Mackey, L. (1971). Development of understanding about the nature of science. Journal of Research in Science Teaching, 8, 57-66.

Maeroff, G. I. (1991). Assessing Alternative Assessment. Phi Delta Kappan, 73(4), 272-281.

Magnusson, S. J., Boyle, R. A., & Templin, M. (1994). Conceptual Development: Re-examining knowledge construction in science. In Annual Meeting of American Educational Research Association, . New Orleans, LA:

Marx, G., & Tóth, E. (1981). Models in science education. impact of science on society, 31(4), 389-397.

Mason, C., Kahle, J., & Gardner, A. (1991). Draw-A-Scientist Test: Future Implications. School Science and Mathematics, 91(5), 193-198.

Matthews, M. R. (1989). The Scientific Background to Modern Philosophy: Selected Readings. Indianapolis, IN: Hackett Publishing Co.

Mead, M., & Metraux, R. (1957). Image of the Scientist among High-School Students. Science, 126, 384-390.

Mehrens, W. A. (1992). Using Performance Assessment for Accountability Purposes. Educational Measurement: Issues and Practice, 11(1), 3-9, 30.

Melograno, V. J. (1994). Portfolio Assessment: Documenting authentic student learning. Journal of Physical Education, Recreation and Dance, 65(Oct.), 50-55+.

Metz, K. E. (1995). Reassessment of developmental constraints on children's science instruction. Review of Educational Research, 65(2), 93-127.

Mitman, A. L., Mergendoller, J. R., Marchman, V. A., & Packer, M. J. (1987). Instruction addressing the components of scientific literacy and its relation to student outcomes. American Educational Research Journal, 24(4), 611-633.

Newmann, F. M., & Wehlage, G. G. (1993). Five Standards of Authentic Instruction. Educational Leadership(April), 8-12.

Nickerson, R. S. (1995). Can technology help teach for understanding. In D. N. Perkins, J. L. Schwartz, M. M. West, & M. S. Wiske (Eds.), Software Goes to School: Teaching for understanding with new technologies (pp. 7-22). New York, NY: Oxford University Press.

Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. New York, NY: Cambridge University Press.

NRC, The National Research Council (1996). National Science Education Standards No. National Academy of Sciences.

NSTA, The National Science Teachers Association (1993). Scope, sequence, and coordination of secondary school science: The content core. Washington, DC: Author.

Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1, 117-175.

Papert, S. (1991). Situating constructionism. In I. Harel & S. Papert (Eds.), Constructionism: Research reports and essays, 1985-1990 (pp. 1-11). Norwood, NJ: Ablex Pub. Corp.

Papert, S. (1993). Literacy and Letteracy in the Media Ages. Wired magazine.

Pearsall, N. R., Skipper, J. J., & Mintzes, J. J. (1996). Knowledge Restructuring in the Life Science: A Longitudinal Study of Conceptual Change in Biology. In J. (. Thomas R. Koballa (Ed.), A paper presented at the Annual Meeting of the National Association for Research in Science Teaching, . St. Louis, MO: University of Georgia.

Perkins, D. N. (1986). Knowledge as Design. Hillsdale, NJ: Lawrence Erlbaum Assoc. Publ.

Perkins, D. N. (1991). Educating for Insight. Educational Leadership, 49(Oct), 4-8.

Perkins, D. N. (1992). Smart Schools: from training memories to educating minds. New York, NY: Free Press.

Perkins, D. N. (1994a). Do students understand understanding? The Education Digest, 59(Jan), 21-25.

Perkins, D. N. (1994b). Where is intelligence? Educational Leadership, 51(May), 105-6.

Perkins, D. N., & Blythe, T. (1994). Putting understanding up front. Educational Leadership, 51(Feb), 4-7.

Perkins, D. N., & Simmons, R. (1988). Patterns of misunderstanding: an integrative model for science, math, and programming. Review of Educational Research, 58(Fall), 303-326.

Perkins, D. N., Crismond, D., Simmons, R., & Unger, C. (1995). Inside Understanding. In D. N. Perkins, J. L. Schwartz, M. M. West, & M. S. Wiske (Eds.), Software Goes to School: Teaching for understanding with new technologies New York, NY: Oxford University Press.

Perry, W. G., Jr. (1970). Forms of intellectual and ethical development in the college years. Fort Worth, TX: HBJ College Publishers.

Posner, G. J., Strike, K. A., Hewson, P. W., & Gertzog, W. A. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science Education, 66(2), 211-227.

Rampal, A. (1992). Images of science and scientists: A study of school teachers' views. I. Characteristics of scientists. Science Education, 76(4), 415-436.

Renninger, K. A. (1992). Individual interest and development: Implications for theory and practice. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.), The Role of Interest in Learning and Development (pp. 361-395). Hillsdale, NJ: Lawrence Erlbaum, Assoc.

Resnick, L. B. (1987). Learning in school and out. Educational Researcher, 16(9), 12-20.

Resnick, M. (1994). Changing the centralized mind. Technology Review 97:32-40 Jul '94.

Resnick, M. (1996). New paradigms for computing, new paradigms for thinking. In Y. Kafai & M. Resnick (Eds.), Constructionism in Practice: Designing, thinking and learning in a digital world (pp. 255-267). Mahwah, NJ: Lawrence Erlbaum Assoc.

Riley, D. (1990). Learning about systems by making models. Computers & Education, 15(1-3), 255-263.

Roth, W.-M. (1992). Dynamic Evaluation (Concept Mapping). Science Scope, 15(6), 37-40.

Roth, W.-M., & Roychoudhury, A. (1992). The social construction of scientific concepts or the concept map as conscription device and tool for social thinking in high school science. Science Education, 76(5), 531-557.

Ryan, A., & Aikenhead, G. (1992). Students' preconceptions about the epistemology of science. Science Education, 76, 559-580.

Rye, J. A., & Rubba, P. A. (1996). An Exploratory Study of the Concept Map as a Tool to Facilitate the Externalization of Students' Understandings about Global Atmospheric Change in the Interview Setting. In J. (. Thomas R. Koballa (Ed.), A paper presented at the Annual Meeting of the National Association for Research in Science Teaching, . St. Louis, MO: University of Georgia.

Schauble, L. (1990). Belief revision in children: The role of prior knowledge and strategies for generating evidence. Journal of Experimental Child Psychology, 49, 31-57.

Schauble, L., Glaser, R., Duschl, R. A., Schulze, S., & John, J. (1995). Students' Understanding of the Objectives and Procedures of Experimentation in the Science Classroom. The Journal of the Learning Sciences, 4(2), 131-166.

Schauble, L., Klopfer, L. E., & Raghaven, K. (1991). Students' transition from an engineering model to a science model of experimentation. Journal of Research in Science Teaching, 28(9), 859-882.

Schibeci, R. A., & Sorensen, I. (1983). Elementary School Children's Perceptions of Scientists. School Science and Mathematics, 83, 14-20.

Schoenfeld, A. H. (1985). Mathematics, technology, and higher order thinking. In R. S. Nickerson & P. P. Zodhiates (Eds.), Technology and education: Looking toward 2020 (pp. 67-96). Hillsdale, NJ: Erlbaum.

Shavelson, R. J., & Baxter, G. P. (1992). Linking Assessment with Instruction. In F. K. Oser Andreas Dick and Jean-Luc Patry (Ed.), Effective and Responsible Teaching: The New Synthesis San Francisco: Jossey-Bass Publishers.

Shepardson, D. P., & Britsch, S. J. (1996). When Dinosaurs Roamed: Hypermedia and the Learning of Mathematics and Science. Journal of Computers in Mathematics and Science Teaching, 15(1/2), 7-18.

Smith, C., Snir, J., & Grosslight, L. (1987). Teaching for conceptual change using a computer modeling approach: The case of weight/density differentiation. (Technical Report No. Cambridge, MA: Harvard University, Educational Technology Center.

Solomon, J. (1992). Images of physics: How students are influenced by social aspects of science. In R. Duit, F. Goldberg, & H. Niedderer (Eds.), Research in physics learning: Theoretical issues and empirical studies (pp. 141-154). Kiel, Germany: Institute for Science Education at the University of Kiel.

Soloman, J. (1993). Teaching Science, Technology & Society. Buckingham, UK: Open University Press.

Spitulnik, M. W. (1995). Students Modeling Concepts and Conceptions: What connections do they make? In Paper presented at the National Association for Research in Science Teaching Annual Meeting. San Francisco. April 21-25, 1995., .

Spitulnik, M. W., Stratford, S., Krajcik, J., & Soloway, E. (in press (1996)). Using Technology to Support Students' Artifact Construction in Science. In International Handbook of Science Education Netherlands: Kluwer Publishers.

Spoehr, K. T. (1994). Enhancing the Acquisition of Conceptual Structures through Hypermedia. Chapter 4. In K. McGilly (Ed.), Classroom Lessons: Intergrating cognitive theory and classroom practice (pp. 75-101). Cambridge, MA: The MIT Press.

Stratford, S. J., Krajcik, J., & Soloway, E. (1997a). Secondary students' dynamic modeling processes: Analyzing, reasoning about, synthesizing and testing models of stream ecosystems. In A paper presented at the annual meeting of the American Educational Research Association (AERA), . Chicago, IL:

Stratford, S. J., Krajcik, J., & Soloway, E. (1997b). Technological Artifacts Created by Secondary Science Students: Examining Structure, Content, and Behavior of Dynamic Models. In A paper presented at the Annual Meeting of the National Association for Research in Science Teaching (NARST), . Oakbrook, IL:

Strike, K. A., & Posner, G. J. (1992). A revisionist theory of conceptual change. In R. A. Duschl & R. J. Hamilton (Eds.), Philosophy of Science, Cognitive Psychology, and Educational Theory and Practice (pp. 147-176). Albany, NY: State University of New York Press.

Symington, D., & Spurling, H. (1990). The 'Draw a Scientist Test': Interpreting the data. Research in Science and Technological Education, 8(1), 75-77.

Vargas, E. M., & Alvarez, H. J. (1992). Mapping Out Students' Abilities. Science Scope, 15(6), 41-43.

Wallace, J. D., & Mintzes, J. J. (1990). The Concept Map as a Research Tool: Exploring Conceptual Change in Biology. Journal of Research in Science Teaching, 27(10), 1033-1052.

Wandersee, J. H., & Fisher, K. M. (1995). A semantic network-based analysis of the AAAS Benchmarks for biology. In Annual Meeting of the American Educational Research Association, . San Francisco, CA:

Warren, C. R. (1990, April). An Exploration of Factors Influencing the Career Preferences of Junior High Students. In Paper presented at the Annual Meeting of the National Science Teachers' Association, . Atlanta, GA:

Waterman, M. (1983). Alternative conceptions of the tentative nature of scientific knowledge. In J. Novak (Ed.), Proceedings of the International Seminar on Misconceptions in Science and Mathematics (pp. 282-291). Ithaca, NY: Cornell University.

Webster's (1985). Webster's Ninth New Collegiate Dictionary. Springfield, MA: Merriam-Webster Inc.

Welch, W., & Pella, M. (1967). The development of an instrument for inventorying knowledge of the processes of science. Journal of Research in Science Teaching, 5, 64-68.

White, B. (1990). Reconceptualizing science and engineering education. Unpublished manuscript. Cambridge, MA: BBN Laboratories

White, R., & Gunstone, R. (1992). Probing Understanding. London • New York • Philadelphia: The Falmer Press.

Wiggins, G. (1989). A True Test: Toward more authentic and equitable assessment. Phi Delta Kappan, 70(9), 703-713.

Wiggins, G. (1993). Assessment: Authenticity, Context, and Validity. Phi Delta Kappan, 75(3), 200-214.

Wisnudel, M. (1994). Constructing hypermedia artifacts in math and science classrooms. Journal of Computers in Mathematics and Science Teaching, 13(2), 127-145.

Wolf, D. P. (1989). Portfolio Assessment: Sampling student work. Educational Leadership, 46(Apr.), 35-39.

Wollman, W. (1977a). Controlling variables: Assessing levels of understanding. Science Education, 61, 371-383.

Wollman, W. (1977b). Controlling variables: A Neo-Piagetian developmental sequence. Science Education, 61, 385-391.

Wollman, W., & Lawson, A. (1977). Teaching the procedure of controlled experimentation: A Piagetian approach. Science Education, 61, 57-70