Assessment of Creative Knowledge Building in Online Problem-based Learning

views updated

Assessment of Creative Knowledge Building in Online Problem-based Learning

Tjaart Imbos and Frans Ronteltap
Maastricht University, the Netherlands

Abstract

Deep learning is facilitated by instructional designs in which active and interactive student behaviors are the core concept, in combination with the use of tools for knowledge building and knowledge exchange. This chapter considers the question of how teachers can assess the progress of learning in an online collaborative learning environment that fosters learning through interaction. In such an environment, students articulate in writing their understanding of the topics in a domain. In a subsequent group interaction, they elaborate and enrich their understanding, as well as clarify misunderstandings and misconceptions. The tutor's challenge is how to assess students' written work to gauge their proficiency in the topics, individually and as a group. Two models are explored for their potential application in assessing knowledge construction. One is latent semantic analysis, a mathematical model that induces the meaning of words by analyzing the relations between words and passages in bodies of text, which may be employed in automatic text scoring. The other is Bayesian networks, graphical models representing sets of variables and their relationships, which may be adopted for the assessment of student competence.

This chapter is based on a paper presented by Tj. Imbos at the IASE Satellite Conference on Assessing Student Learning in Statistics.

Introduction

At Maastricht University, all curricula are problem based. Students work on problems in small learning groups. The groups start their learning with a discussion about a specific problem. This discussion serves to organize the self-directed learning activities that follow the group meeting. The progress of self-directed learning is continuously evaluated, and it ends with a wrap-up and reflection on the outcomes of learning. Figure 12.1 is a graphical representation of the problem-based learning (PBL) process.

Two phases of interaction can be distinguished in the PBL process, with different activities and different functions. Interaction at the beginning is focused on the analysis of the problem by brainstorming. All students are involved in a rapid and spontaneous exchange of ideas about the resolution of the problem. From a cognitive perspective, students engaging in this activity are activating prior knowledge and trying to integrate new concepts into an existing knowledge structure. The interaction in the second phase has the function of broadening students' knowledge base when they share and elaborate their acquired knowledge and understandings relating to the resolution of the problem.

Before we continue, it is relevant at this point to ponder the relation between PBL and creativity. If we define creativity as the ability to find new and unusual solutions to a problem, PBL then is a creative process. From an individual perspective, an attitude of innovation is cultivated as students generate and evaluate new ideas. From a group perspective, creativity is promoted by the interaction between peers. In summarizing, we can say that PBL as a whole is a creative process in which brainstorming in the first phase stimulates the divergence of ideas while the exchange of knowledge in the second phase leads to convergence (Jeong & Chi, 2007).

Maastricht University adopts the seven-step procedure of Schmidt (1983) as its PBL framework. These steps are as follows:

  1. Clarify concepts not readily comprehensible.
  2. Define the problem.
  3. Analyze the problem: brainstorm.
  4. Analyze the problem: compile the results of brainstorming.
  5. Formulate learning issues.
  6. Collect additional information outside the group.
  7. Synthesize and evaluate the acquired information.

The first steps of the procedure focus on problem definition and analysis, as a basis for theorizing toward building a personal theory or a mental model of the problem in question. Collaborative learning situations stimulate discussion, debate, and elaboration, as well as verbalization and explicit formulation of relevant concepts and processes (Van der Linden et al., 2000). Bereiter and Scardamalia advocate the idea of student communities working together to become proficient in particular fields of knowledge (Bereiter, 2002; Scardamalia & Bereiter, 1994), which they call knowledge-building communities. In these communities, students become knowledge builders and participate in knowledge-building discourse. Presented with problems to initiate learning, students engage in productive interaction in a decentralized, open learning environment to build collective understanding and to gain a deep understanding of the underlying theories (Bereiter & Scardamalia, 2003).

The creative brainstorming phase in the seven-step procedure ends with the formulation of learning issues for investigation following the group meeting. A growing number of group learning tools are available to facilitate collaboration. Philip (2007) describes the new possibilities offered by one such tool called Knowledge Forum, an asynchronous platform: "Idea generation can take place during these group sessions, during which all students are given the chance to express their ideas, or in individual notes posted directly to the KF database. While in a typical classroom setting ideas or comments generated in discussion are usually lost, the KF database preserves these ephemeral resources so that students can return to them for comment and reflection. Students are then encouraged to read the notes of other students and soon find that there are differing schools of opinion about the problem. The teacher's job is to ensure that students remain on task and work towards the solution of the problem under study by reading each other's notes and contributing new information or theories to the database."

The role of teachers, as facilitators of learning, will be made easier if they have access to tools for analyzing the knowledge-building process. Advances in cognitive psychology have extended our understanding of students' learning and broadened the range of student performances that can be assessed to gather evidence of the development of knowledge and abilities. Research in cognitive psychology and artificial intelligence has led to the development of tools for describing and representing the knowledge that one possesses.

In this chapter, two mathematical models are explored as tools for understanding and diagnosing the nature of the knowledge demonstrated in students' writing. They are latent semantic analysis and Bayesian networks.

Assessing Students' Written Work

At the School of Health Sciences, students are assigned to small collaborative learning groups for each course. These groups are guided by a tutor. In academic year 2002–2003, a group of students actively used the online knowledge-building tool POLARIS (Ronteltap et al., 2007). While attending a basic statistics course covering topics such as statistical testing, regression analysis, analysis of variance, and analysis of cross-tabulated data, the group posted a total of 167 comments in 30 different discussion threads, with a mean length of 5.7 lines of reasoning. Some examples are presented in Figure 12.2. An interpreted analysis of one of the comments done using Atlas.ti (http://www.atlasti.com), a program for the qualitative and quantitative analysis of text, is shown in Figure 12.3. On the left of the figure is the student's comment, while on the right the interpreted analysis. Such programs are nice and convenient tools, but they are very time-consuming to use because all the codes for interpreting text have to be manually created and are thus subjective. (For an overview of such programs, see Popping, 2000.) There is a need for an automated system for scoring and classifying students' written work.

Building a Knowledge Base for Assessing Student Proficiency

What are the prerequisites for such a system? To begin with, an extensive knowledge base containing all the topics relevant to our courses has to be built by feeding it with expert knowledge as well as knowledge from non-experts with various levels of proficiency: novices, student experts, advanced students, and so on. This database will become the reference against which the writing of students on a topic can be compared, scored, and interpreted. For this purpose, the contents of the database need to be analyzed and categorized into knowledge elements, explanations, dialogues, strategies, and processes. A knowledge base is not simply a repository of inert knowledge; it needs to be updated constantly to remain relevant.

Knowledge is organized in mental structures. Some properties of these individual internal mental structures may reflect what a person understands: the richness of the mental structures, the integration of the studied material, and the use of additional concepts and how they link. To measure understanding, these properties have to be preserved during knowledge elicitation and then expressed in external knowledge representations for evaluation (Bude, 2007). We can elicit the knowledge of experts and subject it to cognitive analysis of discourse as applied in research on expert tutoring. A description of the cognitive processes, reasoning, and knowledge representations involved can then be made. To be useful, this information needs to be classified and compiled. The result is an expert model that can be used to "understand" the writing of students in order to guide them toward better and deeper understanding of the knowledge they are learning.

An expert knowledge base consists of different types of knowledge. The best known are declarative and procedural knowledge. However, for our purpose, a more elaborate system of knowledge qualification is needed. An expert knowledge base can also be characterized by qualities, such as level of knowledge (deep or surface), generality of knowledge, level of atomization of knowledge, modality of knowledge, and structure of knowledge. A useful system for organizing knowledge is proposed by de Jong and Ferguson-Hessler (1996), which considers not only the elements of a knowledge base but also the function that each element fulfills in a performance or task. This is an important feature, for we do not want our students to just acquire knowledge but also to be able to apply that knowledge to solve problems in the domain that they are studying.

The knowledge matrix that de Jong and Ferguson-Hessler propose for organizing knowledge is characterized by two dimensions: types of knowledge and qualities of knowledge. By combining these two dimensions, a close description of a knowledge base relevant to certain types of problems and tasks can be created. Using knowledge matrices in this way, it is theoretically possible to describe a complete knowledge base. To cover a complete statistics curriculum, the knowledge base will be sizable. To build such a large database, knowledge compilation techniques such as those employed in artificial intelligence can be applied.

Reducing the Dimensionality of the Knowledge Base

Researchers dealing with a large number of variables frequently employ data reduction techniques such as factor analysis and cluster analysis to reduce the number of variables for analysis. In the discussion forums in POLARIS, large amounts of text are created. Hence, data reduction is needed. To analyze qualitative data, as students' writing is, a technique comparable to factor analysis is available: latent semantic analysis (LSA). LSA is a mathematical model for extracting and inferring the meaning of words by analyzing the relations between words and passages in bodies of text (Landauer et al., 1998). The essence of semantic information is captured using the method of dimension reduction.

Suppose two students write about their understanding of statistical testing. Even if they both show a good understanding of the topic, their writing will still differ. The need is to find a way to tell that the two samples of writing are comparable, or to quantitatively compare two parts of the text that will identify the similar reasoning processes of the two students. This capability may be afforded by LSA. LSA infers information from the many relations present in the writing of learners. The analysis does not represent a whole knowledge space but only the paths through which students have chosen to find their way in that space—in other words, how the knowledge space is understood by students. The features of LSA are that (1) it does not assume independence of writing actions but instead uses dependencies to infer the structure of the writing; (2) it reduces the dimensionality of the knowledge space; and (3) it makes no a priori assumptions about the space. LSA is therefore self-organizing.

LSA can be used to automatically assess the semantic similarity between any two samples of text. This meets our assessment need. Using LSA, students' written work can be compared against the expert knowledge base and scores automatically computed to reflect the level of proficiency of a group of students or an individual student in a particular topic. LSA has been successfully applied in automatic essay grading, automatic tutoring, human language acquisition simulations, and modeling comprehension phenomena. Various other applications have been reported (Wild) et al., 2007). An LSA-based system has the capacity to handle hundreds of thousands of documents simultaneously (Landauer & Dumas, 1997).

Toward an Evidence-based Assessment System

Suppose we are able to build an automatic system with complex knowledge bases and reduce its dimensionality using LSA, then a new problem arises: how to make sense of the complex data that result and how to interpret the writing of students as evidence of competency. We cannot use the statistical methods and rules of thumb developed for classroom quizzes and standardized tests. To solve this problem, two conditions have to be fulfilled. The first is that we need tools of probability-based reasoning that have proven to be useful in modern test theory and adapt them for the more complex situations that result from the system we have in mind. Second, we need more than a scoring system. We also need to establish principles for designing a complex assessment system. Such principles should guide us through questions such as what inferences do we want to make, what observations do we need to support these inferences, what situations can evoke these observations, and what reasoning can connect them. In other words, we need a framework for designing assessments, an evidence-centered framework (Mislevy et al., 2002). In such a framework, three basic models should be present and connected. They are student models, evidence models, and task models.

The student model describes which competencies should be assessed. The model uses student variables to approximate aspects in the domain of interest. Students are measured and scored on these variables, which are actually unobservable. These variables can be behaviorist, trait, cognitive, or situational, but in all cases the issue is the same: constructing student variables from limited evidence. The number and nature of student variables to include depend on the purpose of the assessment. It can be one summarizing variable or several variables. If there is more than one variable, the empirical or theoretical relations between the variables can be described for each student at a certain point in time. These relations can be described in terms of a probability distribution that can be updated as new evidence about the student's learning becomes available. In that case, the student model takes the form of a Bayesian inference network, or Bayes net (BN) (Jensen, 2001). BNs offer a methodology for managing knowledge and uncertainty in the complex assessment system that we have in mind.

The evidence model is the heart of evidentiary reasoning in assessment, that is, arguments about why and how the observations are made. The evidence model consists of two parts: (1) the responses or writing of students (students' products) and the corresponding observed variables (scores on aspects of students' products) and (2) the statistical submodel, which expresses how observed variables in probability depend on student variables. Examples of statistical submodels are classical test theory, item response theory, latent class models, and factor analysis. These models can be expressed as special cases of BNs, being an extension of the relation between student variables and observed variables.

Finally, the task model delineates situations that would elicit the behavior described in the evidence model. It provides a framework for constructing and describing the situations in which students act and produce their work products. The task model provides the input for the evidence model.

Bayesian Networks: Connecting the Submodels of an Assessment System

Testing and assessment of student competence has been improved with the application of item response theory (IRT). In IRT, an examinee's capability is expressed in terms of an unobservable student variable. The written responses of a student are assumed to be independent, conditional on both the latent variable and the characteristics of the writing task. An IRT model can be depicted as a graphical model in the same way as is done in structural equation modeling with θ as a single parent of all writing tasks, graphically depicted as arrows pointing from θ to the observed writing response Xj. At the beginning of a discussion thread, the full joint distribution P(X1, X2, … Xn, θ) characterizes a student's θ and his or her future responses to writing tasks. This distribution then can be obtained as the product of the initial distribution of θ, p(θ), times the conditional distribution of each response to the writing task, P(Xj|θ), given by the IRT model, and p(θ) is the examiner's "belief" in an examinee's θ. Based on new information, the examinee's θ can be updated, leading to a new posterior distribution for θ. All new information leads to new inferences about students' θ. This process continues until the written discussion is terminated. Using graphical models as a predictive framework, the resulting BNs combine the student model with the task model of the assessment system and vice versa (see Mislevy, 1994, for a detailed discussion). A complete system is of course complicated, but estimation procedures are available (Mislevy, 1994).

Conclusion

Writing is a powerful way to make students and teachers aware of the ongoing learning processes and to document these processes. However, for assessment purposes, students' work needs to be interpreted and scored. In online learning environments, students' postings can snowball, so a technological system for interpreting and scoring them is necessary.

Developing such a system is a huge challenge. A database of knowledge elicited from experts and derived from published sources is needed, as well as a scoring system to compare students' writing against some standard. For our intended use, the system also needs to be able to update the assessments in the case of incremental learning. Despite the enormity of the task, the system can be developed using cognitive methods to describe knowledge and its use, latent semantic analysis, and IRT and BNs in combination as the main tools. Much work needs to be done, but in a few years we hope to report the experiences with a system for automatically scoring and diagnosing students' proficiency in statistics.

References

Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Erlbaum.

Bereiter, C., & Scardamalia, M. (2003). Learning to work creatively with knowledge. In E. de Corte, L. Verschaffel, N. Entwistle & J. van Merrienboer (Eds.), Powerful learning environments: Unravelling basis components and dimensions (pp. 55–68). Oxford: Elsevier Science.

Bude, L. (2007). On the improvement of students' conceptual understanding in statistics education. Maastricht: Maastricht University Press.

De Jong, T., & Ferguson-Hessler, M. G. M. (1996). Types and qualities of knowledge. Educational Psychologist, 31 (2), 105–113.

Jensen, F. V. (2001). Bayesian networks and decision graphs. New York: Springer.

Jeong, H., & Chi, M. T. H. (2007). Knowledge convergence and collaborative learning. Instructional Science, 35, 287–315.

Landauer, T. K., & Dumas, S. T. (1997). A solution to Plato's problem: The latent semantic theory of acquisition, induction, and representation of knowledge. Psychological Review, 104 (2), 211–240.

Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259–284.

Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59 (4), 439–483.

Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–62.

Philip, D. (2007). The knowledge building paradigm: A model of learning for Net generation students. Innovate, 3 (5). Retrieved March 2008 from http://www. innovateonline.info/index.php?view=article&id=368.

Popping, R. (2000). Computer-assisted text analysis. London: Sage.

Ronteltap, F., Koehorst, A., & Imbos, T. (2007). Online knowledge-building interactions in problem-based learning: The POLARIS experience. In O. S. Tan (Ed.), Problem-based learning in eLearning breakthroughs (pp. 169–184). Singapore: Thomson Learning.

Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. Journal of the Learning Sciences, 3 (3), 265–283.

Schmidt, H. G. (1983). Problem-based learning: Rationale and description. Medical Education, 17, 11–16.

Van der Linden, J., Erkens, G., Schmidt, H., & Renshaw, P. (2000). Collaborative learning. In R. J. Simons, J. van der Linden & T. Duffy (Eds.), New learning (pp. 37–57). Dordrecht: Kluwer Academic.

Wild, F., Kalz, M., van Bruggen, J., & Koper, R. (Eds.) (2007, March 29–30). Mini-proceedings of the First European Workshop on Latent Semantic Analysis in Technology-enhanced Learning. Heerlen, Netherlands.

More From encyclopedia.com