Chapter 3. General Description on Design and Redesign the Experiment

After reviewing the literature about the learning philosophies, designing theories and theories concerning with the concept of Parallelism and the PI theory in chapter 2, chapter 3 enters the planing and preparation phase of the design and redesign on the experiment in respect to the concept of Parallelism and the PI theory. This chapter will describe the general design and redesign of the experiment indicating which designing model and guidelines are used, and what kinds of decisions and preparation have been made. Section 3.1 describes the specific context for the design and redesign. Section 3.2 describes the general design and redesign including guidelines used, decisions made and design model referred to Plomp (1982). The final section concludes this chapter.

3.1 Specific Context for the Design and Redesign

Designing and redesigning of the experiment was the speciality in the Faculty of Educational Science and Technology (also shortly called TO faculty in Dutch), University of Twente, the Netherlands. The web-site I have redesigned was developed by members of the research project: Buwalda, Fokkema and by Min. The target of all web-based learning environment is TeleTOP, which is a course management system for e-learning environment developed by TO Faculty. The experiment will be conducted by using the TeleTOP system.

3.1.1 About the TO Faculty

The Faculty of Educational Science and Technology is famous for its combination of knowledge of eductaional and training science and technology support. The faculty consists of five departments including Curriculum, Instructional Technology, Educational Instrumentation, Educational Organization & Management and Educational Measurement and Data Analysis, which cover most aspects in educational domain.

All the academic staffs of the faculty have PhD backgrounds. To meet the requirement of e-learning, most of the instructors prefer to use media-based instructions such as giving presentation by PowerPoint software in TeleTOP, simultaneously using readers, manuals and sheets and so forth. In some courses peer- coaching methods or group work are adopted. Besides, a department called Helpdesk can provide especial technological support during emergent condition.

For the students, the faculty main severs three international Master of Science Programmes in Educational and Training Systems Design (ETSD), Human Resources Development (HRD) and Telematics Applications on Education and Training (TAET). In addition, the faculty offers a four-year post-secondary programme in Educational Science and Technology given in the Dutch language.

Most of the hardware for modern education application are available in the TO faculty. Computers are widely used in every classroom. Printers, copy machines, digital cameras, digital video recorders and automatic projectors are all available. Furthermore, the faculty offers its own network facilities, digital contents, books, papers, library and computer rooms.

In general, the facilities and environment are quite suitable for studying, researching and instructing.

3.1.2 About TeleTOP

The TeleTOP, which is short for TeleLearning at Toegepaste Onderwijskunde Project, is a course-management systems developed at the University of Twente around 1997 that allows learners and instructors to cooperate, communicate and collaborate with each other effectively and efficiently. It is designed to fit the instructor- support strategy and the needs of flexible learning. "Instructors needed to be made aware of possibilities for pedagogical change toward more flexibility and more contribution from their students. They needed to be supported in planning for the redesign of their own courses. They needed to be supported in the actual redesign process as well as the creation of their own WWW environments" (Collis & Moonen, 2001, p153). "With TeleTOP, lecturers can easily set up a site for their particular course, without the necessity of programming skills. The site provides logical and clear access to material, belonging to that particular course, as the basic design of a course Web site is always the same." (Dinkel Institute, 2001).

In the following paragraphs some features of the TeleTOP environment are described and illustrated. Figure 4 shows the interface of the news page.

The common used functions in the navigation bar of TeleTOP are News, Course-Info, Rowster, Email/group, Participants, Workspace, Web links and Archive. For the instruction design and course activity purpose, the first three parts are comparatively crucial. A brief introduction of on these parts are as follows.

News: Whenever the users access the TeleTOP, the news-page always appears on the screen as the start and welcome page. This area generally contains some important information and the latest news usually on the top. For instance, it can inform the participants the starting time of the course or some changes occurring occasionally.

Course Info: Instructors can use this function to offer information on the objectives of the course, the course procedures and ways of evaluation and assessment. Of course this is a nice place for the instructors to introduce themselves and provide more information about the course if they like.

Roster: Roster probably is the most critical part of TeleTOP. The detailed information about the course schedule, the concrete content, the assignment and the submission as well as the necessary feedback can be conducted through this function. According to Collis & Moonen (2001)" A particularly popular Web based tool among our instructors was a matrix-type construction called a roster, to indicate the organization of a course and bring together study materials, lecture notes, student assignments, assignment submissions and feedback, all in one convenient table "

Figure 12. Interface of TeleTOP

General Design and Redesign

In this section, I will state the decisions we made on our design and redsign production as well as on the experiment. Moreover, I will try to explain the reasons why we need such kinds of decisions. A model of the design and redesign is illustrated to give a overview of the whole procedure and process. Here I must emphasize that the first version of the design job, either the program editing or interface design was finished by students and my mentor Min, who is a designer and instructor. He is now working in the Educational Instrumentation department of the Faculty of Educational Science and Technology. I have designed version two together with him.

3.2.1 Decisions on design, redesign and experiment

Many studies and researches on the effectiveness and efficiency of Parallelism and the PI theory in open learning, working and doing environment such as in simulation context have been carried out by Min and other researchers. The outcome shows that Parallelism and the PI theory did work well in that learning environment, but he has no good empirical data until now. Two experiments were done but failed. In an information era if Parallelism and the PI theory as design and information management theory and principle still work well needs to be tested and researched.

From a scientific perspective, more experiments should be done, and furthermore to be conducted to test whether the theory works or not. In another word, the validity and credit of the concept of Parallelism and the PI theory under the web-based learning environment need being examined and inspected. This may be the first and key reason for the design.

Decision one: Making a better design of the first primitive working-environment to examine the validity and credit of the concept of Parallelism and the PI theory.

There had already a first prototype of the experiment design before I joined in and the first version was in Dutch. Since I decided to devote myself to this project, I became a member of the research group. In fact, most jobs of the design had been finished, so what I designed is the second version and this is the reason why I named my project in the thesis title with the words like "empirical study …to a well-designed product".

Decision two: Improving the quality of the experiment to make it suitable to more international students and users to increase the credit and validity of the concept of Parallelism and PI theory.

Some social scientists believe that the strongest chains of reasoning can be built only through experimental design. For them, experimentation is the most effective method for creating a consensus around the existence of a cause-and-effective relationship. The effectiveness, however, depends on many characteristics. (Krathwohl, 1998). To make an effective examination of the validity and credit of Parallelism and the PI theory, we decided to redesign the experiment and improve the quality so as to meet the needs of the international students.

Decision three: Experiencing the new version in relation to the effect of the concept of Parallelism and the PI theory.

This means that after literature studying, empirical research will begin. Within the schedule, the project should start at the middle of March and end at August. The time- table is showing in table 3.

Decision four: We decided that the proper degree of the freedom is twenty subjects. During the process of the experiment, I will choose 20 international students as the subjects to experience the experiment. They will be selected randomly and be international students instead of Dutch students. Among them 16 subjects will conduct the experiment according different sequence of the tasks arranged before, while the other four subjects may finish their tasks freely.

Criteria on the number of the subject are counted on different experiment. Generally, the number of the subject should be the more the better. Since my project contains five tasks, we think twenty subjects can complete one hundred tasks, the validity and credit should be reliable and trustworthy. Of course randomly selecting the subject and setting different sequence of the tasks have the same purpose.

Table 2. Schedule of the project

March 15, 2002 ~ March 31, 2002		Needs analysis and context description (simultaneously)
April 1, 2002 ~ May 25, 2002		Literature study and research (simultaneously)
April 15 2002 ~ May 25, 2002		General and specific design and redesign (simultaneously)
May 25, 2002 ~ June 28, 2002	Experience the experiment
July 1, 2002 ~ July 31, 2002	Data collection and Evaluation

Decision five: We decided that choose 4 types basic tasks which represent the most common used interfaces upon the web-based learning environment and represent the fundamental styles of Parallelism. The fifth task is an extra task, which will be carried out to test the role of distance "x" in the PI theory (See figure13). At the same time, the experiment will be implemented in the real web-based learning environment-the TeleTOP environment.

Figure 13. The concept of Parallelism (adapted from Min)
A: the space for problems: space within which a designer has put a problem, for instance a simulation or a work out task;
B: the space containing instruction or information to solve the 'problem' or 'task';
x: the physical distance between A and B. This distance plays an important part in the PI-theory. Sometimes x is small, sometimes x is large; sometimes the instruction is embedded, sometimes the instruction is the first or the second order of parallelism.

The question will then be: which distance ‘x’ should a designer at least keep between those two aspects A and B? The question is connected with how much information should minimally be side by side on the screen (in square cm) (and remain side by side) when other information is put on top of it by the application? It is not desirable to have a screen full of all kinds of 'useful' information. Then the user will not be able to see the wood for the trees. What is the optimum? Can an optimum be achieved, for instance in density of information or distance between parts of information. Is there an absolute optimum in a work or learning environment or are there (user-related) conditions? Which are those conditions? Etc. (Min, 2002) [Online available at http://projects.edte.utwente.nl/pi/Papers/Papers.html]

3.2.2 Design Model

As to the procedure of the design and redesign of my project, we refered to the "design model" of Plomp (1982) but made a little bit change. In our model, there are five stages: Analyses, Design, Development, Experience (or named Implementation) and Evaluation. Analysis includes three parts: needs analysis, context analysis and subjects analysis. Since chapter one has described the general context and needs analysis and specific context has been offered in section 3.1, the analysis here just introduces the subjects’ analysis. In the design (and redesign) phase a ‘overall blueprint’ will provide the guideline to the procedure of design and redesign as well as to the experiment. Development phase is to describe the concrete steps of the design and redesign. A unique feature in this model is evaluation runs through the whole process of the project. The main purpose is to test if the designed product is the effective solution to the problem. The experience (implementation) phase delivers the final product in a specific context and at the same time working with part function of the evaluation.

Analysis

Analysis could have contained three kinds of analysis: needs analysis, context analysis and subjects, (or learners) characteristics analysis. In section 1.1 of chapter one and section 3.1 of chapter 3 the description of general context and specific context as well as the general problems existing in education domain has been given, which functions as the context analysis and needs analysis. With detailed analysis, we generalized that Parallelism and the PI theory working as a new design theory could be a good solution to the actual problems. It is necessary to apply the PI theory to the instructional design to improve the effectiveness and efficiency of learning.

This section will underline the subjects analysis. Since the project will be carried out in the Faculty of the Educational Science and Technology (TO) of University of Twente, the students in the TO faculty are thus our target subjects.

From the perspective of the physiological characteristics, most of the subjects have high sensory perception. They are aged between 24 to 45 with a good health. Their affective characteristics show that they have a positive attitude toward learning and can be easily motivated to learn. Most of them have their own beliefs with attribution of success. Of course their social characteristics are quite similar: they are good at keep friendly relationship to peers and are feeling toward authority. Most of them have socioeconomic background and have tendencies toward cooperation and competition etc. In addition, they all know how to make use of TeleTOP as their web-based learning environment, though some of them are not so proficiency with web technology.

Through the subjects’ analysis, we can imagine that the experiment could not be so difficult for them. In spite of the different backgrounds, the difference between them should not too big due to the age and similar cognitive characteristics. On the other hand, they might be the optimal targets to examine whether Parallelism and the PI theory can be accepted among young people in a web-based learning environment. After all they are the major users of the web-based learning systems at present.

Design and redesign

After a clear analysis of the problem, the subjects as well as the context, the design and redesign process start. The experiment is to be designed and redesigned guided with Parallelism and the PI theory. In order to make the product more active and effective, many aspects have to be taken into the consideration. First the web technology and skill should be involved. The main program languages for web design like HTML or Java language may be chosen. English is obviously the adopted text language for the experiment, thus translation should be finished properly and in time. Secondly to make the experiment more reliable, the selection of the tasks, the sequence of the tasks and the degree of the freedom should be decided early. Thirdly, to make the evaluation more accurate and credible, the logging system for data collection needs to be designed successfully so that the raw data can be collected easily and correctly with a high validity. Min and I have redesigned the automatic logging system. Fourthly, to guarantee the design, redesign and experiment implement smoothly, the necessary guidelines should be noted.

In a word, the process of design and redesign is the crucial and critical chain in the whole project, it relates the theory and practice and itself is an application process from theory to practice. To an empirical study and "learning by doing" strategy, the design and redesign is no doubt the best workshop.

Development and experience

In the development phase, the actual solution will be applied to the products. In the development process, the decisions and guidelines will convert into the actual actions, steps and methods. While with the development of the products, the experience consequently is needed to have an actual test. Experience process is not only an empirical learning process but also a data collecting process and an actual observation and record process. This process make a full preparation for the assessment and summative evaluation, either for the instructional design and redesign or for the validity and credit of the theory behind the design and redesign. These two phases also play very important roles in the whole project.

Evaluation

As illustrated in our model, evaluation especially the formative evaluations will run through the whole process of my project. In the phases of analysis, design and redesign as well as development, the patterns of formative evaluation are communicating and discussing with experts, using checking lists to revise and modify the mistakes and consulting with peers and colleagues to get more information. As to the summative evaluation, the main methods include interviews, observations and automatic data collecting by using an automatic tracking system. Because evaluation directly concerns with the validity and credit of the experiment, more attention will be focused on it.

Figures 14. The design model

3.2.3 Development of the project

Two important steps are described in the development process of the project. The first point is to setting objectives and guidelines, which are the requirements of design and redesign the project. The second point is to arrange a proper squence of tasks among twenty subjects, which combining with the method of randomly selecting subject aims to increase the validity and credit of the experiment.

Setting objectives and Guidelines

The goals of the design and redesign of the experiment for web-based learning environment are to examine whether the concept of Parallelism and the PI theory fit for the web-based learning environment as a design theory? Comparing with traditonal learning theories, if Parallelism and the PI theory can make the learning process more effective and easier? To achieve the goals, some objectives could be discomposed as follows:

Given a web-based learning environment, if Parallelism can decrease the Extraneous Coginitive Load (ECL) and increase the Germane Cognitive Load (GCL) so that it can improve the effectiveness and efficiency of learning.
Given a web-based learning environment, if Parallelism and the PI theory will offer help to the users who have troubles due to their limited short-term-memory.
Given a web-based learning environment, if the information guided by the principle of Parallelism and the PI theory is preferred by the users in the information with a linear output format.
Given a web-based learning environment, if Parallelism and the PI theory help the users who want to create his own frame of reference?
Given a web-based learning environment, which kind of styles of Parallelism as design principle should be taken into account?
Given a web-based learning environment, which kind of style of parallelism, as design principle should be avoided?

To improve the quality of experiment and design, some guidelines should be noted.

The problem-solving procedures could be taken as the Guidelines for our design, redesign and experiment.
Choice of good design model is half success of the project.
Keeping objective attitude in the whole experiment process.
Motivation is necessary to make the experiment going on smoothly.

Developing different styles of tasks

After a consideration on the purpose of the experiment, five kinds of tasks have been selected and developed, which represent the basic styles of the web pages. Figure 15 to figure 19 offer a visual explanation.

Figure 15. The scheme of Task A: The problem space on the computer is a veritical window, the instruction space is on a sheet with a ‘x’ about 30 cm (the second order parallelism)

Figure 16. The scheme of Task B: The problem space on the computer is a parallel window, the instruction space is on a sheet with a ‘x’ about 30 cm (the first order parallelism)

Figure 17. The scheme of Task C: The problem space on the computer contains a top window, the instruction space is on a sheet with a ‘x’ about 30 cm (the second order parallelism)

Figure 18. The scheme of Task D: The problem space on the computer is a vertical window, while the instruction space is also on the computer screen, which forms a long vertical window (no paper)

Figure 19. The scheme of Task E: The probem space on the computer is a formal window, but the instruction space is on a sheet which is two meters away from the computer (great distance)

Arranging the sequence of the tasks

The keep the validity and credit of the experiment, a lot of ways were adopted such as choosing the subjects randomly, designing an logging system for the data collcetion, arranging an extra task (task 5) to make a comparison and so on. Among them, arranging the sequence of the tasks is an unique way. Table 3 show the maximum choices for the sequences of four tasks.

Table 3. Theoretical sequences of four tasks (original idea)

Sequence	Tasks
#1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24	A A A A A A B B B B B B C C C C C C D D D D D D	B B C C D D A A C C D D A A D D B B A A C C B B	C D B D B C C D A D C A B D A B D A B C A B A D	D C D B C B D C D A A C D B B A A D C B B A D A

At the very beginning, I thought that the comparisons in the experiment occur on the subjects themselves so that there was no need to set different condition for the subjects. Therefore I decided to arrange the experiment scheme in the same sequence, for instance in experiment scheme one, I set the same sequence for all the subjects on a sequence of A, B, C, D and E. A represents task 1, B represents task 2, C represents task 3 and so on.

But after discussing with my mentor, he proposed the different opinion. He thought that following my experiment scheme the validity of the experiment might be reduced due to the repeated sequence, although the subjects were different. With second consideration, I accepted his advice and made a change. Although there exist twenty-four different sequences for the four regulativ tasks, since we decided the degree of freedom was twenty, we then chose sixteen subjects to finish tasks with different sequences on purpose while the other four subjects would choose the tasks randomly. With this method together with the random selection of the subjects, the validity and credit of the experiment should be kept in a high degree. Ttable 4 shows the designed scheme we used in the experiment.

Table 4. The designed scheme of the real experiment

Sequence	Regulative Tasks				Extra tasks
#1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20	A A A A B B B B C C C C D D D D A B D A	B B C C A A C C A A B B A A B B B A A C	C D B D C D D A B D A D B C C A C C B B	D C D B D C A D D B D A C B A C D D C D	E E E E E E E E E E E E E E E E E E E E

Following literature reading of chapter 2, this chapter described the specific context of the design and redesign in section 3.1. In this specific context, TO faculty and TeleTOP were underlined. Section 3.2 is the introduction about the general design and redesign of the experiment. In this section, the design and redesign decisions were mentioned, the design model was emphasized. The last part gave a brief introduction of the development of the project, mainly including the objectives and the tasks sequence arrangement. Relating with this chapter, chapter 4 will give a demonstration of the final objects made in my project and a deep statement about specific design and redesign process.

Enschede, sept, 3, 2002