Formative Evaluation and Revisions

In the first cluster, students appear to have done reasonably well. The significant increase in the percent of questions answered correctly indicates that learning did indeed take place there. Cluster E included significant gains, but students still performed poorly. Scores on specific questions within that cluster indicate that students had trouble on both the pre- and post-tests correctly identifying links to other sites. When correctly identified, the ability to draw appropriate conclusions based on that information rose significantly between the pre-test (31% correct) and the post-test (67% correct).

Cluster C saw modest gains in achievement. Students increased in their ability to identify contact information for the author, but decreased slightly in their ability to assess the author's expertise and the overall assessment of the author's credibility. This may be due to confusion created when no credential information is available about the author. In such cases, the author's credentials are unknown, whereas the students may be indicating that the absence of such information means the author is not a credible source.

Clusters D and F both saw slight decreases in student achievement. Cluster D involved the recognition of time-sensitive material and implications of a date of revision on the reliability of a web page. Student were far better able to identify time-sensitive information on the post-test (87% versus 56%). On the other hand, students declined significantly in their ability to identify a date of revision in a web page (92% on the pre-test compared to 74% on the post-test). Upon further review of the tests, it was discovered that those pages on the pre-test without revision dates fell in the later parts of the test (which were not completed by most students) whereas the pages without revision dates on the post-test appeared earlier in the test. This may indicate that the pre-test score is artificially high, since it is easier to identify a revision date that is there than it is to conclude there isn't one.

Cluster F is arguably the most important of the group. This is where the students, armed with the assessments of the various aspects of the web pages, finally draw a conclusion about the suitability of the page. It could be argued that the test questions do not adequately measure the objective, because the students determine whether a page is suitable for research as part of a complete assessment of the page. Since the answer to this question depends on the answers to the previous questions, students who have made mistakes on previous sections could make decisions about the site which are incorrect but consistent with their answers to previous questions. For example, if a student mistakenly states that the page's content matches the domain name when it actually doesn't, they may conclude that the page can be used for research with caution when in fact it is unsuitable for research. It's clear that a better assessment of this cluster is needed.

Cluster B saw the worst performance, dropping 8% between the pre-test and the post-test. This cluster includes the assessment of the content of the page, including whether the document is well-written, whether it contains evidence of author bias, and whether it coincides with prior knowledge. This drop is attributable to the students' unexpected failure to see the humor in an article on Dihydrogen monoxide. The article points out the dangers of the chemical, implicating it as a cause of death in many accidents and a major component of acid rain. It was expected that students would suspect the document, because it claims the chemical is widely available, and the students have not heard about its danger from other sources. Dihydrogen monoxide is actually water, a point which few students realized. This page was incorrectly classified as one of reliable content by more than half of the students. Since there were few questions asked for this cluster of objectives, this bad question was enough to skew the results.

Looking at the individual students observed during the instruction, the average pre-test score was 69% and the average post-test score was 75%. Of the four, however, two students declined between the pre-test and the post-test. The first student observed, who had an A average in the class, was not at all interested in the instruction. He chose to skip most of the instructional portions, completing only the sections where he was asked to provide answers, and where he was given feedback. Even after suggesting he read the materials, his primary objective was to simply finish as quickly as possible. His post-test score was 73%, significantly lower than his class average of 94%.

The second student seemed to take the instruction much more seriously. With a B average in the class, this student is highly motivated by multimedia, and has been anxiously awaiting the class' venture onto the world wide web. He seemed to follow intently, reading the instruction carefully. In a few cases, he did not read the associated example web pages as carefully, but that didn't seem to affect his progress. His post-test score of 93% was significantly higher than his pre-test score of 53%, and slightly higher than his class average of 91%.

The third student struggles in the class. Her class average of 82% has been earned through hard work, patience, and determination. She is not well motivated to use computers and technology, but she does want to get a good grade in the class, so she tries hard. Throughout the instruction, she read carefully, frequently missing the answers in the sections where she's asked to try her skills. She didn't quite finish the instruction, but was more than halfway through the last section (cluster F) at the end of the 42-minute period. On the post-test, she missed all of the questions on cluster F. Her score of 73% was still a slight increase over her pre-test score of 68%, and probably would have been higher had she finished the instruction.

The fourth student is intelligent but a behavior problem. His 90% average in the class could be higher if he spent more time in class on-task. He had trouble focusing his attention on the instruction for more than a few minutes. After fifteen minutes of instruction, he was off task, after having completed most of the section on evaluating content. When asked to move back to the instruction after he started disturbing other students, he worked much more quickly, skipping the instruction and concentrating only on those sections where he was asked to answer questions. He finished the final four sections of instruction in fifteen minutes. His score on the post-test was a 60%, which is significantly lower than the 74% he received on the pre-test.

Looking at these four cases, it is apparent that students who complete the instruction as it is intended do better than those who skip the instruction in favor of the interactive questions. Clearly, some greater incentive is needed for the students to complete the instruction as intended.

Recommendations

The analysis of the data provided by the pre- and post-test scores of the 78 participants, along with the item analyses of the two tests and the individual observations of four students participating in the instruction leads to several recommendations for improving the instruction:

1. Students had problems defining "bias" and "URL" even though they seemed to be familiar with the concepts. Add some instruction explaining the term "URL," because its definition should be understood by students using the world wide web, so they can accurately document their sources. Replace references to "bias" with references to "option," which is a word with which the students are more familiar.
2. Better explain the difference between links within a web site and links to other web sites. If an author provides links between two of his pages on his own site, that does not add credibility to his work. If he provides links to other sites on the same subject, it may. Students were generally confused between these two cases.
3. A better explanation of "author's credentials" is needed. Since the credentials of the author are so infrequently included on each of his or her pages, a deeper level of complexity is probably appropriate. The page may have links to other pages which contain information about the author, for example. An author's credibility may also be inferred from the text of the page, but not explicitly stated. Also, when the author is an organization, the credibility might be evident but not explicitly stated, as in the case of USA Today.
4. The first web page used in the post-test should be replaced by another page with similar characteristics. The students' inability to recognize the page as a humorous one caused them to make several mistakes on the post-test. The page could still be used as an example within the instruction. The page used to replace it in the post-test should be one which is well-written, but contains obvious factual mistakes.
5. The assessment for cluster F should give students a web page, along with conclusions from the first five clusters for that page, and ask them to draw a conclusion. The current questions assume that the student has correctly assessed all five categories, which may not be the case.
6. The instruction should be redesigned to better motivate the students. The use of grades is not sufficient, nor is the intrinsic student motivation from wanting to find accurate information online. The use of shorter explanations with more examples of web pages may help. The example pages should be shorter and more interesting for the students. The use of clipart, graphics, and visual design elements, omitted because of the time necessary to create them, should be included in the product.
7. An error in the coding for the assessments caused some student essays to be erroneously truncated. The data available indicates that this is related to the students' use of the apostrophe in their essays. This problem should be investigated and fixed. The lost essays were not used in the assessment or the formative evaluation.
8. Some students refused to read the instruction presented, choosing instead to skip ahead to the parts of the instruction requiring a user response. When feedback is given on such responses, the instruction should determine whether the student's responses are correct or incorrect, and provide appropriate feedback. As it is, only the correct answers are displayed along with the student answers. Students who score below a certain level on each section should receive remedial instruction. Only students who receive appropriate scores on the practice exercises should continue with the instruction.
9. The pre-test and post-test should be shortened. Even though they will not provide data which is as reliable, the tests must be short enough to allow the students to complete them in a class period. This was done out of necessity on the post-test, and should also be done one the pre-test.
10. The numerous typographical, spelling, and grammatical mistakes in the instruction should be corrected.

Despite their poor performance on the post-test, it is believed that most students learned something about evaluating web sites from this instruction. Though there are improvements to be made, the instruction is still worthwhile. Further improvements will make it a more valuable tool, and hopefully one which will be used by eighth graders for the next few years.

Return to the EvalWEB Instructional Design.