A Meta-Analysis of the Effectiveness of Teaching and Learning With Technology on Student Outcomes

Previous | Table of Contents

APPENDIX

Information Coded for Each Study

Study Characteristics

Author (Report last name, first; e.g., Doe, John).

Year of Study (Report year of study; e.g., 2000).

Number of Comparisons Within Study (Report number; e.g., 1 or 2 or 3).

Student Sex (Males = 1; Females = 2; Mixed or not specified = 3).

Grade Level (Unspecified = 00; 1st grade = 01; Other grades 2-12 use 02 to 12; Mixed primary [K-3] = 24; Mixed middle [4-6] = 25; Mixed upper [7-8] = 26; Mixed high school [9-12] = 27; K-12 = 28).

Unit of Analysis (Unspecified = 0; Individual = 1; Class = 2; School = 3; District = 4; State = 5; Mixed = 6).

Student Sample Size (Report actual sample size; e.g., 4,024).

School Sample Size (Report actual sample size; e.g., 4,024).

Publication Features (Technology journal = 1; Other educational journal = 2).

Students' Ethnicity (Unspecified = 1; Black = 2; Hispanic = 3; Asian = 4; White = 5; Mixed = 6; Other = 7).

Students' Socioeconomic Status (Unspecified = 0; Lower = 1; Lower middle = 2; Middle = 3; Upper middle = 4; Upper = 5; Mixed = 6).

Country (Unspecified = 0; USA = 1; Canada = 2; Mexico/Latin America = 3; Europe = 4; Asia = 5: South America = 6; Cross-Cultural = 7; Other = 8).

Geographical Region in USA (Northeast = 1; Southeast = 2; Midwest = 3; South Central = 4; Southwest = 5; Northwest = 6; Mixed = 7; Other = 8).

School Type (Unspecified = 0; Public = 1; Private = 2; Special school = 3; Mixed = 4; Other = 5).

Community Type (Unspecified = 0; Urban = 1; Rural = 2; Suburban = 3; Mixed = 4; Other = 5).

Content Area (Content area where technology is used. Unspecified = 0; Reading = 2; Mathematics = 3; Social studies = 4; Science = 5; Reading and math = 6; Language arts = 7; Foreign language = 8; Mixed = 9; Other = 10).

Quality of Study Indicators

Method of Observation of Independent Variable (i.e., technology use. Unspecified = 0; Systematic observation = 1; Informal observation = 2; Student survey or interview = 3; Teacher survey or interview = 3; Administrator survey or interview = 4; Computer logs = 5; Multiple methods = 6; Other = 7).

Pretest Equivalency (Has the initial differences between the two groups been accounted for? Unspecified = 0; Statistical Control (e.g., ANCOVA, regression) = 1; Random Assignment = 2; Statistical Control and Random Assignment = 3; Gain Scores = 4; Matching = 5; Other = 6).

Reported Reliability of Measures (Unspecified = 00; Actual reliability statistic (e.g., 70 or 83).

Manner in Which Outcome Scores Are Reported (Unspecified = 0; Standard scores = 1; Raw scores = 2; Percentile ranks = 3; Gain scores = 4; Other = 5).

Duration of Study (Unspecified = 00; List the number of months that the implementation of the technology occurred).

Cognitive Outcomes (Unspecified = 0; Testing company standardized achievement test = 1; Federal/national standardized test = 2; State-level achievement test = 3; District-level achievement test = 4; School-level test = 5; Grade-level test = 6; Teacher-made test = 7; Researcher-developed test = 8; Authentic assessment = 9; Creativity test = 10; Higher-level thinking test = 11; Other = 12).

Affective Outcomes (Unspecified = 0; Student attitudes toward computers, content areas, anxiety, or instruction = 1; Academic self-concept or motivation = 2; Other = 3).

Behavioral Outcomes (Unspecified = 0; Student time-on-task = 1; student perseverance = 2; Tasks attempted = 3; Tasks completed = 3; Success rate = 4; Positive peer interaction = 5; Interactivity with computers = 6; Other = 7).

Effect Size Coefficient (actual coefficient)

Statistics (Statistics used in determining effect size; Means = 1; t-value = 2; F-value = 3; Chi-square = 4; Other = 5).

Weight (One divided by the actual number of comparisons in the study, e.g., 3 comparisons = 1/3 or .333).

Sources of Invalidity

Type of Design (Quasi-experimental/nonrandomized one group pretest-posttest = 1; Nonrandomized static-group comparison = 2; Nonrandomized pre-post control group = 3; Time series = 4; Randomized posttest-only control group = 5; Randomized pre-post control group = 6; Other = 7).

History (Have specific events occurred between the first and second measurement in addition to the experimental variable? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Maturation (Are there processes within the participants operating as a function of the passage of time [e.g., growing older, more tired] that might account for changes in the dependent measure? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Testing (Is there an effect of taking a test upon the scores of a second testing? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Instrumentation (Do changes in calibration or observers' scores produce changes in the obtained measurement? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Statistical Regression (Have groups been selected on the basis of their extreme scores? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Selection Bias (Have biases resulted in the differential selection of comparison groups? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Mortality (Has there been a differential loss of participants from the experimental and control groups? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Selection-Maturation Interaction (Is there an interaction between extraneous factors such as history, maturation, or testing and the specific selection differences that distinguish the experimental and control groups? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Reactive or Interaction Effect of Testing (Does pretesting influence the participants' responsiveness to the experimental variable, making the results for a pretested population unrepresentative of the effects of the experimental variable for the unpretested universe from which the participants were selected? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Interaction of Selection Biases and Treatment (Are there selective factors upon which sampling was based which interact differentially with the experimental variable? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Reactive Effects of Experimental Arrangements (Are there effects of the experimental setting that would preclude generalizing about the effect of the experimental variable upon persons being exposed to it in nonexperimental settings? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Multiple-Treatment Interference (Are there nonerasable effects of previous treatments applied to the same participants? Adequately controlled by design = 1; Definite weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 4).

Statistical Power (Is the sample size large enough to reject the null hypothesis at a given level of probability, or are the estimate coefficients within reasonably small margins of error? [a sample > 60 for groups such as classes, schools, or districts; a sample >100 for individuals]. Probable threat [< 60 for groups or < 100 for individuals as the unit of analysis] = 1; Adequately minimized [> 60 for groups; > 100 for individuals] = 2).

Technology Characteristics

Type of Technology (Unspecified = 0; PCs = 1; Laptops = 2; Networked labs = 3; HP calculators = 4; Multimedia = 5; Other = 6).

Software (Unspecified = 0; Tutorial = 1; Drill-and-practice = 2; Exploratory environment [e.g., simulations, microworlds, hypermedia, and hypertext] = 3; Tools for other tasks [e.g., word processor for writing, e-mail, or computer-conference for course assignments] = 4; Programming language = 5; Other = 6).

Technology Resources/Support Available (Unspecified = 0; No resources = 1; Minimal resources = 2; Adequate resources = 3; Ample resources = 4; Other = 5).

Role/Focus of Technology (Unspecified = 0; Productivity = 1; Delivery system [e.g., ILS] = 2; Resource [e.g., Internet] = 3; Other = 4).

Quantity of Technology (Unspecified = 0; Few [< 3 per classroom] = 1; Average [4-8 per classroom] = 2; Ample [> 9 per classroom] = 3; Other = 4).

Number of Computer Sessions (Unspecified = 0; List number of sessions [e.g., 12]).

Duration of Computer Sessions (Unspecified = 0; List number of average minutes per sessions [e.g., 40]).

Teachers' Experience with Technology (Unspecified = 0; None = 1; Minimal experience = 2; Average = 3; Experienced = 4; Very experienced = 5).

Students' Experience with Technology (Unspecified = 0; None = 1; Minimal experience = 2; Average = 3; Experienced = 4; Very experienced = 5).

Teacher Training in Technology (Unspecified = 0; List hours of training (e.g., 15).

Feedback and Assessment Practices (Unspecified = 0; No feedback = 1; Minimal feedback = 2; Elaborate feedback = 3; Other = 4).

Learning Responsibility (Unspecified = 0; Student controlled = 1; Teacher directed = 2; System directed = 3; Mixed = 4; Other = 5).

Task Difficulty (Unspecified = 0; Difficult = 1; Moderately difficult = 2; Not difficult = 3; Mixed levels of difficulty = 4; Other = 5).

Type of Learning Task (Unspecified = 0; Basic skills/factual learning = 1; Problem solving = 2; Inquiry/investigation = 3; Project-based = 4; Mixed types = 5; Other = 6).

Type of Technology Program (Unspecified = 0; Basic skills/factual learning = 1; Problem solving = 2; Inquiry = 3; Mixed types = 4; Other = 5).

Pattern of Student Computer Use (Unspecified = 0; Teacher use only = 1; Presentation station = 2; One student per computer = 3; Two students per computer = 4; 3-5 students per computer = 4; > 5 students per computer = 6; Mixed pattern = 7; Other = 8).

Percentage of Students Using Computers (Unspecified = 0; > 10% = 1; 10-25% = 2; 26-50% = 3; 51-75% = 4; 76-90% = 5; > 90% = 6).

Objectives of Computer Use (Unspecified = 0; Remediation of skills not learned = 1; Expressing themselves in writing = 2; Communicating electronically with other people = 3; Finding out about ideas and information = 4; Analyzing information = 5; Presenting information to an audience = 6; Improving computer skills = 7; Learning to work collaboratively = 8; Learning to work independently = 9; Multiple Objectives = 10; Other = 11).

Instructional/Teaching Characteristics

Joint Productive Activity/Collaboration (e.g., Designs instructional activities requiring student collaboration to accomplish a joint product; monitors and supports students collaboration in positive ways. No evidence = 1; Some evidence = 2; Extensive evidence = 3).

Language and Literacy Development (e.g., Connects student language with literacy and content-area knowledge through speaking, listening, reading, and writing activities; encourages students to use content vocabulary to express their understanding. No evidence = 1; Some evidence = 2; Extensive evidence = 3).

Contextualization/Making Meaning (e.g., Begins activities with what students already know from home, community, and school; encourages students to use content vocabulary to express their understanding. No evidence = 1; Some evidence = 2; Extensive evidence = 3).

Challenging Activities (e.g., Designs instructional tasks that advance students' understanding to more complex levels. Assures that students—for each instructional topic—see the whole picture as a basis for understanding the parts. No evidence = 1; Some evidence = 2; Extensive evidence = 3).

Instructional Conversation (e,g, Arranges the classroom to accommodate conversational between the teacher and a small group of students on a regular and frequent basis. Guides conversation to include students' views, judgments, and rationales using text evidence and other substantive support. No evidence = 1; Some evidence = 2; Extensive evidence = 3).

Setting (Unspecified = 0; Classroom = 1; Networked lab within class = 2; Computer lab in school = 3; Other = 4).

Mode of Instruction (Unspecified = 0; Whole-group instruction = 1; Paired = 2; Small-group instruction [3-5 members] = 3; Individualized = 4; Mixed = 5; Other = 6).

Role of Teacher (Unspecified = 0; Deliverer of knowledge = 1; Facilitator of groups/student learning = 2; Modeling processes [e.g., problem solving] = 3; Mixed = 4; Other = 5).

Teacher Qualifications (Unspecified = 0; Alternatively certified or provisional certificate = 1; Certified in content area = 2; Not certified in content area = 3; Other = 4).

Policy

Level (Unspecified = 0; School = 1; District = 2; State = 3; Federal = 4; Other = 5).

Focus (Unspecified = 0; Reducing achievement gaps = 1; Increased use of technology = 2; Increased specific type of use = 3; Other = 4).

Previous | Table of Contents