|
Outline
-----------------------------------------------------------------------------------
I.
Introduction
-
Brink and Wood presents three basic levels of
design- experimental, survey and exploratory-descriptive, each
of which contains two sub levels namely experimental designs:
experimental and quasi-experimental; Survey designs:
comparative and correlational; exploratory-descriptive
designs: descriptive and exploratory.
-
Ideally, at successively higher levels of
design, the degree of control and the level of knowledge about
the variables increases.
-
The level of knowledge about pertinent
variables should be the greatest at the level of the
experiment and is expected to be most limited at the
exploratory level.
-
The choice of the appropriate design is based
on the current level of knowledge about the research topic.
The current level of knowledge and theory about a topic must
be thoroughly examined prior to entering into an experimental
design.
-
Two methods of examining existing theory on any
well studied topic is either to do a conceptual mapping of the
literature to determine what is missing, what is conflicting
and what is causal or to perform meta analysis of the
published literature to determine if there are conclusions
that can be drawn and applied or whether further work is
needed to explain contradictions or omissions.
-
When a causal relationship between variables
can be both predicted and ethically tested, experimental or
quasi experimental designs are warranted.
-
Experiments broadly defined are tests
that involve at least one treatment (independent variable),
units (subjects) to be analyzed by assignment or non
assignment to a treatment and a comparison for inferring
effects that may be attributed to the treatment.
II. Essential characteristics of
Experimental designs
To test a relationship and most confidently
infer this result, experimental designs must be characterized
by three essential elements: randomization, manipulation and
control.
A. Randomization
Randomization refers
to participants being assigned by chance to either receive or
not receive the treatment condition or intervention. A number
of procedures exist for assigning individuals to groups such
as coin toss, a random numbers table or computerized random
number generators. The key characteristic of all these
procedures is that each participant has an equal and known
probability of being assigned to either the control or the
experimental group. Randomization helps eliminate bias by
spreading variability due to extraneous variables equally
across the groups under study. The advantage of assigning
participants to groups in a random manner is that this should
result in the group’s initially being similar to one another
prior to the intervention. Random assignment to condition does
not guarantee that the two groups will be similar to one
another. Based on sampling theory, significant difference(p<0.05)
between the two groups will occur in 1:20 cases of assigning
participants to groups. The sample size should be large enough
or else researcher may wish to consider some additional
methods of distributing important variables such as matching
or use of more homogeneous population. In some research, it is
necessary to randomly assign treatment conditions, to units
other than the individual participants. Randomization may be
cluster, stratified, fixed or by random assignment.
B. Manipulation:
Manipulation
is
the process of maneuvering the independent variable so that
its effect on the dependent variable can be observed. The
causative variable must be amenable to manipulation by the
investigator, i.e. the researcher does something to subjects
in the experimental condition. It is essential that
researchers conduct manipulation checks to see whether the
manipulation had its intended effect or perhaps resulted in an
unintended effect that could compromise the validity of an
experiment. In working to avoid unintended effects researchers
should be cautious of the use of reactive measures which can
influence participants’ responses to the dependent variable.
Even though researcher does not actively manipulate the
control group, it is important that he or she be aware of what
may be happening to them Control group should experience all
the same things as participants in the experimental group,
except the independent variable. Ethical considerations,
organizational policy or some variables like attitudes, age,
disease etc which cannot be manipulated may disallow
manipulation. The ability of the researcher to manipulate the
independent variable is a major source of control in
experimental studies.
C. Control:
Cook and Campbell identify three uses of the term control in
relation to research designs, all of which involve elimination
of threats to valid inference namely the researchers control
over the research environment; control over the experimental
variable; the ability to identify and rule out threats to
internal validity. The last type of control is typically
achieved through the use of a comparison or control group and
through attention to sources of variance. In many nursing
studies control groups receive the usual or traditional
methods of care rather than no treatment against which effects
of the experimental intervention are measured.
Kerlinger
discusses the merits of experimental designs in terms of their
ability to control variance or to take into account factors
that may contribute to differences in the dependent variable.
To provide valid answers to research questions, three kinds
of variance must be considered: systematic or experimental
variance which refers to the systematic effect of independent
variable on the dependent variable and is a type of variance
which should be enhanced; extraneous variance which refers to
the effects of extraneous variables on the dependent
variables, which is to be controlled by the design by building
the extraneous variable into the design as an independent
variable, eliminating or holding constant an extraneous
variable by selecting participants as homogeneous as possible
on that variable, matching participants and using statistical
control; error variance which refers to the variability of
measures due to random fluctuations including errors of
measurement, which can be controlled by standardizing the
instrument/ measurement conditions (keeping the time of the
day, place instructions, personnel constant) or using
sensitive, reliable instruments. Internal validity is the
primary objective of experimental methodology
III.
Threats to internal and external validity:
A. Internal Validity:
Cook and
Campbell identified 12 types of extraneous variables that if
left uncontrolled my produce effects that the researcher
could mistake for the effect of the independent variable.
These are
1. History:
refers
to the events or circumstances other than the introduction
of the treatment variable that occur coincident with the
time interval between the pretest and posttest
measurements. E.g. An event that received much media
coverage and attention and that may have influenced
responses in the target population. Researchers must be
attuned to changes at one study site that could make
comparison of outcome variables across several hospitals
biased.
2. Maturation:
refers
to changes within the study participants themselves that
occur overtime and that are not related to any specific
event such as tiring, gaining weight, becoming more
knowledgeable. It becomes difficult for the researcher to
determine whether changes observed over time can be
attributed to the independent variable, to maturational
changes in the participants or perhaps to an interaction
effect between the treatment and maturational changes.
3. Testing:
refers
to the learning that results from being tested at time 1
that affects responses to the test at time 2, regardless of
the introduction of the treatment variable. Thus the process
of measuring itself can introduce a threat to internal
validity, especially when reactive measures are used.
Participants may recall information on the pretest or be
sensitized to aspects of the experiment, especially when
tests are unusual or memorable.
4. Experimental mortality:
refers to nonequivalent attrition of study participants from
the experimental and control groups that renders meaningful
comparisons between the groups difficult. Random attrition
is a far less serious threat than when participants are
systematically dropping out of one treatment condition more
than another. When the latter occurs researcher should
attempt to evaluate a small random sample of dropouts to
determine why they left the study.
5. Instrumentation:
refers to changes that occur in the measurement instruments,
observers or raters that potentially produce changes in the
dependent variable measurements. This threat of validity is
most pronounced in studies using repeated measures designs
6. Statistical regression:
refers to movement of mean scores from Time 1 to time 2 that
most often results when study participants are selected on
the basis of scores that are at the extremes of the
distribution.
7. Selection bias:
refers
to the selection of participants on a nonrandom basis that
may produce differences in the experimental and comparison
group participants with regard to the criterion measurement
irrespective of the differential exposure to the treatment.
8. Interactions with selection:
means
that a number of the previously described threats to
validity can interact with selection causing spurious
treatment effects e.g. selection-maturation,
selection-history, selection-instrumentation.
9. Diffusion or imitation of treatments:
refers to the introduction of a treatment that involves
information when the experimental and control group
participants may be able to interact with one another,
directly or indirectly and learn about information intended
for others.
10. Compensatory equalization of
treatments:
refers to the use of an experimental treatment that has
actual or potential value to participants in cases in which
authorities or participants may be unwilling to tolerate an
imposed inequity in the distribution of the treatment.
11.
Compensatory rivalry by respondents receiving less desirable
treatments:
refers to the assignments of study participants to the
experimental and control groups in which control group
participants are disadvantaged by the absence of the
treatment in contrast to experimental participants and thus
are motivated to compete for equity. Compensatory
equalization is mainly a response by administrators and
compensatory rivalry is a response by participants.
B. External Validity:
Cook and Campbell identified six main factors, which if
controlled the researcher can achieve generalizability by
replicating the study with different participants, in
different settings and at different times. They are:
1. Interaction of selection and treatment:
refers to the effects obtained that are applicable only to
the specific individuals who participated in the study.
2. Interaction of setting and treatment:
refers to the effects obtained that are applicable only to
the specific setting in which the experiment is conducted.
The milieus of the settings may vary widely, with some being
more innovative more pleasant and more competitive. The
question is whether or not results obtained in one setting
can be generalized to other settings that because of their
particular environments would be different from the original
settings.
3. Interaction of history and treatment:
refers
to the effects obtained that are applicable only to the
specific time period within which the study is conducted.
Unusual occurrences that coincide with a study period can
make the extrapolation of results to other periods of time
questionable. Although the researcher can attempt to plan in
a way that avoids obvious unusual occurrences, it is often
impossible to avoid happenings that could make the findings
unique to the study time period. Replication of study at
different times is the logical approach to counteracting the
interaction effect of history.
4. Reaction or interaction effect of pretesting:
means
that following exposure to pretest, the participants no
longer remain representative of the target population which
has not been pretested. Thus the findings cannot be
generalized to the target population This effect occurs
because the nature of the pretest makes participants aware
of certain issues or events of which they would not
otherwise be aware, causing them to respond to the treatment
in a unique way.
5. Reactive effect of experimental procedures:
is the
effect produced by the procedures of the experiment that
make the participants who are exposed to these procedures no
longer representative of the target population - “Howthorne
effect”.
6. Multiple treatment interference:
refers
to effects produced by multiple exposures of participants to
a treatment so that the results may be generalizable only to
individuals who also receive the same multiple exposures to
the treatment in the same sequence.
IV. Types
of experimental designs:
There
are of two major types of experimental designs, true
experiments and quasi experiments. True experiments include
the random assignment of units to comparison groups for
inferring a change that has been caused by treatment. Random
assignment is an essential component of true experiments that
is designed to achieve comparability of comparison groups.
Quasi experiments have a treatment, outcomes and units to be
analyzed, but random assignment of units to comparison groups
is not included for determining the groups of units to be
compared. An important assumption underlying all true
experimental designs is that equivalence of groups is
maintained throughout the course of the experiment and is not
compromised by things such as differential attritions – i.e.
variable dropout rates may make experimental and control
subjects different on critical factors at the time of the
analysis. If experimental and control groups become
nonequivalent, the design then becomes quasi experimental.
Refer to Appendix A
V. True
experimental Design/ Randomized clinical trials/ controlled
trials:
A. Overview of true experimental design
The experimental level of research is designed to test theory
in laboratory settings or in controlled clinical trials, thus
the purpose is to test theory. The greater the degree of
control in the research setting, the greater the confidence,
that the research findings are accurate. The experimental
design the most controlled of all research designs entails
manipulation of the independent variables and requires control
of all intervening variables. On the basis of theory developed
from previous research, each step in the experiment requires a
predictive hypothesis regarding the effect of the independent
on the dependent variables.
Cause and effect is always predicted based on
theory, thus predictive hypotheses are written at the
beginning of the study stating the precise nature of both the
manipulation of the independent variable and its effects on
the independent variables. All assumptions are spelled out or
are either verified by previous research or tested by the
current research. All logical steps between cause and effect
are specified. Although causal laws can never be proved
researchers in experimental designs should be aware of three
criteria for causality: a temporal relationship – cause must
precede the effect in time; an empirical relationship - there
must be an evidence that the independent variable and the
dependent variable are associated; a spurious relationship –
the relationship cannot be explained by the influence of a
third variable. The key issue of the design is internal
validity or the assumptions that changes in the dependent
variable are actually due to the independent variable.
Experimental and control groups are created by
random assignment. Ethical concerns for the protection of
human and animal subjects are the most restrictive.
Experimenters more than any kind of researchers, are required
to establish that all subjects rights are protected to the
greatest degree possible and potential harmful effects are
counterbalanced by potential benefits. Data collection is
quantitative and prospective. Data analysis is designed to
discriminate between and among experimental and control
groups. Although experimental studies are the most directly
applicable to nursing practice because of their controlled
samples, they are the least widely generalizable in and of
themselves. This is simply due to the controlled narrowness of
the study and sample. Experiments are deigned to be repeated
on many samples with small variations in the independent
variable over time.
A single experiment on a single small sample
adds to the test of a part of the theory but not of the entire
theory. Many experiments with different samples may be
required to increase generalizability to the point where
findings can be widely applied in practice. Thus the true
experiment is regarded as the cornerstone of scientific
research, the most powerful strategy for testing causal
hypothesis and achieving the four criteria namely:
establishing causal relationships; manipulating independent
variable; measuring the impact of independent variable on the
dependent variable; minimizing or accounting for the effects
of factors other than the independent variable on the
dependent variable.
B. Types of true experimental designs:
A number of different experimental designs meet the criteria
for true experiments. These include but are not limited to the
classical experimental designs, the factorial design, the
multiple treatment groups-repeated measures design and the
Solomon four group design.
1. The classical experimental design: Kothari
describes them as formal designs either with or
without a control group namely before and after without
control design, before and after with control design.
Similarly post test only design with control grou.p
a. pretest-posttest design/ before-after design:
All experimental groups are variations on the basic classical
experimental designs which consist of two groups, an
experimental and control group and two variables, an
independent and dependent variable. Units to be analyzed are
randomly assigned to each of the experimental and control
groups. Units in the experimental groups receive the
independent variable that the investigator has manipulated and
participants in the control group do not receive the
independent variable treatment. Pretest and post test measures
are taken on the independent variables and the control group
participants are measured at the same time as the experimental
group although no planned change or manipulation has taken
place with regard to the independent variable in the control
group. Researchers use this design when they are interested in
assessing the change from the pretest to the posttest as a
result of a treatment or intervention.
R O1 X O2
R O1 O2
(R:
Random assignment, X: Treatment, O: Observation)
b. Post test only design:
One group receives a treatment whereas the other group
receives no treatment and serves as control. The key
difference in the post test only design is that, neither group
are pretested and only at the end of the study are both groups
measured on the dependent variable. Some researchers favor
this design because they are concerned that the pretest
measures will sensitize participants or that a learning effect
might take place that influences individuals’ performance on
the posttest.
R
X O2
R
O2
(R: Random assignment, X: Treatment, O: Observation)
c. N=1 True experiments or single subject
design: The sample is one subject, who is exposed to
two or more treatments on various occasion. It may involve
before and after designs. The design gives the researcher
opportunity to focus on an individual and therefore pay more
attention to details. It is particularly suitable to the
principle of patient centered care, since the interaction
between the individual and the treatment is unique, although
lessons learned can be applied to other cases as well. But
findings can’t be generalized.
2. Factorial designs:
Classical experimental designs allow the manipulation of a
single variable at a time, holding all other conditions
constant. Factorial designs permit the manipulation of more
than one independent variable at a time. Furthermore,
interaction effects between variables are revealed and the
simultaneous testing of multiple hypotheses is thus allowed.
The researcher can study the effects of each of the variables (known
as main effects) and the interactions between them or their
joint effects on the independent variable.
Though commonly used design is 2 x 2 factorial design,
experiments can be conducted with any number of categories ad
independent variables. Each factor must have two or more
levels thus when new factors are added the analysis becomes
more complex. E.g. 4 X 5 design means that one of the factors
has four categories and the other has five. These designs can
also be extended to have more than two factors as in 3 x 2 x 4
factorial design, which means that there are three factors
consisting of three, two and four categories respectively.
Factorial designs are also referred to as “levels of
treatment” designs. When one of the factors cannot be
manipulated, such as sex or race it is referred to as blocking
variable. The design that incorporates the blocking variable
is known as the randomized block design. Stratified
randomization is often used in this case to ensure that
approximately equal numbers of participants within categories
are randomly assigned to the various groups.
3. Multiple Treatment Groups- repeated measures
design:
This design uses several experimental groups each of which
receives a different treatment. Control is achieved through
comparison among groups and from measures taken on the
dependent variable for all groups at time 1 and time 2. It is
always not true that a control group will be used that has no
treatment at all. There can also be more than one pretest and
posttest measure for the dependent variable for all groups.
4. Solomon Four Group Design:
This is a rigorous design by controlling effects on the
dependent variable that may be due to factors other than the
independent variable. It is essentially a combination of
pretest-posttest and posttest only designs. This design
requires a large group of homogeneous subjects to make up the
four groups, thus is used less often than other designs. It is
difficult to introduce the treatment simultaneously for all
groups, in order to avoid extraneous temporal effects and
statistical analysis complex. The design consists of four
groups of which two are control and two are experimental.
Pretest and posttest are carried out with one control and one
experimental group only and post tests only for the other
control and experimental group.
|
Exp. Group 1 |
O1 |
X |
O2 |
|
Control Group1 |
O2 |
- |
O4 |
|
Exp. Group 2 |
-- |
X |
O5 |
|
Control Group2 |
-- |
- |
O6 |
C. Strengths and weaknesses of the design:
1. Strengths:
Ability to diminish bias; permit the researcher to maximize
systematic variance and to control the extraneous and error
variance; control the threats to validity; degree of
confidence they allow the researcher in inferring causal
relationships, because of high internal validity
2. Limitations:
the number of potentially interesting research variables that
are not within researcher’s purview to manipulate; ethical
concerns; organizational prohibitions; human characteristics
often interfere with the researcher’s ability to control a
number of variables experimentally; sample representativeness/
homogeneous sample; many cases population is unknown; requires
many replications to achieve generalizations; artificiality in
laboratory testing; Hawthorne effect; many of the phenomena to
be tested lack theoretical background which could be either
exploratory or descriptive studies than experimental; issues
related to the outcome measure like relevant outcome
variables, validity, reliability and systematic bias.
VI. Quasi experimental designs:
A.
Overview of Quasi experimental designs:
this design
is
distinguished from the true experiment by the lack of random
assignment of participants to groups. This lack of control
limits confidence in the internal validity of the study.
Otherwise the design is exactly same as the experimental
design. Data are predominantly quantitative, data analysis
distinguishes between and among treatments and among treatment
groups. Many field experiments actually use a quasi
experimental design when the assignment of participants to the
various experimental and control groups cannot be controlled
by the researcher. E.g. The nurse may want to use an inpatient
hospital population in an experiment but physicians control
who is admitted to the hospital. A nurse therefore may not
have control over extraneous or intervening variables in the
available population.
The researcher is often not allowed control of
the treatment variable and cannot achieve randomization
because of ethical considerations, institutional policies or
other situational factors. In such circumstances, the
researcher chooses quasi experimental studies. Quasi
experiments do not have equivalence created by random
assignment or that do not have control groups for comparison,
rather comparisons are made with nonequivalent groups or with
periodic measurement of the same group that may be different
due to a number of variables extraneous to the causal variable
of interest. Refer to Appendix B
B. Types of Quasi experimental designs:
| 