IMPACTS OF SACC PROGRAMS
Preventing Problem Behaviors and Raising Academic Performance in the Nation's Youth: The Impacts of 64 School Age Child Care Programs in 15 States
Supported by the Cooperative Extension Service Youth-at-Risk Initiative
| Dave Riley University of Wisconsin |
Sharon Junge University of California |
| Jill Steinberg University of Wisconsin |
Ina McClain University of Missouri |
| Chris Todd University of Illinois |
Copyright/Access Information
ACKNOWLEDGMENTS
EXECUTIVE SUMMARY
…Big impacts, both in lives made better and in public monies
saved…
As part of its Youth-at-Risk initiative, the Cooperative Extension
Service has started or assisted school-age child care (SACC) programs
in high-risk communities across the nation. The intent of these
programs is not just to keep children out of trouble, but to actively
promote their positive development.
To determine if these programs are having their desired impacts,
we surveyed the SACC staff from 64 programs serving 2,664 children
in 15 states. Classroom teachers and school principals associated
with these children were also surveyed, providing three independent
estimates of program-related changes in children.
The results confirm that Extension – assisted SACC programs are
having multiple, positive impacts on children, their families,
and their schools. These impacts range from improved social skills,
to reduced problem behaviors, to increased academic performance.
For example:
Both principals and classroom teachers say that the SACC program
has caused 34% of the children to become more cooperative with
adults.
Classroom teachers say that over one-third (37%) of SACC program
children “have learned to handle conflicts by talking or
negotiating more often, instead of just hitting or fighting.”
Over one-third of principals (35%) reported that the SACC program
had “led to decreased vandalism in the school.”
One-third of the SACC program children were getting better grades
due to their participation in the program, according to classroom
teachers (33%) and principals (36%).
Principals and teachers were in agreement that about 16% of the
SACC program children had “avoided being retained in grade
for next year because of their involvement in the program.”
This created a savings of over one million dollars for the school
districts.
These are big impacts, both in lives made better and in public
monies saved.
OUR TASK
A study of program impacts in 15 states
Unsupervised children with little to do after school have been
a concern of educators and policy makers for more than a decade.
When a lack of supervision combines with other risk factors (such
as substance abuse at home, low self-esteem, learning difficulties,
etc.) we know that the likelihood of poor developmental outcomes
can increase greatly.
In an effort to prevent these problems, the Cooperative Extension
Service has helped establish or support high-quality programs,
providing attention from caring adults after school in high-risk
communities across the nation. Compared to traditional 4-H clubs,
children have roughly ten times as many contact hours in school-age
child care (SACC) programs, so the potential for Extension to
positively affect the development of youth is greatly multiplied
by operating through SACC programs.
Have these programs had positive impacts on children, impacts
that are dramatic enough for third-party observers such as classroom
teachers and school principals to notice? To determine if they
have, we simultaneously surveyed staff of school-age child care
(SACC) programs receiving funding through the Extension Service's
Youth-at-Risk national initiative, as well as the classroom teachers
and principals of the elementary schools serving children in these
SACC programs.
The Evaluation Study
This study taps the expertise of professionals who observe
the children daily.
The impacts of 64 Extensionsupported SACC programs upon 2,525
children (ages 4 to 15) was investigated by use of a “key
informant” survey of three kinds of observers: the SACC lead
teachers, the classroom teachers associated with those children,
and the school principals. The response rates to our questionnaire
survey were 91% for SACC staff, 45% for teachers, and 51% for
principals. The distribution of the sample across 15 states is
shown in Table 1.
By surveying Principals and Teachers, this study taps the expertise
of professionals who observe the children daily, and who can see
how they have changed over the course of a year as compared to
children not in the SACC programs. This method is also efficient,
with each respondent reporting on several children. But this method
also has weaknesses. While 43 principals returned our questionnaires,
42 did not, and we need to worry, therefore, if the nonrespondents
might have painted a different picture of the program's impacts.
For reasons such as this, we must exercise caution and remain
tentative in stating the conclusions of the study. (See “Appendix:
Data Quality” at the end of this report.)
The SACC programs are part of Extension's federally funded “YouthatRisk”
initiative. The communities in which these SACC sites are located
were selected by the state Extension Services as especially likely
to benefit from the programs. Of the 43 elementary schools for
which we have data from principals, 79% received Chapter 1 funding.
TABLE 1: SAMPLE
Number of Respondents
| California |
26 |
13 |
150 |
| Connecticut |
4 |
1 |
21 |
| Georgia |
2 |
4 |
30 |
| Illinois |
3 |
3 |
13 |
| Iowa |
1 |
1 |
5 |
| Kentucky |
3 |
2 |
18 |
| Mississippi |
1 |
1 |
2 |
| Missouri |
1 |
1 |
1 |
| New Hampshire |
2 |
2 |
14 |
| New Jersey |
1 |
1 |
17 |
| New York |
2 |
2 |
0 |
| North Carolina |
5 |
3 |
36 |
| Ohio |
1 |
3 |
8 |
| Oklahoma |
5 |
4 |
19 |
| South Dakota |
1 |
2 |
8 |
| Overall number |
58 |
43 |
342 |
| Response rate |
91% |
51% |
45% |
How many of the children in these SACC programs
might be considered “at risk” in their development?
The school principals told us that 39% of the program children
were “working below grade level in terms of their academic
skills at the beginning of the year.” The SACC staff gave
a smaller estimate (28%), while the classroom teachers offered
a harsher assessment, saying that 42% were behind academically.
In the area of social skills, principals again gave estimates
midway between those of SACC staff and classroom teachers. They
reported 35% of children were performing below average, with SACC
staff reporting 29% and classroom teachers 41%. These estimates
are not too different, and suggest that about a third of the SACC
program children were having difficulties in school, academically
or in social behavior or both. See Table 2.
TABLE 2
Percentage of SACC program children judged below grade level,
according to 3 observers
| Below grade level in ACADEMIC skills | |||
| Below average in SOCIAL skills |
All respondents – the SACC Staff, the Classroom
Teachers, and the Principals – were asked about changes in the
children which they could say were specifically due to the effects
of the SACC program. Some respondents wrote that they found this
attribution of cause and effect to be difficult: Since changes
in children are usually due to many factors working together,
it was difficult to give clear credit to the SACC program. Based
on these comments, we suggest our method elicits conservative
estimates of actual program impacts.
Where it was logical to do so, the staff, teachers, and principals
were asked the same questions, but some questions were asked only
of those respondents who were in a position to answer accurately.
For example, we asked school teachers, but not SACC staff, about
changes in children's grades and homework.
Were the SACC programs having clear, positive impacts upon the
children? According to all three types of informants, the answer
was “yes.” We will summarize the impacts in three areas:
increasing pro-social behaviors, decreasing problem behaviors,
and raising academic performance. Following that, we will describe
the characteristics of the programs themselves, and their connections
to the Extension Service.
INCREASED PRO-SOCIAL BEHAVIORS
Rejection by peers in childhood predicts lifelong negative
consequence.
We expected these programs might have special benefits for
children who were low in social skills. Like a good 4-H program,
a good SACC program offers opportunities for recreation-based,
multi-age activities in the company of a caring adult. SACC programs,
however, have about 10 times more contact hours with children
than traditional 4-H programs, so the potential impacts are much
greater.
SHY AND REJECTED CHILDREN
Two areas of social skill were of special concern: shy children
and rejected children. Rejection by peers in childhood is important,
not just in current heartache, but also because it predicts lifelong
negative consequences in social relations and personality development.
When asked whether any children “who were shy have become
more outgoing, more skilled at joining group activities”
because of the SACC program, 97% of SACC staff answered yes, and
they named 20% of the children in this category. But since SACC
staff were reporting on their own programs, we might reasonably
expect a tendency to exaggerate the impacts of the program. That
is why the school teachers were asked the identical question.
The classroom teachers reported an even greater percentage of
children, 33%, who had become less shy and more socially skilled
“because of their involvement in the SACC program.”
TABLE 3: INCREASED PRO-SOCIAL BEHAVIORS
Percentage of children increasing their Pro-Social Behaviors,
according to 3 observers
| 1. Shy child became more outgoing. | |||
| 2. Rejected child learned to make friends. | |||
| 3. More cooperative with adults. | |||
| 4. Increased responsibility for planning SACC program. |
|||
| 5. Developed new interests. |
*Note: Question not asked. Respondents were asked about
only those child behavior they were likely to have directly observed.
Here are three examples of such a child, described
by SACC staff and classroom teachers:
“…When she came to the program, I observed that she
was shy and withdrawn. She didn't like to participate. I visited
her school to talk to her teacher to find the best way to help
her. She was also withdrawn at school. In order to help her,
I had to build her confidence in herself. Once her self-esteem
was restored, she was able to become more outgoing.”SACC Staff in New Jersey
“…One child was shy and a little backward. She enjoyed
the attention she received in the program. It was a positive
influence in her life.”Classroom Teacher in Ohio
“I would say socially, both children have had positive changes
throughout the school year. Both children were quiet and withdrawn
from the other students in class. I feel the SACC program gave
these two children an opportunity for more socialization with
other children of varied ages, and as a result, they both became
well-liked in class, participated more in class instruction,
and became more confident in themselves.”Classroom Teacher in Connecticut
Did the SACC programs provide opportunities for rejected children
to learn how to make friends? Most SACC staff saw this happening,
with positive impacts on 17% of the children they served. The
classroom teachers confirmed this estimate, giving the SACC programs
credit for positive impacts in this area on 23% of their students
from the SACC program.
“I am very glad [student] is in a program like [this
one]. It may not show right away, but the skills he learns there
will help him in the future, I'm positive. I've seen a willingness
to help others, which is a great help. It gets him more accepted
by others who would otherwise not like him due to his behavior
problems.”Classroom Teacher in New Hampshire
We have a young boy in our program that has very poor social
skills. He does not interact positively with other children.
We have been able to work on this child's social skills through
game playing. Through collaboration with his teacher and parents
as well as SACC staff we have tried to bang out the positive
qualities in this child. With a gradual increase in self-esteem
levels, confidence, and morale he now seems a little happier!”SACC Staff in Connecticut
“She has gained friendship, learning to be a friend,
and have a friend. She has learned to mature into a nice young
lady and learned to deal with not always being a winner. She
does a lot more smiling and less frowning or crying. ”Classroom Teacher in Oklahoma
RESPECTING THE AUTHORITY OF ADULTS
Learning to work well in a group of one's peers is one aspect
of pro-social behavior. Another which we asked about is learning
to get along with the adults in the program. SACC programs, we
suspected, have the potential to teach young people to live within
rules, and to respect the reasonable authority of adults. This
kind of learning, if it has not taken place already within the
home, is crucial in the years of middle childhood, before young
people reach the wider world of adolescence.
When asked if any children “have become more cooperative
with adults now, more willing to follow the directions and rules
of adults” because of the SACC program, all of the SACC staff
answered “yes” and named 25% of the children. But did
this improvement generalize across settings such that teachers
and principals could notice a change as well? The answer is “yes.”
The classroom teachers reported this improvement in 34% of the
children, and the principals in 34%, identical estimates.
“Have seen positive changes in student due to child being
able to have supervision after school. The child has to answer
to adults and is forced to keep “better company” while
being supervised. He has actually picked up a few good habits.”Classroom Teacher in Kentucky
“One of my students really needed some guidance. His attitude
towards school and his teachers has really changed.”Classroom Teacher in Georgia
“There is a 10-year old boy who has a short fuse and thinks
he does not have to follow the rules. What we did that was effective
was to set boundaries for him and stick by them. Very few adults
have been consistent with him. He responded with greater participation
in the activities and with significantly fewer out-of-control
rages.”SACC Staff in California
Leadership is a form of pro-social behavior upon which 4-H programs
have traditionally focused. In this study, we assessed leadership
by asking SACC staff if any of their children had “learned
to take more responsibility for planning and running the program
and its activities.” Nearly all of the SACC programs saw
this, and they reported that 22% of the children had grown in
this way.
HELPING CHILDREN TO EXPLORE NEW INTERESTS
Besides learning to work well with peers and with adults, a
third area of pro-social behavior we asked about was the development
of new interests by children. The middle years of childhood are
a period in which children like to do real things for themselves,
in which their horizons widen greatly, and in which they develop
new and sometimes lasting interests. A good program for preadolescent
children will expose them to a wide variety of activities, and
provide the opportunity for exploration in depth of those activities
which capture their interests. At its best, new interests developed
during this period can form the basis for lifelong vocational
or avocational pursuits. Did children in these SACC programs develop
“interests they would not otherwise have, in new topics or
activities”? Yes. The SACC staff reported that 36% of the
children had developed new interests. The classroom teachers verified
this by reporting an even greater percentage of children (46%)
with new interests gained in the SACC programs.
“The arts and crafts program is very limited at our school
and even more so for the 5th and 6th graders. The after-school
program gave these kids an opportunity to explore their artistic
and creative abilities.”Classroom Teacher in Oklahoma
“The particular student I had this year opened up many new
interests. He became particularly fond of gardening based on
a classroom project as well as a 4-H project. Since this particular
student would have gone home to an empty home, it was nice he
could have assistance in the 4-H program with his homework!Classroom Teacher in California
“For one child in particular, the SACC program has offered
an opportunity for choices and time to pursue interests – not
always possible in the classroom. This child still has problems,
but has evidenced ‘dramatic positive changes' in social and academic
skills.”Classroom Teacher in California
REDUCED PROBLEM BEHAVIORS
Reducing the violence and aggression that have become so common
in America's playgrounds ought to be a high priority.
We generally expect that children who have positive social skills
will have relatively few behavior problems. A program that teaches
children to work well with peers and cooperate with adults (the
outcomes reviewed above) should thereby reduce behavior problems
among its children. Rather than assume this to be true, however,
we asked the SACC staff, school teachers, and principals directly
about behavior problems, and the possible effect of the SACC programs
in reducing them.
When asked if “some of these children have begun to demonstrate
fewer behavior problems” because of their involvement in
the SACC program, school principals and classroom teachers reported
improvement in 26% and 29% of their children, respectively, while
SACC staff reported these gains for 19%. Here are several examples:
“We had one child that was labeled by the state guidelines
as Behavior Disordered. He received wonderful structure, but
in a non-threatening manner. The SACC program became a positive
place for this little guy. He looked forward to being in the
program.”School Principal in Illinois
“A new sixth grade boy came to our school and was having
a lot of discipline problems related to not fitting in. Once
he started in the [SACC] program, he made a dramatic improvement
which lasted the remainder of the year.”School Principal in Kentucky
“One of my boys has been doing…much better with a structured
after-school atmosphere, rather than just being ‘free' to do
as he pleases..”Classroom Teacher in South Dakota
“The child benefiting most dramatically is one of the older
children in the SACC program. Attending the program has resulted
in positive changes in classroom behavior and academic progress.
It has also improved behavior at home and in the community. This
association with young children has increased his maturity and
leadership skills. He has also become more helpful and considerate
of others.”School Principal in North Carolina
Increased after-school supervision can have an impact on the wider
school community as well. Indeed, one-third of the principals
(35%) reported that because of the program, there had been a decrease
in the amount of vandalism in and around the school.
Reducing the violence and aggression that have become so common
in America's playgrounds and classrooms ought to be a high priority
of any youth-serving program. One of the ways that a good program
can reduce aggression is by teaching more acceptable ways to solve
problems. Both the SACC staff and the school personnel agreed
that the SACC programs are doing this. The SACC staff reported
25% of the children “have learned to handle conflicts by
talking or negotiating more often, instead of just hitting or
fighting.” Principals saw this behavioral change in 23% of
children, and the classroom teachers in 37% of the SACC children
in their classes.
“Through consistent supervision, students have been provided
a structure that has reduced the number of office referrals.
Students manage conflicts in a more positive way – less fighting
and more dialogue.”School Principal in California
“One student in particular has learned to resolve conflicts
by telling the adults in charge. He has also learned to control
his temper. Prior to the [SACC] program this student exhibited
aggressive, immature behavior.”Classroom Teacher in Connecticut
TABLE 4: REDUCTION IN PROBLEM BEHAVIORS
Percentage of children decreasing their Problem Behaviors, according
to 3 observers
| 1. Fewer behavior problems. | |||
| 2. Decreased vandalism in school. | |||
| 3. Learned to handle conflicts by talking rather than hitting. |
*Note: Question not asked. Respondents were asked about
only those child behaviors they were likely to have directly observed.
**Note: This one number represents percentage of principals
(and schools), rather than percentage of children.
IMPROVED ACADEMIC BEHAVIORS.
Did SACC programs in targeted neighborhoods improve children's
academic performance?
Social skills and behavior problems are not the only domains in
which children might benefit from a high quality SACC program.
We were also interested in the possibility that SACC programs
in targeted neighborhoods might improve children's academic performance.
The classroom teachers suggested that this was true. They reported
that 33% of the SACC program children had developed an interest
in recreational reading, 21% had improved their school attendance,
and 33% were turning in more or better quality homework. Not surprisingly,
this had led to better grades for 34% of the children. It had
even led, in the teachers' estimation, to 17% of the children
avoiding being held back in grade, and 13% avoiding placement
in special education. The principals confirmed these estimates
with their responses (see Table 5). In each case, the classroom
teachers and principals reported that these improvements were
specifically “because of their involvement in the SACC program.”
“The five children in my class who attend have definitely
done better in their homework skills – this was a very important
highlight for me because they weren't getting it done at home.
Also – the Books-Across-America program – the 4-H kids ended
up being the only ones who participated and they all won awards.”Classroom Teacher in California
“I worked with L. one-on-one sometimes. I helped him understand
things and put things in a simple light. He loved to draw so
I encouraged that in him, but with the drawing that he did, he
had to count the things that he drew. So when he drew, he counted
too…”SACC Staff in Mississippi
“I have a student who entered the after-school program 5
weeks ago. Since then I've noticed her behavior has settled down.
She is completing assignments in class, is not as distracted
as before – seems more focused. Her homework is passed in each
day. Before we were lucky to see one assignment in a month. Her
social skills have improved too. The after school program has
been a positive change in this student. Wish she had enrolled
sooner.”Classroom Teacher in New Hampshire
“S. has shown improvement in the quality of her work. She
participates in class discussions. She is more ready to give
an answer to a question.”Classroom Teacher in New Jersey
These are big impacts not only for the children, but also in
monetary terms. In these school districts, the average cost to
repeat a year of school was $4,318. We therefore estimate that
the SACC program saved taxpayers over one million dollars last
year just by preventing retention in grade of the 253 children
named by the principals.
TABLE 5: IMPROVED ACADEMIC BEHAVIORS
Percentage of children improving their Academic Behaviors,
according to 3 observers
| 1. Developed interest in reading | |||
| 2. Improved school attendance | |||
| 3. Better homework | |||
| 4. Improved grades | |||
| 5. Avoided retention in grade | |||
| 6. Avoided placement in special education |
*Note: Question not asked. Respondents were asked about
only those child behaviors they were likely to have directly observed.
EXTENSION'S ROLE
“…Going to where the client is….”
What is the connection of these SACC programs to the Extension
service? Most (91%) of the SACC programs we evaluated are supported
by Extension Youth-at-Risk (federal) funds, and 88% are operated
by Extension. Two-thirds (67%) have 4-H clubs and activities as
part of their programs, and 82% have received staff training from
Extension agents. The SACC staff from these 64 programs met with
the local Extension agents between O and 150 times in the previous
year (with 15 meetings as the median response). The SACC staff
who responded reported they clearly gained from the training and
consultation provided by the Extension agents.
“They sent me to training school and had Telenet that
we were required to listen to. We also had staff meetings to
discuss problems or questions that arose, which was helpful ”SACC Staff in Illinois
“[The Extension Agent] provided constant support, resources,
and help in any area of need. Has been personally invested in
the success of the program. Most helpful – experience and knowledge!”SACC Staff in Kentucky
“[Extension Agents] provided organized county-wide 4-H activities
which were opened for my group. Youth were able to participate
in these activities which they otherwise would not have been
able to do.”SACC Staff in New Hampshire
“I think it's wonderful that finally the after school programs
can now incorporate 4-H. Most parents in my program love the
4-H aspect of it and the children do too! (I hope they continue
to provide Resource manuals, because they are priceless.”SACC Staff in California
These SACC programs represent an innovative response by the Extension
Service to the changing demography of American family life and
work life. Most children today are being raised by employed parents,
whether in a single-parent household or a 2-parent 2-earner household.
SACC programs represent a safe and enriching alternative to the
“latchkey child” situation, in which children at young
ages are left without adult supervision.
By “going to where the client is,” in this case a school-age
child care program, the Extension System has adapted its traditional
programs for youth development to these recent societal changes.
Because children are in SACC programs for many more hours each
week than in a traditional 4-H club, Extension's flexible approach
to delivery of its programs has led to a greatly expanded potential
for effecting children in positive ways.
TABLE 6: THE EXTENSION SERVICE'S ROLE IN THE SACC PROGRAM
| 1. Received federal Youth-at-Risk Initiative funds. |
91% |
| 2. Extension helped start the SACC program. |
74% |
| 3. Program is currently operated by Extension. |
88% |
| 4. 4-H Clubs operate within the SACC setting. |
67% |
| 5. SACC staff received training from Extension. |
82% |
CONCLUSION.
…multiple, positive impacts…
The SACC staff, classroom teachers, and school principals showed
considerable agreement in their questionnaire responses. This
agreement supports the validity of the results reported here.
Surprisingly, in many cases the principals and classroom teachers
saw even greater gains due to the SACC programs than did the staff
of those programs.
We suggest that the SACC programs operated by the Cooperative
Extension Service appear to be having multiple, positive impacts
on the children, their families, and their schools. These impacts
range from social skills, to reduced problem behaviors, to increased
academic achievement, and are evident not only to the staff of
the SACC program, but also to local child development experts
such as school principals and classroom teachers.
Based on these findings, we believe Extension is well justified
in helping to establish additional school-age child care programs
in communities which lack them. In addition, Extension's proven
capability in training adults to work with youth should be shared
far more broadly, with the vast number of existing SACC programs,
which have little or no contact with Extension at present. These
non-Extension SACC programs represent a crucial non-school, non-family
environment for millions of American youth today. Extension can
greatly multiply its positive impact on youth development by working
with and through these SACC programs.
How solid are these data?
APPENDIX: DATA QUALITY
Before conclusions can be believed, we must consider the quality
of the evidence. Two areas of clear concern for this study are
the generalizeability of the findings (given the response rates),
and the validity of the data (given their origin in self-report
questionnaires).
GENERALIZEABILITY
Only about half the recruited principals and classroom teachers
actually responded with completed questionnaires. We should ask
whether the non-respondents would have painted a different picture
of program impacts. It is not unreasonable to suggest that those
with more favorable responses are more likely to respond. If so,
then our estimates of impact are over-estimates.
To test this possibility, the 17 programs with the highest return
rates were analyzed separately. All 17 programs had at least 50%
response rates from all three types of respondents. The overall
response rates in this sub-sample were 100% for SACC staff, 88%
for principals, and 66% for classroom teachers.
Were the estimates of program impact substantially different in
this high-response-rate sub-sample? The answer was no. Considering
the percentages reported in Tables 3, 4, and 5 of this report,
in nearly half the cases (44%) the corresponding number in the
sub-sample was within two percentage points. The mean difference
across all pair-wise comparisons was 3.5%. Thus we can discern
little difference between the high-response-rate sub-sample and
the overall sample used for this report. We suggest, then, that
the results of this study are likely to generalize to other SACC
programs operated in high risk communities by the Extension Service.
VALIDITY OF THE DATA
If one wished to conduct the best possible evaluation of the
impact of SACC programs, using a method in which the validity
of the conclusions would be as unassailable as possible, one would
pick an experimental design utilizing direct observations or tests
of children's attitudes and abilities. As with many studies using
human subjects, however, this research design is difficult to
apply here. At a minimum, to create an experimental design would
require waiting lists for each of the programs, and the use of
random assignment to the program from the waiting lists. In practice,
few of these programs have extensive waiting lists. Moreover,
there are ethical concerns with denying needed services to children.
Conducting longitudinal research using non-program children (i.e.
latchkey children or parent-supervised children) as a comparison
group is also problematic. Prior research has shown that children
in SACC programs are systematically different from non-program
children right from the start of the year. Estimating change from
pretest to posttest can be misleading if children in the two groups
(treatment and control) are on different developmental trajectories,
so establishing true equivalence of the comparison group is crucial.
Unfortunately, forming equivalent groups for comparison, while
not impossible, is very difficult for this topic.
The method selected for this evaluation has a weaker claim to
validity than an experiment, but nonetheless has strengths of
its own. The strengths of our method were (1) the use of expert
informants, (2) three independent observers for each child, (3)
direct attribution of causality, written into the questionnaire
items, and (4) the use of open-ended comments about specific children
to verify that real changes did take place. These strengths will
be explained below. But a general threat to the validity of the
data remains: the data are not based on objective tests or observations,
but on a self-report questionnaire survey.
The threats to the validity of self-report data are many. We focus
on the two most important: the social desirability bias and the
attribution bias.
The social desirability bias derives from respondents having a
psychological motive to paint a desirable picture of themselves,
or to otherwise provide responses they believe will be pleasing
to the investigator. In the current study, we need to be most
concerned with the responses of SACC staff, since they are reporting
on the impacts of their own programs. To counter-balance the potential
bias of this method, we asked identical questions of school principals
and classroom teachers. We expected that in many cases their natural
inclination will have been, not to give credit to the SACC programs,
but rather to see the school as responsible for positive changes
in the children. Indeed, some principals and teachers wrote comments
exactly to this effect. The strategy of using multiple observers
of each child, therefore, should have mitigated much of the impact
of the social desirability bias.
The fairly high level of correspondence between the responses
of SACC staff, school principals, and classroom teachers, (shown
in Tables 3, 4, and 5), all reporting on the same children, provides
some reassurance that the particular viewpoint of any one observer
was not strongly biasing the final results. To display this convergence
between multiple sources, we have graphed in Figure 1 the responses
to the four items which were asked of all three respondents. As
can be seen, the first three items, which all index problem behaviors,
ranged from 19% to 37%, while the item indexing a change in parental
participation (a very different outcome) had no estimates in this
range (varying from 10% to 17%). The mean difference between these
two groups of items was 14%, while the mean difference of estimates
within each item was 7.3% for the problem behavior items, and
a 4.7% for the parental involvement item. Thus the difference
was greater between groups than within, supporting the claim that
the different observers were converging in their estimates of
particular item content, and discriminating different item contents.
The attribution bias refers to the tendency among humans to seek
order even in chaos, and in particular to believe that cause-and-effect
relationships exist even when they do not. Thus, even if children
are changing randomly (or, for example, are experiencing different
timing and pace of biological maturation), most respondents will
develop personal theories about what is causing the changes.
The attribution bias does not specify that the SACC program, in
particular, will be the beneficiary of the bias. We therefore
avoided wording questions in such a way as to encourage the spread
of the causal attribution bias to the SACC program. Specifically,
we avoided wording the questionnaire items so that the SACC program
could be given some small and partial credit for changes in children
(i.e. we avoided questions like “Did the program contribute
to…”). Instead, the questionnaires asked for a firmer commitment
to cause and effect: “Because of their participation in the
[SACC] program,” did any children change in each way.
FIGURE 1: CONVERGENT VALIDITY CHECK USING THREE DATA SOURCES
Percentage of Children or Parents reported changing, according
to 3 observers
| 1. Child became more cooperative with adults. | ||||
| 2. Child had fewer problem behaviors | ||||
| 3. Child learned to talk rather than hit | ||||
| 4. Parents increased involvement in program or school |
As evidence that this strategy may have worked, many respondents
wrote notes on their questionnaires reminding us that many forces
contribute to a child's development, not just the program we were
evaluating. Here are two examples:
“We have other programs, so it would be difficult to
give credit to any particular program.”School Principal in Oklahoma
“It's very hard to determine who or what makes a difference
in the behavior changes of the children at this age. A lot of
behavior in these two children reflects the home situations –
maturation, the school programs and expectancies, and the adjustment
to a new school.”Classroom Teacher in California
A separate form of evidence for the validity of the data comes
from the quotations which have been included in this report. The
quotes are not representative in any sense; they were included
not as evidence, but rather to exemplify and put a human face
on the numerical findings. Still, by the fact that they provide
so coherent and convincing a picture of the same changes in children
as in the numerical data, they lend credibility to the numbers.
In other words, the testimony of SACC staff, classroom teachers
and school principals about particular children who made great
gains assures us that at least some children were benefiting in
the ways suggested by the numerical data.
By thoughtful attention to our methods of investigation, we have
attempted to minimize the effects of non-random recruitment, social
desirability biases, and causal attribution biases. We cannot
claim to have excluded these biases altogether, but we believe
a claim can be supported that these biases have been minimized.
The results of this study do not obviate the need for randomized
field experiments, but they provide the most convincing evidence
to date that Extension-supported SACC programs are causing significant,
positive changes in the lives of children.
DOCUMENT
USE/COPYRIGHT
National Network for Child Care – NNCC. Part of CYFERNET, the
National Extension Service Children Youth and Family Research
Network. Permission is granted to reproduce these materials in
whole or in part for educational purposes only (not-for-profit
beyond cost of reproduction) provided that the author and Network
receive acknowledgment and this notice is included:
Reprinted with permission from the National Network for Child
Care – NNCC. Riley, D. (1994). *Preventing problem behaviors
and raising academic performance in the nation's youth: The impacts
of 64 school age child care programs in 15 states.* Madison,
WI: University of Wisconsin Cooperative Extension Service.
Any additions or changes to these materials must be preapproved
by the author.
AVAILABLE FROM::
Dave Riley
Cooperative Extension Service
University of Wisconsin – Madison
120 Family Resources Building
1300 Linden Drive
Madison, WI 53706
Phone::608-262-3314
FAX::608-262-5335
INTERNET::riley@wisplan.uwex.wisc.edu
COPYRIGHT PERMISSION ACCESS::
Dave Riley
Cooperative Extension Service
University of Wisconsin – Madison
120 Family Resources Building
1300 Linden Drive
Madison, WI 53706
Phone::608-262-3314
FAX::608-262-5335
INTERNET::riley@wisplan.uwex.wisc.edu
FORMAT AVAILABLE:: Internet
DOCUMENT REVIEW::
Level 3 – National Peer Review
DOCUMENT SIZE:: 64K or 18 pages
ENTRY DATE:: August 1996
Contact Us | Non-discrimination Statement and Information Disclosures | © Iowa State University, 2002 | Last update: 8/3/06
