Year One Evaluation

Section II - Preliminary Evaluation Report
 
The Year 1 evaluation was conducted by an outside evaluator and was designed to document project activities and collect information to be used in assessing project outcomes. Evaluation activities were designed to answer the following questions:
  • What are the demographic characteristics of the students who participate in the Girls RISE project? How do participants assess program activities? What recommendations do participants have for program improvement?
  • What knowledge, concepts and skills do girls acquire as a result of their participation in Girls RISE?
  • What problems do project staff encounter in implementing the Girls RISE activities? What institutional and community characteristics facilitate or delay implementation?
  • What recommendations do project staff, administrators, students and parents have to improve the program in the future?

 

 
Data Collection
 
Demographic and other information about project participants was collected from application forms, student nomination forms and interview sheets. In addition, a questionnaire was designed to collect beginning information about participating girls' attitudes about math and science, their prior experiences with technology and their personal knowledge about scientists and engineers. The first group of program participants completed the questionnaire in early November; the additional participants joining the summer program completed the questionnaire in June.
In early August 1997, the project evaluator conducted individual interviews with all project staff and observed classes at both the Museum's Technology Center and the summer satellite center, Southside Elementary School. Additionally, she reviewed project records and the project web site. All program participants were asked to complete an evaluation form, which was developed to collect information about student perceptions about Girls RISE activities at the end of the first and second summer sessions. In addition, the project evaluator interviewed a sample of student participants. The evaluation findings which follow provide information about participant and staff perceptions of the project, as well as recommendations for project improvement.
 
Findings
 
As shown in Table 1 below, all participants rated all aspects of the program as excellent or very good.
 
Table 1- Girls RISE Participants' Perceptions of the Program
Participant Ratings of Girls RISE Program Activities and Content (N = 21)
Program Activities  Excellent  Very Good Good Fair Poor Missing
 Classes
 Working in Teams  48%  33%  19%  0%  0%  0%
 Journal Writing  19  43  38  0  0  0
 Guest Speakers  48  29  14  5  0  0
 Field Trips  62  38  0  0  0  0
 Design Competition
 Team Presentation  38  33  19  0  0  0
 Program Content
 Electricity/Series/Parallel Circuits  48  33  19  0  0  0
 Basic Schematics  38  38  24  0 0  0
 Wiring a House  33  38  19  0  0  10
 Ohm's Law  24  27  24  10  5  10
 Gears & Carrying Power  48  29  14  10  0  0
 Friction  57  29  10  5  0  0
 Motor Carrying Power  57 19 24 0 0 0
 Technology Training
 Internet  76  19  5  0  0  0
 Computer Graphics  57  33  5  5  0  0
 E-mail  38  43  10  9  0  0
 Web Pages  76  14  10  0  0  0
             
 Overall Assessment  81%  19%  0  0  0  0
 
 
 
 
Girls RISE project participants were also asked the extent to which the program increased their interest and knowledge of engineering. Table 2 provides information on their responses.
 
Table 2 - Girls RISE Participants' Assessments of Change In Their
Interest and Knowledge of Engineering (N = 21)
 
Tell how strongly you agree with each of the following statements.
1 = strongly agree, 2 = agree, 3 = uncertain, 4 = disagree, and 5 = strongly disagree
Statements 1 2 3 4 5
Science and engineering activities are more interesting to me 57% 43% 0% 0% 0%
I know more about engineering careers 68 24 10 0 0
I am more interested in science and engineering careers. 48 37 13 0 0
I am more interested in computer technology. 68 14 18 0 0
My problem-solving skills have improved. 48 38 14 0 0
I am better able to work with others in a team 48 43 5 0 0
I know more about engineering. 86 14 0 0 0
I know more female engineers. 57 43 0 0 0
I spent a lot of time with engineering mentors in this program. 62 33 5 0 0
 
Conclusions
 
The goal of the Girls RISE program is to foster sustained participation in engineering-related studies by reducing the attrition that occurs as girls move from middle school to the high school. Findings from the evaluation show that Miami Museum of Science staff were able to achieve program goals through the success implementation of this model demonstration program. Staff established an informal, non-competitive learning environment in which students felt safe to experiment with engineering concepts and processes. They developed activities that permitted students, working in teams, to apply the engineering principles they were learning to real life projects within the confines of the Museum. When asked to evaluate the program at the end of the two summer sessions, all of the participants rated all aspects of the program as "excellent" or "very good." Several students said that they felt incredibly fortunate to be part of the program and that they wished there were some way more girls could find out about and participate in programs like Girls RISE.
The findings also indicate that the program's short term goals were accomplished. All participants stated that, as a result of the program, they knew more about engineering, knew more female engineers and were more interested in science and engineering activities. As indicated in the following quotes (Figure 1), although only a few girls indicated that they were considering an engineering career at the beginning of the program, almost all of the girls who were interviewed, post summer program, had specific plans to attend math/science magnet high schools and planned to pursue engineering or other-science related careers. Of the 15 girls who were interviewed, 13 said that they had made changes in their plans for the future since participating in Girls RISE and of the two who reported no changes, one said the program helped her develop a better perspective on how things work and are made and the second said the program gave her a better perspective of the options available to her.
 
 

Figure 1

Girls RISE Participants' Plans for High School, College and Future Careers, July 1997

(N = 15)
 
  1. I want to go to MAST (Marine Biology) or Northwestern Magnet school. It has a music program. I plan to major in computer or environmental engineering and minor in music. On the field trip to the University of Miami we met an engineer who made film. I think he minors in music. After college, I want to become a computer or environmental engineer.
  2. I want to stay where I am at NW Christian Academy but my mom wants me to go to MAST Academy or Turner Tech because she thinks my school won't prepare me for what I want to do. I want to go to FSU and maybe major in chemistry. I love chemistry or anything to do with science. I want to be a scientist.
  3. I want to study higher math and sciences. I've heard good things about MAST Academy and a new school, Coral Reef, that will have an environmental focus. Whatever I choose to be, I want to be the best. I want to be valedictorian. I will definitely be an engineer or a scientist.
  4. I am interested in two occupations: engineering and business & finance and computers. I want to go to Jackson, a business and finance magnet school. Then I'm hoping to get a scholarship for either business or engineering. I go to Carver Middle School, a magnet for languages, where I take German. I think I will try out zoology. For college I haven't decided yet, but I'm interested in being a pediatrician or an environmental engineer.
  5. I would like to go to Northwestern, a medical magnet. I'm now attending a summer medical program at UM. This is the last two weeks and I think I'm going to get in. I will apply in 10th grade if I don't go in 9th grade. I have to put in maximum effort to get maximum results. Even if I don't understand it at first, if I stick with it I get it. I want to be a biomedical engineer.
  6. I'm not really sure. I want to do something with math and science. I think a graphics engineer. After college I am thinking about being an engineer.
  7. I plan to take as many courses as I can in computer graphics and foreign language. In college want to major in engineering and computer science. I was the only female in the engineering club in middle school because the other girls aren't interested. Before I didn't know what I wanted to do. After college I want to be an engineer and also plan to be a college professor and teach engineering.
  8. At first, I wanted to be a lawyer. But now I'm thinking about becoming an environmental engineer.
  9. I want to go to MAST Academy because I want to be a marine biologist. I love dolphins but I'm considering engineering now. I'm thinking about FIU or FSU for marine biology or engineering or Georgia Tech where my stepsister goes. I would like to get masters or a Ph.D.
  10. I want to go either to MAST or DASH, a design and architecture magnet school then to UM for engineering. I want to be a biomedical engineer or architectural engineer or architect.
  11. I want to go to MAST but I'm not sure yet about college. I want to be a scientist but I'm not sure about the field.
  12. I want to go to Coral Reef because they have many different magnets there. They have engineering. I want to get experience in test engineering and some background in computer engineering. For college, I wanted to try some kind of Ivy League like Duke or Harvard but ever since the field trip to the University of Miami I've been considering there. I want to get a masters degree and have a career as a test engineer.
  13. I'm looking into Coral Reef, a new magnet school. I want to be a pediatrician and they have a good medical program. Otherwise, I'll go to Killian my zoned school. I'm not too worried about high school because I am just starting the 7th grade. I've wanted to go to the University of Miami since I was a little girl. If I don't become a pediatrician I want to be a meteorologist and become a weather forecaster.
 
Formative Evaluation- Suggestions for Improvement
 
Nearly all staff and students remarked that they thought the Girls RISE program was just about "perfect the way it is." The following suggestions were made by the evaluator to improve the program in the future and to help other people who may want to implement a similar program.
 
1. Project Design
 
Both staff and students emphasized that Girls RISE is a comprehensive, integrated program. They believed that all project components, including the field trips, mentors, cooperative learning teams, guest speakers, and hands-on instruction in computer skills and engineering principles, support and complement each other. It is essential, therefore, that all components be maintained if the program is replicated.
Staff and students believe that it is very important to continue to expose girls to as many different aspects of engineering as possible. Before participating in the program, some students had little or no knowledge of what engineers did. For the most part, those who had some idea about engineering knew only that engineers designed or built things. Through exposure to the mentors and other female engineers in different fields, almost every girl has decided that she can pursue an engineering career in her area of interest.
One mentor suggested that staff should use the information the girls provide when they complete the project application forms regarding their career interests to research and plan activities in advance which will help students see how engineering relates to their own interests.
 
2. Project Staffing
 
Several staff and students felt the program would be improved if there were more staff, especially during the school year. Mentors and students related that the student volunteers provided valuable assistance during the second summer session and should be incorporated as a permanent part of the program. One mentor said, "It's good for the girls and good for the staff too." A student volunteer suggested that there should be more volunteers because she said, "The mentors do a lot but they have to work with the teams. I help out the individuals."
During the 1996-97 Saturday academic year component, the project director, who was the staff person dedicated to the project, acted as the only mentor to the 24 participants. One Museum staffer worried that this staffing pattern would make the project difficult to replicate because she believed few people would have the skills to direct hands-on instruction with such a large number of students. A student remarked that there needed to be more mentors available during the school year (not just in the summer), so that the students could engage in more hands-on activities.
 
3. Staff Training
 
A more formal staff orientation and training should be provided for mentors. In addition to information about the computer programs, the mentors need time to try out hands-on projects before using them with the students. They also should be provided with some information about adolescent development and some training in group process and conflict resolution/social problem solving.
 
4. Parent Involvement
 
Although parents were involved in culminating activities, staff should plan a formal orientation meeting for parents with translators, if necessary, so that all parents can understand the goals and objectives of the program. Many parents actively support their daughters' interest in science as evidenced by the number of the girls who reported that they were encouraged to participate in the program by their parents who read about Girls RISE in the newspaper. Because the Girls RISE participants are still young, it is important that all parents know how to best encourage them in their career interests. Parents will also need to know how to help their daughters select appropriate high schools and high school curriculum. Since not all parents are able to come to meetings, other means of communication such as telephone calls, newsletters, etc. will also have to be used.
 
5. Space and Equipment
 
Staff and students wished that there were more space, especially in the computer lab. The program applications and student work were stored on the computer hard drives, and because the lab was used by other people, sometimes Girls RISE staff and students found that necessary applications had been removed and/or their work had been inadvertently deleted. One mentor suggested that the Museum consider allowing project programs and work to be saved on the file server with passwords rather than on individual computers. Another mentor recommended that the girls have some kind of individual storage space, especially for use in storing their on-going project work. She suggested the kinds of cubbies used in preschools. Plastic or cardboard shoe boxes might be sufficient.
 
6. Student Assessment
 
One mentor suggested that the girls be provided with a written assessment of their work, focused on their strengths, at the end of the summer session accompanied by guidance on things they needed to continue to develop. She felt in this way, girls who did not continue in the program would have something concrete to which they could refer in the future. Another mentor felt that program staff needed to find ways to continue to provide girls with on-going guidance and support after the program was over so that they would continue to have access to personal support and accurate information when they needed it. These might include e-mail mentors, Girls RISE reunions, newsletters, etc.
 
7. Student Recruitment
 
Mentors remarked how impressed they were with the students' enthusiasm and motivation and hoped that the recruitment and selection process remained the same. Several students said that they felt incredibly fortunate to be part of the program. They wished there were some way more girls could find out about and participate in Girls RISE.
 
Section I Section III

HomePage Table of
Contents		 Program
Overview		 Starting a
Girls RISE		 Explorations Girls on
the RISE

 

For further information contact Wendie James.