Printable Version of this PageHome PageRecent ChangesSearchSign In

Evaluation Blog

<December 2018>

Evaluation Notes: 14 April 2005

Prior to session:
This will be "Switching Gears" day, in which the 5th grade and 6th grade science clubs get together. The plan is to have 30 minute blocks. During block 1 half the VIM students and half the KSI students will be doing VIM; the other halves will be doing KSI. At the end of the 30 minute block the two groups will switch. During the last 30 minute block we will hold 2 15 minute discussions: first a group-wide discussion of VIM and secondly a discussion of KSI.

During our 30 minute block we plan to group the students approximately 2 VIM and 2 KSI students per group. We will give each group a basic balloon car which has much room for improvement. The instructions to the students will be that only KSI students can modify the cars, but with instruction from the VIM students. In this way we hope to avoid the problem of VIM students taking over the entire exercise. As the groups modify their cars they will test them and hopefully get better and better results.

Evaluation Notes: 7 April 2005

Prior to session:
As most students have not completed their first balloon car, this week we plan to allow them that opportunity. We would still like them to use SMILE and the handhelds for this design. However, as we would like the students to have the opportunity to build a rubber-band car, this week after the students complete their balloon car we will allow them to build a rubber-band car or to build a better balloon car, at their choosing. Prior to letting them go, however, we plan to have a brief discussion where we tell them what the plan is for week 10.

After session:
Today we had about 10-11 students attend, and although this number was relatively small they made a lot of noise. We forgot to bring styrofoam cups today, so most groups could not work very well on their balloon cars. Because of this and the energy level of the students we decided not to use the laptops nor the handhelds. In retrospect this was a very wise decision. Many groups were able to complete their balloon car design. Four groups proceeded to rubber-band cars. It is my assessment that not many students understand any better how to make a balloon car than they did when they first messed about with them. Primarily construction issues held most cars up: straws were not attached to the cups and flew out without propelling the cars; balloons, when inflated, touched the wheels and locked them up; even some of the base coaster car functionality was lacking as many wheels went untightened and axles unstraigtened. Of the 4 groups that tried rubber-band car 2 of them had working cars and 1 group would have if they had been allowed 10 more minutes. The 2 working cars worked relatively well. One of the groups got hung up on wanting to add more weight in order to add friction. Partly this is my fault because I showed them 2 rubber-band cars, one of which was heavier and ran better. Their primary problem, however, was that the wheels were not tightened properly and slipped. I showed this to one of the team members, but the other team member (the one obsessed with weight) took the car away and added weights to it.

I'm not sure if anyone learned much today, but they seemed to have fun doing it.

Evaluation Notes: 31 March 2005

Prior to session:
This week we intend to allow the students to begin iterating over their final balloon car designs. Some groups last time did not complete the modelling activities. I suspect we will allow those groups to complete them if they would like. I think it might be a good idea to bring the poster paper with our question list from last time to refresh the students' memories about what variables might be important. We can hold a very brief discussion in which we emphasize the variables that the software models and what effect their various values had.

We hope to allow the students to proceed with the design iterations at their own speed on their own. For the design activities we will emphasize that students must follow a certain procedure for each design iteration:
  • They may only change one thing per iteration (e.g. 2 engines instead of one).
  • For each iteration they must complete the design iteration plan in SMILE prior to testing.
  • They must use the handhelds to record the results of testing their design.
  • They must upload those results and then analyze them within SMILE.
Hopefully by enforcing these constraints better designs will occur, even if it means that students are able to do fewer designs. We intend to follow this procedure for this session as well as for the next session. We anticipate that some groups may only be able to complete 1 or 2 iterations while others may complete 5-6. This is by design: some groups go through these iterations faster than others and we want them not to be bored while waiting for others to complete an iteration.

The principal component missing from this, however, is any notion of cross-group discussion. There is no gallery walk or pin-up session, so groups will not get an opportunity to learn from each other. Perhaps we can incorporate some kind of class discussion in the beginning of the next session, after each group has had an opportunity to complete at least one iteration.

After the session:
This week John was unable to stay because he was teaching at GSU, so Swaroop and I were on our own. We assembled the kids at 3:00 and held about a 25 minute discussion about the variables that were important for the balloon car, with special emphasis on those that were explored in the software from the week before. I felt the discussion went a bit long, and we got hung up on the first variable and a discussion about air pressure which neither Swaroop nor I were really ready to discuss. I think we did OK with discussing air pressure, especially considering we hadn't prepared. I think through a combination of the software from last week as well as our discussion we got rid of some misconceptions about what could make the balloon cars go farther.

After the discussion we told the students they could begin making their coaster car and then informally testing it. After that we wanted them to enter their data in SMILE for their balloon design, build it, test using the handheld, and then analyze it. In practice most groups opted to begin with SMILE so they could quickly get to building their cars. Two groups in particular were upset that we asked them to build a boring balloon car rather than work on their car from the previous week. One group completed testing of their car, and one other group was close to being able to test. Otherwise most groups still have some work to do on their first balloon car.

Evaluation Notes: 17 March 2005

Today our plan was to hold a discussion with the students about the variables of the balloon car and what effects their different values may have. Afterwards we intended to have the students use Swaroop's simulation software again, this time with the balloon car. After the software activity we wanted to have another discussion with the students about what they learned using the software.

We held the initial discussion as planned. Swaroop and I both thought it went well, continuing our tradition of feeling well about discussions. This time, however, nearly every student (~16 present) participated in the discussion, with maybe just 3-4 students not talking. I had drawn a picture of a balloon car on the board, and used it to help illustrate some of the key variables. I first asked the students to remind us of what they did 2 weeks ago. They were able to do so. Then we talked about which of the 3 engine types performed best and why. As I put up each suggestion I made sure to put question marks around it to show that we were not sure if the prediction / explanation offered was correct. After discussing the 3 variables we asked the students to think about other variables. Many different design features were mentioned, many of which we had not really anticipated (e.g. placing a plate behind the straw so the air hit it; other students were quick to point out that if the plate was attached to the car there would be no effect). Question marks were placed around each variable and explanation.

After this discussion Swaroop led the students through a demo of his software using the projector. He performed the first experiment on his laptop and asked the students to follow along. Insofar as we were able to maintain the students' attention and get them to stay with Swaroop this demo helped them to perform the tasks, I believe. We allowed the students some free time (~5 min) in explore mode, during which most groups tried to build a "super" car which would go off the track. Afterwards Swaroop led them through a demo again and then let the students go on their own.

As the students were experimenting I went around and asked them to fill out their design sheets as they went. We wanted the students to fill out design sheets which basically represented their design for the next session. On the sheet were rows for each critical variable and a space for filling out the value of it and the "why". By the end of the session most groups had filled out about 3 of the rows. Most groups did not complete all 4 experiments, and one student even asked if they could complete the experiments next session.

Our tentative plan for next session is to allow them to begin to build a standard balloon car, using SMILE and the handhelds. Then we will let them go on their own, doing as many iterations as they would like, as long as they fill out a design iteration and record results using the handhelds for each new design and as long as they only change 1 thing per new design. I think this will be difficult to enforce, as many groups enjoy tweaking their design in between individual trials, let alone in between different design iterations.

Evaluation Notes: 3 March 2005

Today our plan was to hold a short gallery walk regarding the coaster cars and then allow the students to mess about with balloon cars which we had created. We had a difficult time deciding how to hold the gallery walk discussion. It had been 2 weeks since the students tested their coaster cars, so we feared that they would forget about the important variables of the coaster car design and construction. However, the discussion went relatively well. Most of the students who participated were the students who usually participate and who were also present last week. A couple of variable were mentioned that we, at least, do not think are that important (e.g. aerodynamics and shape / configuration of chassis). We took these opportunities to relate to vehicles from 2 weeks ago, showing that one group that had a funky design in which the chassis hung from the wheels did about as well as everyone else. We also allowed the group that did the best to describe their design to show the class that a standard design could perform very well if assembled with care.

After the discussion we gave each group a base and 3 different balloon engines: normal-length straw, short-straw, and double/normal-length straw. We asked each group to record a prediction about which would perform better and then to record their data when they tested it. We then told the class that after they tested their 3 engines they could try out different designs, as long as they wrote down their prediction and their data when they tested it.

Without exception every group performed only one trial for each of the 3 basic engines. Many groups found that their predictions were incorrect, but we did not push them hard enough to think about why that was. After they were finished with these 3 most groups began trying other designs. We tried to have groups write predictions and data for these other designs but for the most part failed. A couple of groups never completed their “for fun” design and therefore never got a chance to test it. Several groups became unfocused and stopped working on cars altogether. Several students were just roaming the hallway shooting balloons at each other.

I believe this was a far more successful messing about activity than the one we held on week 1. However, I am not sure if we got the students to think critically about the important variables in balloon car design, as their “for fun” designs were just as wacky as their “for fun” coaster car designs were. No groups seemed to be going about things methodically. They would change 50 things in one design and then test it. We almost held a discussion at the end, but ran out of time because some of the groups that were legitimately working wanted to test their final design. I believe a discussion may have helped us focus the students’ attention on the important variables. Perhaps that’s how we should begin next session (2 weeks from today because of their spring break).

Evaluation Notes: 24 February 2005

Prior to going to the school:
Last week the students finished testing their cars and most student groups were able to analyze their results in SMILE. Two groups have not yet had this opportunity. This week we will begin by allowing groups to further analyze data in SMILE (if they need to) and to describe on poster paper their design and their results. Each group will then have the opportunity to present to the class their results and what they think worked very well in their car and what they could improve. We will ask groups to put on their poster paper a sketch of their design (with the important parts highlighted), the data table of their results, and what they would change if they were to do it again (we should restrict them, again, to using the materials provided, with no extra wheels, etc). We hope that by having groups present their designs in this way the class will have a chance to compare designs across groups. We have 2 groups whose vehicles went very far, while the other 4 did not do as well. We hope to stress the “normalcy” of those cars that did very well so that other groups will realize that they can just follow better construction practices to make their cars go further. The primary mistake most groups made was not tightening the wheels enough. Hopefully we can emphasize this as well.

After our gallery walk Swaroop will help lead the students through activities with his software. We will elicit design recommendations from the gallery walk. These should transition the students into working with the software.

After the session:
Only 7 students attended today’s session. Apparently there was a meeting for some other special activity going on from 3:00 to 4:00. We waited for 30 minutes to see if any more students would show up. We had some technical difficulties with the laptops not supporting the resolution needed for the simulation software. We were able to find 2 computers which did support the proper resolution. Because of this and the lack of students, we had to alter our plans for the day. We still wanted students using the software, but we couldn’t have all students use it at once. We decided on splitting the students up to have some of them work in the hallway with me while the others did computer stuff, and then we would switch.

We divided the students into 2 groups of 4 and 3 each. The group of 4 went into the hallway with me, where we had setup 2 ramps. I helped the students test and improve their cars they had made the previous week. I tried to encourage students to help each other, as the normal groups had basically dissolved with only 7 students there. Unfortunately they did not help each other across what remained of their groups too much. While in the hall the students were excited about testing their cars. I kind of view it as a more structured messing about than the one we tried to hold during the first week. I hope that I impressed upon a few of the students the importance of good construction rather than strange designs (like adding a fifth wheel in the center). We will see next week if we have them do a gallery walk.

Evaluation Notes: 17 February 2005

Our plan today was to finish building cars, test the cars, hold a gallery walk discussion, and then possibly begin with the simulation software. Two groups’ data did not get saved to SMILE last week, so Swaroop and John were to help them get their data quickly entered. While they were doing this we envisioned there would be at least two groups who were ready to test. Those 2 would go out in the hall with me, where we had one ramp setup. The other 2 groups could stay in the classroom and continue working on their cars. The plan for the discussion involved having the groups analyze their data in SMILE (fill out the remaining elements for the design iteration) and then, on a sheet of poster paper, sketch their design and list their results. We would then post all the results and each group would come up one by one and present what they thought was important from what they saw. After each group went we would hopefully lead the class in discussing design recommendations.

In practice, we found that we made 2 fatal flaws. The first was only having one ramp setup. This meant that 2 groups would be in the hall and 4 would be in the classroom. Before testing groups might not have anything to do. After testing they also might not have anything to do. Because I was the only one that knew how the handhelds worked, and I was in the hall, the students were unable to upload their data to the laptops once they completed their testing. Finally I walked Swaroop through how it was supposed to work and he was able to help them. We spent probably the entire first hour testing. This time the testing procedure the students followed was more rigorous, perhaps because we limited the number of students in the hall and because I was with them (or, perhaps because of the handhelds; wouldn’t that be nice?). Two groups had problems in which their data was lost on the handhelds; I am unsure as to why this happened. One group misplaced their handheld (I believe someone from another group picked it up to play with it) and so I had to re-enter their data. Luckily in all 3 cases the group members were able to remember their data.

Once everyone was done testing, we began a piecewise attempt at getting the students to analyze their data in SMILE, with Swaroop and myself going around the room, uploading the data for the students and then asking them to get on a laptop and fill out the remaining questions in the design iteration. One thing I noticed about the design iteration tool is that when you edit the artifact, you cannot see the answers to elements you have already filled in. So if you have a results table, once you hit edit all you see in the results element is the cryptic [! TableAttachment: Table Caption !]. So the students cannot analyze their data as readily as they can for the Experiment Plan; with the experiment plan at least you can still see the plan, even if you can’t see the results table. I should look into changing this for the next time SMILE is used (create a Design Iteration and a Design Iteration Analysis tool). Right about the time that we got the last group logged into a laptop it was 4:25 and time to clean-up.
Some groups did not finish analyzing their results in SMILE. Some groups finished with enough spare time to build a second car. We did not have time to hold any class discussion. Next week I think we should have students enter analysis in SMILE if they have not done so, review their analysis if they did so last week, and prepare their posters for presenting to the class. Hopefully we could be done with the presentations by 3:45 at the latest and have time for Swaroop’s software.

Handheld notes

Most students were able to use the handheld fairly easily. I had to help them a little bit with the data table wizard, although not nearly as much as I had to in the past 2 experiences. I also seem to be finding that the input method on the Pocket PC is not at all intuitive to children. It is very difficult to enter text because Microsoft presents the user with so many options. Additionally it is exceedinlgy easy to mis-type something because of small buttons.

In the chaos I was unable to get the students to use the notes or to insert averages. I do believe some groups used the notes fields to enter whether the car went straight or curvy. At least three groups clicked on the notes button to enter their data. I think I should put more thought into this value detail screen. Once the students got to that screen several of them did not know what to put in the ‘Value:’ field and instead typed their value where the notes go. I should have some prompts to indicate what the different fields are. I also noticed a bug that when the value is changed in the detail screen it is not updated on the main screen (I probably just need to do a screen refresh).

Another usability issue I found was that the Remote vs. Local login is still tricky. I am either not hiding the difference well enough or I need to make the difference more explicit. For instance, if you login to the server but have a local copy on the handheld, when you save you will overwrite the local copy (and any data table that might be there). I believe this may have happened for group JEMS in which SMILE was restarted on the handheld after being cradled, resulting in a server login which overwrote the group’s data.

Last modified 17 October 2005 at 8:10 pm by Jason Williams