I tried out Socrative for a grammar revision quiz with a group of 30 PhD students (mainly Italian, but also students from Turkey, Albania, Venezuela and Pakistan) doing the course we run 4 times a year on Writing Scientific Papers in English. Each of the 5 weeks of the course has a flipped element, as the students watch short grammar videos (simple screencasts of powerpoint presentations with commentary). The grammar topics for the first two weeks were, firstly, the use of verb tenses in a typical Introduction section, and then the use of articles, specifically in a Methodology section. After the videos, students have to do an online quiz and a pre-lesson “activation” task regarding papers in their own field. At the end of each week, 2 students from each sub-group of 6 write the section of a paper under consideration that week on a given topic, then their colleagues give a peer review.

For the Methodology section writing, the topic was an innovative method for cooking chicken, I selected 6 sentences they had written containing either an error with verb form, with the use of articles, or one of each. I then had to decide how to use Socrative with these sentences and chose to use the “short answer” option, perhaps unwisely. I wanted the students to identify and correct any errors in the sentences but without telling them what type (verb form / article / both) of error/s each of the 6 sentences contained. Uploading their sentences was the easy part, whereas uploading the correct version was problematic. As I had opted for short answers, the space to write the correct answers is, unsurprisingly, short.

I found that I could copy and paste the complete sentence and then correct the errors, but I could only see a few words at a time in the answer box and had to use the arrow keys to scroll through the sentence. I thought this may cause problems, and it did. I toyed with the idea of getting students to only write the corrections, not the complete sentence, but this would have been even more complicated to set up and explain, and I also verified that the answer box for students doing the test was not restricted. So, I copied, pasted and corrected each of the original sentences and provided explanatory feedback in the appropriate box, a particularly useful feature that automatically leads to formative assessment. Launching the quiz, I opted for instant feedback, as students were to do it in their own time. I had told them about the quiz on Thursday afternoon at the end of the lesson, although I wasn’t sure about the format or number of questions at that time. However, they were told to keep an eye on the updates on the VLE we use from Friday evening onwards.

I prepared the quiz on Friday afternoon after a particularly heavy week’s work, and I was very conscious that I had to make sure the correct version was totally correct, given the complication of the tiny answer box I had to work with. Fortunately one of my admin colleagues was working late and I asked her to try out the quiz. Her English is akin to that of the students on the course (high B2 to C1 level) and she was enthusiastic about the quiz, although she did get a couple wrong, including, fatefully, question 4 …

Now fairly confident that the quiz worked, I had to make sure my instructions to the students were as clear as possible. I posted these as an update on the VLE, and they read as follows (it’s easier to copy and paste them than explain):

A quick revision quiz of the grammar topics from the first couple of weeks (verb forms / articles) is now available under the links icon. You need to insert LISTER656 as the “room name”, and then you should be taken directly to the quiz. There are 6 questions, which consist of sentences taken from your Methodology writing (anonymously!). Each sentence contains errors either regarding verb form or the use of articles, or both. So, there may be just one error of verb form, just one error regarding the use of articles, or two errors (one of each) in every sentence. You should copy and paste the complete sentence in the answer box and make any corrections required by changing a verb form and/or substituting, eliminating or adding an article. Submit your answer and you’ll immediately discover if it is correct, along with some explanatory feedback. Then do the next question, and so on. Hopefully this should take you no more than 10 minutes. If possible, please do the quiz before midnight on Monday 10th February.

Thanks.

On Saturday afternoon I logged in to Socrative to see how things were going and check how results are visualised. One drawback is that when students had attempted a question more than once and had answered incorrectly, it doesn’t seem possible to verify the chronological order of their answers. Anyway, I noticed that one question in particular, number 4, was proving impossible to answer, and at that moment I also received a message from one of the students who couldn’t understand why her answers to a couple of questions were not being marked as correct. While in two cases there were oversights on her part, the case of question 4, and the problems it was causing everyone else, led me to discover that I’d somehow eliminated a letter from the word “heterogeneous” in the correct version of the sentence, which meant even if students correctly identified and rectified any errors, their answers were being marked as wrong. I immediately thanked the student who had helped put me on to this and sent everyone this update:

Thanks to everyone who has done the revision quiz through the Socrative site so far. Checking your answers, I’ve discovered that most of you actually got question 4, the one about the heterogeneous mix, right, but I’d made a mistake, as when I copied and pasted the answer, a letter disappeared from “heterogeneous”. Sorry! That should now be fixed for anyone who hasn’t done the test yet or if you want to try it again. Don’t spend too long on it and don’t worry about getting things wrong, as long as you can understand what the correct answers should be through the instant feedback.

I’m trying out this tool for the first time as part of an online course on using digital technologies, so thank you for being part of this experimentation. Any feedback on its utility would be welcome, either by mail or in class on Tuesday.

Enjoy the rest of the weekend.

At this point I had decided to inform them that they were being very willing but unwitting guinea pigs. I then did what I should have done at the start, and logged in as a student to make sure the quiz was now working correctly. What I discovered was that the correction I’d made didn’t appear, as the quiz was still “live”, so I hastily wrote another update: 

It seems that because the quiz is “live”, the correction I made to the answer for question 4 doesn’t appear, so for question 4, apart from correcting any errors of verb form and/or article use, make sure you write “heterogenous”, so without the “e” before the “o”. 

Thanks.

10 minutes later I was able to verify that a student had done the test and answered question 4 correctly, following my guidelines about the misspelling.

I compared and analysed results occasionally over the rest of the weekend, but later noticed another slip-up, with a spacing error leading to wrong answers for question 3, so further skewing the results. As in class we’d been discussing limitations with methodology and results, and how to minimize them, my final update read:

I’ve closed the original version of the revision quiz, but if anyone still wishes to do it, or do it again, there’s a corrected version available using the same link and code. The “heterogeneous” problem has been fixed, as has another problem which I discovered with question 3 (there was an erroneous extra space between words in the original sentence, which didn’t appear in the answer). Thanks again for participating in this experimentation, despite its minor limitations regarding the methodology, which slightly affected the outcome ( = the researcher messed up).

Ian

The definitive corrected version of the quiz is still available to try out if you wish (room name LISTER656).

All in all, it’s been a salutary lesson about making absolutely certain that things are set up perfectly. I thought I’d taken all the necessary precautions, including the trial with my colleague, but it was actually a good indication of the potential of the tool that I was able to make adjustments on the go, thanks to viewing results as they came in and virtual interaction with the students.   

Despite the aforementioned problems with the quiz I created, the students were quite enthusiastic about the experience and how it had made them think even more about the use of verb tenses and articles. Yesterday evening I received a message from a student who was convinced that there was another error in my quiz, but had actually been getting the answer to question 1 wrong, like several of her coursemates, due to interference from Italian (they put a definite article before a percentage). Today she said she is unlikely to ever make that error again, which is positive from a formative viewpoint, although it does suggest the feedback I gave to that particular question wasn’t clear enough.

All told, I think there’s going to be an element of trial and error with any new tool, but learning from mistakes can be very beneficial.