How to Create Thousands of Versions of a Test?

One of the difficulties of moving to a true mastery system is the logistics of multiple versions of a test.  How many different versions can you manage on a day to day basis?  When Aaron Sams and I first started implementing the Flipped-Mastery model, we had three paper versions of every test.  Students who struggled either memorized each test version or in some cases other students had taken cell phone pictures of the test and were cheating.   We were using Moodle as our learning management system.  We discovered it had the ability to administer tests, so we wondered if there was a way to solve the management of many different versions.  One day as I was reading posts on the Moodle forum, I stumbled upon a solution to my dilemma. When I realized what I had found, I literally did a dance in my living room.

Though we were using Moodle, what we did can also be done with other Learning Management Systems.  Quia and Fishtree gave me demos of their products and I have been impressed with how easy they are to use and how robust their analytics are.  I am also aware that Blackboard has this feature though I have not personally reviewed it.  I imagine that almost all learning management systems have this feature, but I am not certain so check with your system administrator.  So, instead of walking you through step by step with each system, I want to give you the big picture of how to write thousands of versions of a test which covers specific objectives.

Step 1:  Create a list of discrete objectives.  This is simply a list of what it is you want assessed.  This is good teaching and forms the basis for your questions to come.  Below is the list of objectives for our unit on Gas Laws.  GasLawObjectives

Step 2:  Create a bank of questions for each objective.  So for each objective listed, we usually wrote between eight and ten questions.

Step 3:  Let the Learning Management System/Web Tool randomly select questions.  This is where the magic happens.  Once you have all of your banks of questions written, you have the computer randomly select questions from each bank.  So if there are eight questions that assess objective one, then have it pick one question.  And for objective two, you might have it choose two questions out of your bank of twelve.  Repeat until you have “built” your test.

Step 4:  Students take the exam.  Each student will get their own exam and it will be different.  Some will have the same questions, but they will be generated randomly.


  • This requires access to computers for your students.  We had eight computers in each classroom.  And since students were not taking the exams at the same time, this was sufficient.
  • We came to realize that we didn’t need to have long tests.  A good exam can take a shorter period of time.
  • This a lot of work.  Writing good, meaningful questions that assess the same objective is hard.  We spent many hours creating questions but it was well worth the time spent.

What do you think? Do you have any strategies for mastery exams?  I would love to hear from other Flipped-Mastery teachers out there.

Portions of this blog post will be included in an upcoming article for Educational Leadership that I co-wrote with Aaron Sams.  Learn more about Educational Leadership by clicking HERE

10 Responses

  1. Pingback : How to make 1000's of Versions of an Exam Using Moodle - Turning Learning On Its Head

  2. I just started using Blackboard’s COURSEsites for my high school math courses for two major reasons. It is free and I can make random variables for my questions and choose how many different banks to make for each question. That way, I write one version of each question with variables and it makes the rest. Still figuring some of the ins and outs, but it looks promising enough to make my quizzes and practice quizzes.

    1. Ben: Nice thoughts. I have heard from several teachers who like coursesites. There really isn’t one learning management system that is best for everybody, but we each need to find the right tool for us. I’m glad you have found one that works for you. Keep up the great work on flipped class.

  3. Hey Jon, I’m working on moving on to a mastery based setup for my CS1 (Computing Science 1 Programming course in Python) for January and debating over which support tool to use.

    I am very familiar with Moodle (been using it since 2003) but have found my students engagement much higher if I create a Facebook Group for discussion. Moodle has a big advantage as well in setting up restricted access to assessments and gradebooks which I cannot seem to replicated easily in other ‘non-LMS’ setups (Google Apps perhaps).

    I’m leaning towards using Moodle for tracking learning objectives, progress and assessment and continue to leverage my students’ “always logged-into-facebook” behaviour as my communication mechanism.

    As always, I am welcome to advice. I also have a big task in front of me coaching a small to medium size group of teachers here starting next semester and will be hitting the forums/twitter looking for others in a similar coaching position for sharing advice and support.

    1. Facebook for discussion maybe, but objective tracking, etc maybe on moodle. I’m not sure there is a best way, but one that will be easy for you to manage and also easy for the students to access. Simplicity for everybody is the key.

  4. Simi Frost

    Do these assessments work well with tablets or smart phones? I do not have 8 computers. I only have two so I am trying to figure out how to make this work in my classroom. Maybe I can get in a few tablets, maybe they can take them on their phones, etc.

    1. For the most part. It depends on the LMS. We used the Moodle LMS and it worked great on all devices. That said and not all learning management systems are created equally. So check with yours to see if it plays well on other devices.

    2. I agree with Jon here, I’ve been running my own Moodle for myself and my colleagues and it works very well with mobile devices. *Not* with an “app” but just loading the Moodle via the integrated web browser on the tablet or phone.

  5. Pingback : Mastery Simplified: Five Tips to Make Mastery a Reality – Flipped Learning Simplified

  6. Pingback : EdTech, Teaching, and Learning Tweet Wrap for the Week Ending 03-26-16 — Emerging Education Technologies

Leave a Reply