All question types have the same common properties; General, Question, Stats, Feedback, Link to Objective and Link to Analysis. The only difference between them is the Question page.
Questions also have these common pages: Cross Reference, Document Links, Version Comments, Past Versions.
General
The General page item is the first menu item to be filled in. This menu item is common to all of the question types.
•Always select from question pool: Check the box if you want to ensure that this question is always selected on any exam that contains it in its question pool.
•Authorized for practice: Check the box if you want this question to be eligible for use on practice tests in the Learning Station. Right-click on the checkbox for more information.
•Time to Complete and Difficulty: You may find it useful to enter estimates for these values since it is from these that an exam's total difficulty or time to complete can be computed.
The topic is initially set to the objective behavior statement. You can edit the topic here. If you don’t, then when you save the question, VISION will offer you the opportunity to edit it then or copy the question stem into the topic.
This brief description will appear on several lists and on various screens you will use in the process of generating a test from the test question bank. A brief one-liner will make it easier to convey the essence of the test question in less space than the complete question text.
Selecting the status of a test question
Select the status that reflects the current stage of development. Keep in mind that a question cannot be used on a test until its status is set to Active.
When you change the question’s Status to ‘Active’, VISION will check the question, and, if it finds any missing components, display an error message with the missing items and instructions on what to do next.
•If you select Active, VISION activates the question and allows it to be selected on a test.
•You can also add or delete statuses by selecting "(Edit List)" if you have permission to do so though the VISION Security program.
•The status you choose now, while creating the question, can always be changed later.
Note that VISION will not let you set a question to Active status until it is complete. This means that it must have a question stem and an answer, as well as:
•Multiple Choice questions must have at least one distractor and one answer.
•Matching questions must have all items linked to at least one choice.
•Scenario questions must have at least one sub question.
A security event controls the ability to change a question’s status to or from Active. To view your security profile, select Project and then Security Profile from the Main Menu. Contact your system administrator to request any changes in your security profile.
Using the Comments field
Comments can be anything you feel might be important for documentation of the question. For example, you might want to make a note about a particular graphic or some particular content that should be addressed in the question because a co-worker has indicated that the question will be completed at a later time. This is a good place to make such a note. If you do not need this field, you can also change the “comments” label to something more useful by clicking on it.
When new test questions are entered, do not change the status of the question to Active on the General Info menu item until you are sure the question is truly ready for use on a test.
Question
Each Question page has settings specific to the type of question you have created. Select the question type below:
Stats
Every question type has a Stats page. After a test is administered, the learners’ responses can be entered for each question in the Test Statistics window. This happens automatically for tests administered through the VISION Learning Station. The question statistics page, illustrated below, shows the overall results from these recorded uses of the question.
You will find that as more test results are entered into the system, VISION will continually calculate, refine and update these statistics. The more students that respond to the question, the more reliable these statistics become.
Used together, these statistics should help you evaluate the performance of a question on tests. The statistics won’t tell you where the question is flawed or if the instruction is inadequate, but they can help you assess the technical adequacy of the question or the effectiveness of the instruction.
As you may know, the statistics outlined here are not the ones typically computed for norm-referenced or criterion-referenced tests. It is our belief that the statistics outlined here are not appropriate for many circumstances. In some cases it is impractical to collect the data that’s required for their computation, however, we encourage you to examine these statistics, assess their worth or meaning, and determine their adequacy for your testing environment.
Note that you have to click on the Question Stats and Exam Usage buttons to view these stats.
We’ll take a closer look at the Question Stats that are on the Stats page. For each of the values described next, note that there are two boxes: the first box shows the overall value, while the second box shows just the value for the exam you selected in the bottom panel.
Learner Encounters
The Learner Encounters field displays the total number of students that have seen and responded to the question (excluding responses marked as "ignore" by the test administrator). For example, if this question appeared on 10 tests, and there were 20 learners who took each test, this field would contain 200.
Current Point Value
The Points field displays the number of points the question is worth. It is displayed only for the instructor’s convenience, and is not a computed statistic; it is simply the point value assigned by the question author.
All tests that this question has been used on, except tests that are private or otherwise unviewable by you, are shown in the Test History pane in the lower portion of the Stats menu item.
Average Score
This field displays the average score, which is calculated by adding all scores for the question and then dividing the total by the number of students who responded to the question.
% Average Score
This field displays the % average score, which is used primarily as a cutoff to flag items for analysis. This number is computed by dividing the question’s average score by the question’s point value.
Difficulty
This is merely an interpretation of the percent average score. The percent average score is converted by an equation into a value with the range 0 to 6, which represents the “difficulty”.
Standard Deviation
This field displays the standard deviation, which tells you the variability (or spread) of the scores around a mean. The benefit of reviewing this statistic is that it allows you to flag questions that may need to be modified. If the standard deviation is high, you may want to review the question. The formula for standard deviation is:
For questions with only two possible scores (such as 0 and 1), the standard deviation may not be a very useful metric.
A large number in this field could also mean that the objective content might not have been understood by the students. You might need to review the content for the objective that the question is linked to and then make modifications as necessary.
The following two statistics apply ONLY to Multiple Choice questions:
•Correct Answer: This field displays the number of times the correct answer was selected.
•Distractors: This list shows all of the distractors (wrong answers) for the question. Next to each distractor is a number that represents the number of times the distractor was selected by the learner on a test.
Using the answer and distractor data, you should be able to identify multiple choice questions that are not reliable. Look for: distractors that are picked disproportionately more often than other distractors; distractors that are rarely or never picked; and right answers that are rarely selected, or too seldom missed.
Exam Usage
All tests that the question has been used on are shown in this field. Right-click on the field for more information.
Feedback
If your site uses VISION Web, every question type has a Feedback page. In the VISION Learning Station, learners are given the opportunity to leave feedback regarding their test questions. This is collected and presented on the Feedback page.
Including archived feedback
Check this box to show or hide learner feedback that has been set as archived in the "Feedback entries" box.
Feedback entries
Feedback left by learners in the Learning Station are shown in this field, along with the exam ID on which they left it. Right-click on the field for more information.
Feedback
View the feedback selected above.
Resolution
Enter the action taken to "answer" or redress the selected feedback entry.
Link to Objective & Link to Analysis
Questions can be linked to more than one objective or Analysis hierarchy Skill/Knowledge node (if that feature is enabled for your project). These pages list the objectives and analysis nodes that the question is linked to.
