Chapter 8A: Report Manager – Examinee and Item Reports

Table of Contents

 

Informative reports are essential to the management of an assessment program. The fourth tab (Figure 8.1) of FastTest provides functionality to analyze important aspects, providing the information needed to make decisions. For example, a summary report of the item bank can be easily produced, which highlights areas of the bank most in need of new items. Reports that are used often can be saved for quick access at later date. Continuing with the same example, the item bank summary report could be saved, and re-run weekly as item development continued to fill in weaker areas of the bank. Also, item, examinee responses, and examinee data that have recently been generated can be found and downloaded under Recent Exports.

Figure 8.1: Report Manager Tab

report mngr tab

There are four categories of reports available for use: examinees, items, tests, and users. Each category holds a number of reports, which are presented when the user clicks on the “Reports” button in the upper left (as shown in the image above). These are described below.

  • The following table provides descriptions for various reports

report descriptions

  • The following table details which reports are available to each user role

 

user role access for reports

Examinee Reports

Examinee Test Detail

This report gives an item-by-item summary of an examinee’s test. For conventional tests, each item is shown along with the score that the examinee received for that item. For CAT tests, the values of the θ estimate and standard error on an item-by-item basis are also displayed. The IRT parameters for each item will be shown, as well. If the CAT test uses Content Constraints, the subscore that each item was selected from will appear as well.

Figure 8.3: Examinee Test Detail Snippet

examinee test detail snippet

Examinee Group

The Examinee Group report summarizes all of the examinee results in a given group across all tests. It is separated into two tables. In the first table, each subgroup-test pair is listed alongside the number of examinees, the average score, and the passing rate. In the second table, data is aggregated over all examinees in the selected group and its subgroups. Each test that any of the examinees have taken appears alongside the number of examinees, the average score, and the passing rate.

Examinee Group and Test Summary

The Examinee Group and Test Summary Report summarizes the scores of all examinees in a given group that have taken a given test. If the test has a cutscore, it will show whether each examinee passed and calculate the overall passing rate. If the test has subscores, it will report the average score for each subscore.

Note: This report will include examinees from the selected group as well as all subgroups of the selected group.

Item Reports

Cross-Test Item Analysis

In the classical test theory paradigm, item statistics are usually analyzed within a single test. However, in some situations it might be useful to know some statistics for an item across all tests on which it has been used. This report provides a snapshot of the item, including statistics that are recorded in the bank, as well as the number and proportion of examinees selecting each option across all tests. The Cross-Test Item Analysis Report can be especially useful when an item has been slated for review or possible retirement. For example, if the actual P value was 0.68 it would be much lower than the recorded value of 0.97, suggesting that review is appropriate.

Figure 8.4: Cross-Test Item Report Snippet

cross test item analysis

Bank Summary

The Bank Summary Report is designed to provide a complete summary of the items in a bank, which describes the status of the bank and helps to direct future development. The first two tables in this report are counts of item status and item type in each of the categories, as shown below. This helps to identify categories that have the fewest items and are in the greatest need of new items. It can also identify the categories with the most items up for review, and therefore used to direct the reviewers.

Figure 8.5: Bank Summary Report Snippet

bank summary

Additional tables in this report provide frequency distributions of item usage (number of tests on which the item is used), item owners (to see which item writers are most productive), and item statistics. The final section of this report provides a simple list of the folder structure. This is useful because it can be used to copy an item bank structure without all of the items. For example, a previous year’s item bank structure could be copied from this report, and pasted into a new bank for the next year, which would automatically create all the same folders.

Item History

The Item History report provides a history of major actions on the item, such as comments and changes that bring about new versions (text and key changes). This is also useful during the item review process.

Figure 8.6: Item History Report Snippet

item history

Chapter 8B: Report Manager – Test Reports, User Reports, and Running Reports

Table of Contents

Test Summary

The Test Summary Report covers important aspects of a test. First, it provides a summary of the status of the test, such as the number of examinees assigned and the number/percent completed so far, and the number of printed versions available. The second part provides the Response Type breakdown of the test. The third part provides the test development statistics (Figure 4.13) in the format of a report that can be provided to stakeholders.

Figure 8.7: Test Summary Report Snippet
test-sum4

Test Group Summary

The Test Group Summary report summarizes the test-level aspects of all the tests in a test group. For example, the table below shows a group of seven Algebra tests, four of which are protected and being administered to students, while three are not yet in use.

Figure 8.8: Test Group Summary Report Snippet

test group summary

 

Test and Item Analysis Report

The Test and Item Analysis Report (TIAR) is a report designed to allow you to evaluate the psychometric properties of your test utilizing the tried-and-true paradigm of Classical Test Theory.  This report is run from the Reporting tab of FastTest, where you select which test form you would like to analyze, and (optionally) specify a date range or implement a spuriousness correction for discrimination statistics.

The spuriousness correction serves to correct the item-total point-biserial for the fact that the total score contains scores from the item itself, and therefore you are correlating the item with itself in some small part, which obviously inflates the correlation.  The correction calculates the total score without that particular item, so that item scores can then be correlated with the total score more fairly. Note that as the test contains more items, the contribution of any one item is smaller and smaller, and the spurious correction becomes negligible.  However, it can be very important in short tests, such as 20 items or fewer.

The report is divided into two primary sections.  The first section provides an overview of the test as a whole.  It includes simple statistics such as the number of examinees scheduled for this test, the number that actually completed the test, and the number of total/scored items on the test.  It also provides statistics on the distribution of scores, including the mean (average), standard deviation, variance, minimum (lowest score observed), and maximum (highest score observed).

Arguably the most important statistic in this table is the coefficient alpha estimate of internal consistency reliability, which is a generalization of the KR20 reliability index.  More information on these can be found in any psychometric textbook, but it can be broadly interpreted as the reliability (consistency) of our measurement process on a scale of 0.0 to 1.0, with 0.0 being random numbers and 1.0 being perfectly reliable measurement.  The example below is a relatively accurate test, as the reliability index is 0.94.

The standard deviation and the reliability are also combined into a single index called the standard error of measurement (SEM).  This is an estimate of the variation in observed scores if a person were to take the test an infinite number of times.  A small number translates to greater precision in the scores.

Figure 8.9

TIAR report

FastTest also provides a complete statistical evaluation of each item and its answers.  The example below is a typical 4-option multiple choice item.  The Weight column displays the points awarded for each answer, in this case just 1 point for the correct answer only.  The N column shows the number of examinees that selected each answer, and the Diff (P) column converts that to a proportion (called “difficulty” but typically denoted statistically by P).  In this example, 37% of the sample selected “candy” (which is the correct answer) but 37% also selected “confetti.”  The Disc (r) column displays the discrimination value, as indexed by the point-biserial correlation (Rpbis), which correlates responses on that answer to the total score on the test.  A positive value means that examinees of higher ability are selecting that answer.  This means, in this example, that the 37% of candidates that selected “candy” were of high ability (Disc=0.49) because they knew the correct answer, while the 37% that selected “confetti” were of low ability (-0.22) as they did not know the correct answer.

The Diff and Disc for the correct answer also serve as the overall Diff and Disc.

This table is absolutely essential in diagnosing the possibly reasons that an item is performing poorly.  In this example we have an extremely difficult item, but we know it is still high quality because the Disc value is so high (>0.20 is typically “good”).  If the Disc was close to 0.0, that would mean that examinees found the item confusing, as there was no clear correct answer.  In rare cases, you might find that an answer thought to be incorrect has a high Disc value, meaning that the examinees tend to think it was correct.  That item should obviously be reviewed by experts.

Figure 8.10

  itar2

User Reports

User Activity

The User Activity report is available to Workspace Administrators for evaluating the productivity of users in the workspace, both at an item and test level, as shown below.

Figure 8.11: User Activity Report Snippet

user activity snippet

 

  • User List
    • The User List report is available to Workspace Administrators for generating a list of users across workspaces. The report can be filtered at three levels:
      • User role
      • Active/Expired workspace
      • Account status

 

  • The report will provide:
    • User’s name
    • Workspace
    • Role
    • Email
    • Phone number
    • Last login

Figure 8.12 User List Report

user lists

 

  • Who’s Logged In
    • This report will provide a list of all users currently logged into the workspace(s), and is only accessible to Workspace Administrators.

 

Workspaces Reports

  • Content Hierarchy Report
    • This report will generate an XLXS file with the current content hierarchy for the workspace. An example is shown below.

 

Figure 8.13: Content Hierarchy Report Example

content hierarchy report

 

  • Test Credit Audit Summary
    • This report summarizes the workspace’s test credit usage, and is only available to Workspace Administrators.

Figure 8.14: Test Credit Audit Summary Example

 test credit1

test credit2

  • Test Session Usage by Workspace (Detail)
    • This report details which tests have been taken by which workspace. An example report is provided below

Figure 8.15: Test Session Usage by Workspace (Detail) Example

test session usage report

  • Test Session Usage by Workspace (Summary)
    • This report summarizes which tests have been taken by which workspace. An example report is provided below

Figure 8.15: Test Session Usage by Workspace (Summary) Example

test session usage summary

Running Reports

To run a report, select the desired report from the drop-down menu. Information will then appear in the center of the screen regarding the relevant parameters of the report. For example, the User Activity report simply provides a list of user activity in a workspace for a given date range, so boxes will appear for a Start Date and End Date. For an Item History Report, the category structure of the banks will be shown, allowing the user to navigate to the desired item, as shown in the image below.

Figure 8.16: Browsing Bank for Item History Report

item history report

The Reporting module also allows you to specify the file format of the report. HTML, PDF, DOCX, RTF, XLS, and TXT formats are supported across the reports. For the User Activity Report, a CSV format replaces the TXT format as an option. Once you have selected the format, click Run to produce the report, Save to save the report configurations, or Reset to start over. Saving the report will keep it in the pane on the right-hand side of the screen for quicker access in the future, if it is a report that will be used often.

Figure 8.17: Report Options

reports options