How Vignettes Are Developed

Vignette Development
Bill Johnson

William K. Johnson

Manager, Examination Development, NCARB

In our first Demystifying the ARE post, we took a look at who develops ARE questions. Next, we broke down how a question makes its way onto the exam. This week we’ll look at how a vignette is developed and becomes a part of the exam.

As with written questions, it takes about two years for a vignette to move through the development process. Here are the steps involved:

  1. Deciding what vignettes to write.
    There are a total of 11 vignettes for all seven exam divisions. NCARB has several different versions (we call them scripts) of each vignette. Scripts are randomly assigned when you schedule an exam, so you will never see the same script twice. Each year, NCARB conducts an analysis of the vignette pool to determine which vignettes are running low on potential scripts. These areas are then assigned to our volunteers, all licensed architects, who author new scripts.

  2. Authoring new vignettes.
    Each vignette has a template authors must follow to ensure consistency and uniformity across scripts. Templates control everything from the number of rooms in a Building Layout script to how many trees can be on the site for a Site Design vignette. Templates are why there is always a chimney on the Roof Plan, always a clerestory on Structural Layout, and never more than one object to place in Site Grading.

  3. Turning draft vignettes into scripts.
    The completed template and associated script are then converted into a functioning program, which is tested by development professionals (TDPs) at Alpine Testing. This allows the custom software files to run in test centers. The new vignette is then officially designated the “A” isomorph (or parent version) of a script—more on this later.

  4. Initial pretesting of scripts.
    All new scripts are distributed to a committee of recently licensed architects (RLAs) who generate multiple solutions using the same software candidates will use in the test centers. The RLAs also complete detailed checklists for each new script to evaluate how they compare to a set of standards.
  5. Group review of pretested scripts.
    Results from the pretesting and checklists are examined by the Vignette Development sub-group to verify compliance with authoring templates and graphic standards. Each new script for a vignette is also compared with existing scripts to ensure scripts of equal difficulty are being developed. When a script survives this review, it gets sent out for even more pretesting. If the script performs well, which means the solutions produced are clean with accurate scoring and checklists revealed no necessary edits, then the script is put in the queue for “isomorphing” (creating similar versions). Scripts that don’t pass the quality control checks are revised or simply deleted.

  6. Creating additional script versions and more pretesting.
    Alpine TDPs morph each approved “A” script into three additional versions, (B, C, and D). These new versions are now ready for their own pretesting. While the four versions are related, each is unique and must be vetted to meet the same quality standards.

  7. Pretest results and checklists are analyzed.
    Final pretest solutions and checklists are again compiled and presented to the ARE Vignette Development & Grading subcommittee to ensure accuracy and to determine if the isomorphs are of equal difficulty. Again, if edits are needed to improve the script, then the script will go through additional pretesting. If it is clean, then it is marked to go live in the test channel.
Vignette Development Committee

Members of the FY14 ARE Vignette Development & Grading subcommittee.

Vignettes in the test channel are not forgotten; their performance is analyzed each year. Vignettes must perform within acceptable ranges of difficulty, neither too easy nor too difficult, and average length of time to complete. Those that don’t meet performance standards are reviewed by the Vignette Grading sub-group to see if some aspect of it can be modified to improve performance. If it is believed the vignette can be salvaged, it is modified and begins the pretesting process over. Poor performing scripts are retired.

In the next Demystifying the ARE post, we’ll look at how the ARE is scored. Stay tuned!

comments powered by Disqus