1. Determine level-of-learning objectives.
2. List indicators of desired behaviors.
3. Establish criterion objectives.
4. Develop criterion-referenced test items.
This process is useful for tests that apply to the cognitive and affective domains of learning, and also can be used for skill testing in
the psychomotor domain. The development process for criterion-referenced tests follows a general-to-specific pattern. [Figure 6-5]
Figure 6-5. The development process for criterion-referenced tests follows a general-to-specific pattern.
Instructors should be aware that authentic assessment may not be as useful as traditional assessment in the early phases of training,
because the learner does not have enough information about the concepts or knowledge to participate fully. As discussed in Chapter
3, The Learning Process, when exposed to a new topic, learners first tend to acquire and memorize facts. As learning progresses, they
begin to organize their knowledge to formulate an understanding of the things they have memorized. When learners possess the
knowledge needed to analyze, synthesize, and evaluate (i.e., application and correlation levels of learning), they can participate more
fully in the assessment process.
Determine Level-of-Learning Objectives
The first step in developing an appropriate assessment is to state the individual objectives as general, level-of-learning objectives. The
objectives should measure one of the learning levels of the cognitive, affective, or psychomotor domains described in Chapter 3. The
levels of cognitive learning include knowledge, comprehension, application, analysis, synthesis, and evaluation.
For the understanding level, an objective could be stated as, “Describe how to perform a compression test on an aircraft reciprocating
engine.” This objective requires a learner to explain how to do a compression test, but not necessarily perform a compression test
(application level). Further, the learner would not be expected to compare the results of compression tests on different engines
(application level), design a compression test for a different type of engine (correlation level), or interpret the results of the
compression test (correlation level). A general level- of-learning objective is a good starting point for developing a test because it
defines the scope of the learning task.
List Indicators/Samples of Desired Behaviors
The second step is to list the indicators or samples of behavior that give the best indication of the achievement of the objective. The
instructor selects behaviors that can be measured and which give the best evidence of learning. For example, if the instructor expects
the learner to display the understanding level of learning on compression testing, some of the specific test question answers should
describe appropriate tools and equipment, the proper equipment setup, appropriate safety procedures, and the steps used to obtain
compression readings. The overall test must be comprehensive enough to give a true representation of the learning to be measured. It
is not usually feasible to measure every aspect of a level of learning objective, but by carefully choosing samples of behavior, the
instructor can obtain adequate evidence of learning.
Establish Criterion Objectives
The next step in the test development process is to define criterion (performance-based) objectives. In addition to the behavior
expected, criterion objectives state the conditions under which the behavior is to be performed, and the criteria that must be met. If
the instructor developed performance-based objectives during the creation of lesson plans, criterion objectives have already been
formulated. The criterion objective provides the framework for developing the test items used to measure the level of learnin g
objectives. In the compression test example, a criterion objective to measure the understanding level of learning might be stated as,
“The learner will demonstrate understanding of compression test procedures for reciprocating aircraft engines by completing a quiz
with a minimum passing score of 70 percent.”
Develop Criterion-Referenced Assessment Items
The last step is to develop criterion-referenced assessment items. The development of written test questions is covered in the
reference section. While developing written test questions, the instructor should attempt to measure the behaviors described in the
criterion objective(s). The questions in the exam for the compression test example should cover all of the areas necessary to give
evidence of understanding the procedure. The results of the test (questions missed) identify areas that were not adequately covered.
Performance-based objectives serve as a reference for the development of test items. If the test is the pre-solo knowledge test, the
objectives are for the learner to understand the regulations, the local area, the aircraft type, and the procedures to be used. The test
should measure the learner’s knowledge in these specific areas. Individual instructors should develop their own tests to measure the
progress of their learners. If the test is to measure the readiness of a learner to take a knowledge test, it should be based on the
objectives of all the lessons the learner has received.
Aviation training also involves performance tests for maneuvers or procedures. The flight instructor does not administer the practical
test for a pilot certificate, nor does the aviation maintenance instructor administer the oral and practical exam for certification as an
aviation maintenance technician (AMT). However, aviation instructors do get involved with the same skill or performance testing that
is measured in these tests. Performance testing is desirable for evaluating training that involves an operation, a procedure, or a
process. The job of the instructor is to prepare the learner to take these tests. Therefore, each element of the practical test should be
evaluated prior to sending an applicant for the practical exam.
Practical tests for maintenance technicians and pilots are criterion-referenced tests. The practical tests, defined in the Airman
Certification Standards and Practical Test Standards (ACS/PTS), are criterion referenced because the objective is for all successful
applicants to meet the high standards of knowledge, skill, and safety required by the regulations. The purpose of the ACS/PTS is to
delineate the standards by which FAA inspectors, designated pilot examiners (DPEs), and designated maintenance examiners (DMEs)
conduct tests for ratings and certificates. The standards reflect the requirements of Title 14 of the Code of Federal Regulations (14
CFR) parts 61, 65, 91, and other FAA publications, including the Aeronautical Information Manual (AIM) and pertinent advisory
circulars and handbooks.
The objectives in the ACS/PTS ensure the certification of pilots and maintenance technicians at a high level of performance and
proficiency, consistent with safety. The ACS/PTS for aeronautical certificates and ratings include areas of operation and tasks that
reflect the requirements of the FAA publications mentioned above. Areas of operation define phases of the practical test arranged in a
logical sequence within each standard. They usually begin with preflight preparation and end with postflight procedures. Tasks are
titles of knowledge areas, flight procedures, or maneuvers appropriate to an area of operation. Included are references to th e
applicable regulations or publications. Private pilot applicants are evaluated in all tasks of each area of operation. Flight instructor
applicants are evaluated on one or more tasks in each area of operation. In addition, certain tasks are required to be covered and are
identified by notes immediately following the area of operation titles.
Since evaluators may cover every task in the ACS/PTS on the practical test, the instructor should evaluate all of the tasks before
recommending the maintenance technician or pilot applicant for the practical test. While this evaluation is not necessarily formal, it
should adhere to criterion-referenced testing.
Critiques and Oral Assessments
Used in conjunction with either traditional or authentic assessment, the critique is an instructor- to-learner assessment. These methods
can also be used either individually, or in a classroom setting.
The word critique sometimes has a negative connotation, and the instructor needs to avoid using this method as an opportunity to be
overly critical of learner performance. An effective critique considers good as well as bad performance, the individual parts ,
relationships of the individual parts, and the overall performance. A critique can and usually should be as varied in content as the
performance being evaluated.
A critique may be oral, written, or both. It should come immediately after a learner’s performance, while the details of the
performance are easy to recall. An instructor may critique any activity a learner performs or practices to improve skill, proficiency,
and learning. A critique may be conducted privately or before the entire class. A critique presented before the entire class can be
beneficial to every learner in the classroom, as well as to the learner who performed the exercise or assignment. In this case, however,
the instructor should avoid embarrassing the learner in front of the class.
There are several useful ways to conduct a critique.
Instructor/Learner Critique
The instructor leads a group discussion in an instructor/learner critique in which members of the class are invited to offer criticism of
a performance. This method should be controlled carefully and directed with a clear purpose. It should be organized, and not allowed
to degenerate into a random free-for-all.
Learner-Led Critique
The instructor asks a learner to lead the assessment in a learner-led critique. The instructor can specify the pattern of organization and
the techniques or can leave it to the discretion of the chosen leader. Because of the inexperience of the participants in the lesson area,
learner-led assessments may not be efficient, but they can generate learner interest and learning.
Small Group Critique
For the small group critique, the class is divided into small groups, each assigned a specific area to analyze. Each group presents its
findings to the class. It is desirable for the instructor to furnish the criteria and guidelines. The combined reports from the groups can
result in a comprehensive assessment.
Individual Learner Critique by Another Learner
The instructor may require another learner to present the entire assessment. A variation is for the instructor to ask a number of
learners questions about the manner and quality of performance. Discussion of the performance and of the assessment can often allow
the group to accept more ownership of the ideas expressed. As with all assessments incorporating learner participation, it is important
that the instructor maintain firm control over the process.
Self-Critique
A learner critiques personal performance in a self-critique. Like all other methods, a self-critique receives control and supervision
from the instructor.
Written Critique
A written critique has three advantages. First, the instructor can devote more time and thought to it than to an oral assessment in the
classroom. Second, learners can keep written assessments and refer to them whenever they wish. Third, when the instructor asks all
learners to write an assessment of a performance, the learner-performer has the permanent record of the suggestions,
recommendations, and opinions of all the other learners. The disadvantage of a written assessment is that other members of the class
do not benefit.
Whatever the type of critique, the instructor should resolve disagreements and correct erroneous impressions. The instructor also
makes allowances for the learners’ relative inexperience. Normally, the instructor should reserve time at the end of the lear ner
assessment to cover those areas that might have been omitted, not emphasized sufficiently, or considered worth repeating.
Oral Assessment
The most common means of assessment is direct or indirect oral questioning of learners by the instructor. Questions may be loosely
classified as fact questions and HOTS questions. The answer to a fact question is based on memory or recall. This type of question
usually concerns who, what, when, and where. HOTS questions involve why or how and require the learner to combine knowledge o f
facts with an ability to analyze situations, solve problems, and arrive at conclusions.
Proper quizzing by the instructor can have a number of desirable results, such as:
1. Reveals the effectiveness of the instructor’s training methods
2. Checks learner retention of what has been learned
3. Reviews material already presented to the learner
4. Can be used to retain learner interest and stimulate thinking
5. Emphasizes the important points of training
6. Identifies points that need more emphasis
7. Checks comprehension of what has been learned
8. Promotes active learner participation, which is important to effective learning
Characteristics of Effective Questions
The instructor should devise and write pertinent questions in advance. One method is to place them in the lesson plan. Prepared
questions merely serve as a framework, and as the lesson progresses, should be supplemented by such impromptu questions as th e
instructor considers appropriate. Objective questions have only one correct answer, while the answer to an open-ended HOTS
question can be expressed in a variety of possible solutions.
To be effective, questions must:
1. Apply to the subject of instruction.
2. Be brief and concise, but also clear and definite.
3. Be adapted to the ability, experience, and stage of training of the learners.
4. Center on only one idea (limited to who, what, when, where, how, or why, not a combination).
5. Present a challenge to the learners.
Types of Questions to Avoid
Effective quizzing does not include yes/no questions such as “Do you understand?” or “Do you have any questions?” Instructors
should also avoid the following types of questions:
⦁ Puzzle—“What is the first action you should take if a conventional gear airplane with a weak right brake is
swerving left in a right crosswind during a full flap, power-on wheel landing?”
⦁ Oversize—“What do you do before beginning an engine overhaul?”
⦁ Toss-up—“In an emergency, should you squawk 7700 or pick a landing spot?”
⦁ Bewilderment—“In reading the altimeter—you know you set a sensitive altimeter for the nearest station
pressure—if you take temperature into account, as when flying from a cold air mass through a warm front,
what precaution should you take when in a mountainous area?”
⦁ Trick questions—these questions cause the learners to develop the feeling that they are engaged in a battle
of wits with the instructor, and the whole significance of the subject of the instruction involved is lost. An
example of a trick question would be one in which the response options are 1, 2, 3, and 4, but they are
placed in the following form:
A. 4
B. 3
C. 2
D. 1
⦁ Irrelevant questions—diversions that introduce only unrelated facts and thoughts and slow the learner’s
progress. Questions unrelated to the test topics are not helpful in evaluating the learner’s knowledge of the
subject at hand. An example of an irrelevant question would be to ask a question about tire inflation during a
test on the timing of magnetos.
Answering Learner Questions
Tips for responding effectively to learner questions, especially in a classroom setting:
1. Be sure that you clearly understand the question before attempting to answer.
2. Display in terest in the learner’s question and frame an answer that is as direct and accurate as possible.
3. After responding, determine whether or not the learner is satisfied with the answer.
Sometimes it is unwise to introduce considerations more complicated or advanced than necessary to completely answer a learner ’s
question at the current point in training. In this case, the instructor should carefully explain to the learner that the question was good
and pertinent, but that a detailed answer would, at this time, unnecessarily complicate the learning tasks. The instructor should invite
the learner to reintroduce the question later at the appropriate point in training.
Occasionally, a learner asks a question that the instructor cannot answer. In such cases, the instructor should freely admit not knowing
the answer, but should promise to get the answer or, if practicable, offer to help the learner look it up in available references.
Scenario-Based Training
Since flying has become more complex, the focus of training needs to address pilot decision-making and risk management, a concept
the authors call Single Pilot Resource Management (SRM). Since SRM training requires practicing the decision-making process in
real time and in real situations, a new form of GA training that goes beyond the traditional task and maneuver-based training is
recommended. The FITS program emphasizes combining traditional task and maneuver training with Scenario Based Training (SBT)
to teach the advanced pilot judgment and risk management skills required in the SRM environment.
In many respects, scenario-based training is not a new concept. Experienced flight instructors have been using scenarios to teach cross
country operations, emergency procedures, and other flight skills for years. Scenario-based training puts the learner pilot into the
normal cross-country environment much earlier than traditional flight training programs. The goal is to begin training the pilot,
through meaningful repetition, how to gather information and make informed and timely decisions. We routinely refer to this
process as “experience.” Scenario-based learning does not preclude traditional maneuver-based training. Rather, flight maneuvers are
integrated into the flight scenarios and conducted, as they would occur in the real world. Those maneuvers requiring repetition may
still be taught during concentrated settings. However, once they are learned, they are then integrated into realistic flight situations.
The flight instructor is crucial to the implementation of a scenario-based training program. In this capacity, an instructor serves in the
learning environment as an advisor and guide for the learner. The duties, responsibilities, and authority of the flight instructor include
the following:
⦁ Orient new learners to the scenario-based training system.
⦁ Help the learner become a confident planner and in-flight manager of each flight and a critical evaluator of
their own performance.
⦁ Help the learner understand the knowledge requirements present in real world applications.
⦁ Diagnose learning difficulties and help the individual overcome them.
⦁ Be able to evaluate learner progress and maintain appropriate records.
⦁ Provide continuous review of learning.
The flight instructor is the key to success, and different instructional techniques are required for successful SBT. Remembering that
the learning objective is for the learner to be more ready to exercise sound judgment and make good decisions; the flight instructor
turns the responsibility for planning and execution of the flight over to the learner as soon as possible. The flight instructor will
continue to demonstrate and instruct skill maneuvers in the traditional manner, however, when the learner begins to make decisions
the flight instructor should revert to the role of mentor and/or learning facilitator.
Each situation a learner faces may not have one right, and one wrong answer. Instead, learners will encounter situations in training
that may have several "good" outcomes as well as the potential for some "poor" ones. While the flight instructor should understand in
advance which outcomes are positive or negative, the instructor allows the learner to make both good and poor decisions. This does
not mean that the learner should be allowed to make an unsafe decision or commit an unsafe act. However, it does allow the learner to
make decisions that fit their experience level and result in positive outcomes.
Chapter Summary
This chapter described the methods and techniques aviation instructors use to assess learner progress. A well-designed and timely
assessment defines what is worth knowing, enhances motivation, and improves learning. Today’s learners want to know the crite ria
by which they are assessed, and they want practical and specific feedback. Aviation instructors familiar with different
types of assessments know how and when to use them to promote a productive learning environment.