Aviation Instructor's Handbook (FAA-H-8083-9)
Chapter 6: Assessment
Introduction
Assessment is an essential component of teaching and learning, and it provides the instructor with immediate feedback on the quality
of instruction. Instructors continuously evaluate a learner’s performance in order to provide guidance, suggestions for improvement,
and positive reinforcement.
This chapter examines the instructor’s role when assessing levels of learning, it describes methods of assessment, and it discusses how
instructors may construct and conduct effective assessments. Since learners are different and each situation is unique,
instructors individualize an assessment as needed. Aviation instructors can use the techniques and methods described in this
chapter as appropriate.
Assessment Terminology
This chapter presents two broad categories of assessment. The first is traditional assessment, which often involves the kind of written
testing (e.g., multiple choice, matching) and grading that is most familiar to instructors and learners. To achieve a passing score on a
traditional assessment, the learner usually has a set amount of time to recognize or reproduce memorized terms, formulas, or data.
There is a single answer that is correct. Consequently, the traditional assessment usually assesses the learner’s progress at the rote and
understanding levels of learning. However, carefully crafted scenario questions can assess higher levels of learning.
The second category of assessment is authentic assessment. Authentic assessment requires the learner to demonstrate not just rote and
understanding, but also the application and correlation levels of learning. Authentic assessment generally requires the learner to
perform real-world tasks and demonstrate a meaningful application of skills and competencies. In other words, the authentic
assessment requires the learner to exhibit in-depth knowledge by generating a solution instead of merely choosing a response.
In authentic assessment, there are specific performance criteria, or standards, that learners know in advance of the actual assessment.
The terms “criteria/criterion” and “standard” are often used interchangeably. They refer to the characteristics that define acceptable
performance on a task. Another term used in association with authentic assessment is “rubric.” A rubric is a guide used to score
performance assessments in a reliable, fair, and valid manner. It is generally composed of dimensions for judging learner
performance, a scale for rating performances on each dimension, and standards of excellence for specified performance levels.
Whether knowledge or skill, an assessment can be either formal or informal. Formal assessments usually involve documentation, such
as a quiz or written examination. They are used periodically throughout a course, as well as at the end of a course, to measure and
document whether or not the course objectives have been met. Informal assessments, which can include verbal critique, generally
occur as needed and are not part of the final grade.
Other terms associated with assessment include diagnostic, formative, and summative.
⦁ Diagnostic assessments assess learner knowledge or skills prior to a course of instruction.
⦁ Formative assessments, which are not graded, provide a wrap-up of the lesson and set the stage for
the next lesson. This type of assessment, limited to what transpired during that lesson, informs the
instructor what to reinforce.
⦁ Summative assessments, used periodically throughout the training, measure how well learning has
rogressed to that point. For example, a stage-check, a chapter quiz, or an end-of-course test can measure
the learner’s o verall mastery of the training. These assessments are an integral part of the lesson, as well as
the course of training.
Purpose of Assessment
An effective assessment provides critical information to both the instructor and the learner. Both instructor and learner need to know
how well the learner is progressing. A good assessment provides practical and specific feedback to learners. This includes direction
and guidance indicating how they may raise their level of performance. Most importantly, a well-designed and effective assessment
provides an opportunity for self-evaluation that enhances the learner's aeronautical decision-making and judgment skills.
A well-designed assessment highlights the areas in which a learner’s performance is incorrect or inadequate, it helps the instructor see
where more emphasis is needed. If, for example, several learners falter when they reach the same step in a weight-and-balance
problem, the instructor should recognize the need for a more detailed explanation, another demonstration of the step, or special
emphasis in the assessment of subsequent performance.
General Characteristics of Effective Assessment
In order to provide direction and raise the learner’s level of performance, assessment needs to be factual, and it should align with the
completion standards of the lesson. An effective assessment displays the characteristics shown in Figure 6-1.
Figure 6-1. Effective assessments share a number of characteristics.
Objective
The personal opinions, likes, dislikes, or biases of the instructor might affect an assessment. A conflict of personalities can alter an
opinion. Sympathy or over- identification with a learner, to such a degree that it influences objectivity, is known as “halo error.” To
what extent does effective assessment need to focus on objectivity and actual learner performance? If an assessment is to be effective,
it needs to be honest; and it must be based on the facts of the performance as they were, not as they could have been.
Flexible
The instructor should evaluate the entire performance of a learner in the context in which it is accomplished. Sometimes a go od
learner turns in a poor performance, and a poor learner turns in a good one. A friendly learner may suddenly become hostile, or a
hostile learner may suddenly become friendly and cooperative. The instructor fits the tone, technique, and content of the assessment
to the occasion, as well as to the learner. An assessment should be designed and executed so that the instructor can allow fo r
variables. The ongoing challenge for the instructor is deciding what to say, what to omit, what to stress, and what to minimize at the
proper moment.
Acceptable
Consider that learners do not like negative feedback. What makes an honest assessment acceptable to the learner? A certificate or
credential alone rarely suffices. Learners need to have confidence in the instructor’s qualifications, teaching ability, sincerity,
competence, and authority. Usually, instructors have the opportunity to establish themselves with learners before the formal
assessment arises. If not, however, the instructor’s manner, attitude, and familiarity with the subject at hand serves this purpose.
Assessments presented fairly, with authority, conviction, sincerity, and from a position of recognizable competence tend to work well.
Comprehensive
A comprehensive assessment is not necessarily a long one, nor need it treat every aspect of the performance in detail. While it
includes strengths as well as weaknesses, the degree of coverage of each should fit the situation. The instructor might report what
most needs improvement, or only what the learner can reasonably be expected to improve. The instructor decides whether the g reater
benefit comes from a discussion of a few major points or a number of minor points.
Constructive
An assessment is pointless unless the learner benefits from it. Praise can capitalize on things that are done well and inspire the learner
to improve in areas of lesser accomplishment. When identifying a mistake or weakness, the instructor needs to give positive guidance
for correction. Praise for its own sake or negative comments that do not point toward improvement or a higher level of perfo rmance
should be omitted from an assessment altogether.
Organized
An assessment must be organized. Almost any pattern is acceptable, as long as it is logical and makes sense to the learner. A n
effective organizational pattern might be the sequence of the performance itself. Sometimes an assessment can begin at the point at
which a demonstration failed, and work backward through the steps that led to the failure. A success can be analyzed in similar
fashion. Alternatively, a glaring deficiency can serve as the core of an assessment. Breaking the whole into parts, or building the parts
into a whole, is another possible organizational approach.
Thoughtful
An effective assessment reflects the instructor’s thoughtfulness toward the learner’s need for self-esteem, recognition, and approval.
The instructor refrains from minimizing the inherent dignity and importance of the individual. Ridicule, anger, or fun at the expense
of the learner has no place in assessment. While being straightforward and honest, the instructor should always respect the l earner’s
personal feelings. For example, the instructor should try to deliver criticism in private.
Specific
The instructor’s comments and recommendations should be specific. Learners cannot act on recommendations unless they know
specifically what the recommendations are. A statement such as, “Your second weld wasn’t as good as your first,” has little
constructive value. Instead, the instructor should say why it was not as good and offer suggestions on how to improve the weld. If the
instructor has a clear, well-founded, and supportable idea in mind, it should be expressed with firmness and authority, and in terms
that cannot be misunderstood. At the conclusion of an assessment, learners should have no doubt about what they did well and what
they did poorly and, most importantly, specifically how they can improve.
Traditional Assessment
As defined earlier, traditional assessment generally refers to written testing, such as multiple choice, matching, true/false, fill in the
blank, etc. Learners typically complete written assessments within a specified time. There is a single, correct response for each item.
The assessment, or test, assumes that all learners should learn the same thing, and relies on rote memorization of facts. Responses are
often machine scored and offer little opportunity for a demonstration of the thought processes characteristic of critical thinking skills.
Traditional assessment lends itself to instructor centered teaching styles. The instructor teaches the material at a low level, and
the measure of performance is limited. In traditional assessment, fairly simple grading matrices such as shown in Figure 6-2 are used.
Due to this approach, a satisfactory grade for one lesson may not reflect a learner's ability to apply knowledge in a different situation.
Still, tests of this nature do have a place in the assessment hierarchy. Multiple choice, supply type, and other such tests are useful in
assessing the learner’s grasp of information, concepts, terms, processes, and rules—factual knowledge that forms the foundation
needed for the learner to advance to higher levels of learning.
Characteristics of a Good Written Assessment (Test)
Whether or not an instructor designs his or her own tests or uses commercially available test banks, it is important to know the
components of an effective test. (Note: This section is intended to introduce basic concepts of written-test design. Please see
Appendix A for testing and test-writing publications.)
Figure 6-2. Traditional grading.
A test is a set of questions, problems, or exercises intended to determine whether the learner possesses a particular knowledge or skill.
A test can consist of just one test item, but it usually consists of a number of test items. A test item measures a single objective and
calls for a single response. The test could be as simple as the correct answer to an essay question or as complex as completing a
knowledge or practical test. Regardless of the underlying purpose, effective tests share certain characteristics. [Figure 6-3]
Figure 6-3. Effective tests have six primary characteristics.
Reliability is the degree to which test results are consistent with repeated measurements. If identical measurements are obtained every
time a certain instrument is applied to a certain dimension, the instrument is considered reliable. The reliability of a written test is
judged by whether it gives consistent measurement to a particular individual or group. Keep in mind, though, that knowledge, skills,
and understanding can improve with subsequent attempts at taking the same test, because the first test serves as a learning device.
Validity is the extent to which a test measures what it is supposed to measure, and it is the most important consideration in test
evaluation. The instructor must carefully consider whether the test actually measures what it is supposed to measure. To estimate
validity, several instructors read the test critically and consider its content relative to the stated objectives of the instruction. Items that
do not pertain directly to the objectives of the course should be modified or eliminated.
Usability refers to the functionality of tests. A usable written test is easy to give if it is printed in a type size large enough for learners
to read easily. The wording of both the directions for taking the test and of the test items needs to be clear and concise. Graphics,
charts, and illustrations appropriate to the test items must be clearly drawn, and the test should be easily graded.
Objectivity describes singleness of scoring of a test. Essay questions provide an example of this principle. It is nearly impossible to
prevent an instructor’s own knowledge and experience in the subject area, writing style, or grammar from affecting the grade
awarded. Selection-type test items, such as true/false or multiple choice, are much easier to grade objectively.
Comprehensiveness is the degree to which a test measures the overall objectives. Suppose, for example, an AMT wants to measur e
the compression of an aircraft engine. Measuring compression on a single cylinder would not provide an indication of the entire
engine. Similarly, a written test must sample an appropriate cross-section of the objectives of instruction. The instructor makes certain
the evaluation includes a representative and comprehensive sampling of the objectives of the course.
Discrimination is the degree to which a test distinguishes the difference between learners and may be appropriate for assessment of
academic achievement. However, minimum standards are far more important in assessments leading to pilot certification. If necessary
for classroom evaluation of academic achievement, a test must measure small differences in achievement in relation to the objectives
of the course. A test designed for discrimination contains:
1. A wide range of scores
2. All levels of difficulty
3. Items that distinguish between learners with differing levels of achievement of the course objectives
Please see Appendix B for information on the advantages and disadvantages of multiple choice, supply type, and other written
assessment instruments, as well as guidance on creating effective test items.
Authentic Assessment
Authentic assessment asks the learner to perform real-world tasks and demonstrate a meaningful application of skills and
competencies. Authentic assessment lies at the heart of training today’s aviation learner to use critical thinking skills. Rather than
selecting from predetermined responses, learners must generate responses from skills and concepts they have learned. By using open-
ended questions and established performance criteria, authentic assessment focuses on the learning process, enhances the
development of real-world skills, encourages higher order thinking skills, and teaches learners to assess their own work and
performance.
Learner-Centered Assessment
There are several aspects of effective authentic assessment. The first is the use of open-ended questions in what might be called a
“collaborative critique,” which is a form of learner-centered grading. As described in the scenario that introduced this chapter, the
instructor begins by using a four-step series of open-ended questions to guide the learner through a complete self-assessment.
Replay—the instructor asks the learner to verbally replay the flight or procedure. While the learner speaks, the instructor listens for
areas where the account does not seem accurate. At the right moment, the instructor discusses any discrepancy with the learner. This
approach gives the learner a chance to validate his or her own perceptions, and it gives the instructor critical insight into the learner's
judgment abilities.
Reconstruct—the reconstruction stage encourages learning by identifying the key things that the learner would have, could have, or
should have done differently during the flight or procedure.
Reflect—insights come from investing perceptions and experiences with meaning, requiring reflection on the events. For example:
1. What was the most important thing you learned today?
2. What part of the session was easiest for you? What part was hardest?
3. Did anything make you uncomfortable? If so, when did it occur?
4. How would you assess your performance and your decisions?
5. How did your performance compare to the standards in the ACS?
Redirect—the final step is to help the learner relate lessons learned in this session to other experiences and consider how they might
help in future sessions. Questions might include:
⦁ How does this experience relate to previous lessons?
⦁ What might be done to mitigate a similar risk in a future situation?
⦁ Which aspects of this experience might apply to future situations, and how?
⦁ What personal minimums should be established, and what additional proficiency flying and/or training
ight be useful?
Any self-assessment stimulates growth in the learner’s thought processes and, in turn, behaviors. An in-depth discussion between the
instructor and the learner may follow, which compares the instructor’s assessment to the learner’s self-assessment. Through this
discussion, the instructor and the learner jointly determine the learner’s progress. The progress may be recorded on a rubric as part of
a training program. As explained earlier, a rubric is a guide for scoring performance assessments in a reliable, fair, and valid manner.
It is generally composed of dimensions for judging learner performance, a scale for rating performances on each dimension, an d
standards of excellence for specified performance levels.
The collaborative assessment process in learner- centered grading uses two broad rubrics: one that assesses the learner’s level of
proficiency on skill-focused maneuvers or procedures, and one that assesses the learner’s level of proficiency on single-pilot resource
management (SRM), which is the cognitive or decision-making aspect of flight training.
The performance assessment dimensions for each type of rubric are as follows:
Maneuver or Procedure “Grades”
⦁ Describe—at the completion of the scenario, the learner is able to describe the physical characteristics and
cognitive elements of the scenario activities but needs assistance to execute the maneuver or procedure
successfully.
⦁ Explain—at the completion of the scenario, the learner is able to describe the scenario activity and
understand the underlying concepts, principles, and procedures that comprise the activity, but needs
assistance to execute the maneuver or procedure successfully.
⦁ Practice— at the completion of the scenario, the learner is able to plan and execute the scenario. Coaching,
instruction, and/or assistance will correct deviations and errors identified by the instructor.
⦁ Perform— at the completion of the scenario, the learner is able to perform the activity without instructor
assistance. The learner will identify and correct errors and deviations in an expeditious manner. At no time
will the successful completion of the activity be in doubt. (“Perform” is used to signify that the learner is
satisfactorily demonstrating proficiency in traditional piloting and systems operation skills).
⦁ Not observed—an y event not accomplished or required.
For example, a learner can describe a landing and can tell the flight instructor about the physical characteristics and appearance of the
landing. On a good day, with the wind straight down the runway, the learner may be able to practice landings with some success while
still functioning at the rote level of learning. However, on a gusty crosswind day the learner needs a deeper level of understanding to
adapt to the different conditions. If a learner can explain all the basic physics associated with lift/drag and crosswind correction, he or
she is more likely to practice successfully and eventually perform a landing under a wide variety of conditions.
Assessing Risk Management Skills
⦁ Explain—the learner can verbally identify, describe, and understand the risks inherent in the flight scenario,
but needs to be prompted to identify risks and make decisions.
⦁ Practice—the learner is able to identify, understand, and apply SRM principles to the actual flight situation.
Coaching, instruction, and/or assistance quickly corrects minor deviations and errors identified by the
instructor. The learner is an active decision maker.
⦁ Manage-Decide— the learner can correctly gather the most important data available both inside and outside
the flight deck, identify possible courses of action, evaluate the risk inherent in each course of action, and
make the appropriate decision. Instructor intervention is not required for the safe completion of the flight.
In SRM, the learner may be able to describe basic SRM principles during the first flight. Later, he or she is able to explain how SRM
applies to different scenarios that are presented on the ground and in the air. When the learner actually begins to make quality
decisions based on good SRM techniques, he or she earns a grade of manage-decide. The advantage of this type of grading is that
both flight instructor and learner know exactly where the learning has progressed.
Let’s look at how the rubric in Figure 6-4 might be used in a flight training scenario. During the postflight debriefing, flight instructor
Linda asks her learner, Brian, to assess his performance for the day using the Replay, Reconstruct, Reflect, and Redirect guided
discussion questions described in the Learner-Centered Assessment section presented earlier in this chapter. Based on this
assessment, she and Brian discuss where Brian’s performance falls in the rubrics for maneuvers/procedures and SRM. This part of the
assessment may be verbally discussed or, alternatively, Brian and Linda separately create an assessment sheet for each element of the
flight.
Figure 6-4. Rubric for assessing flight training maneuvers.
When Brian studies the performance levels, he decides he was at the “Perform” level since he had not made any mistakes. Where he
had rated the item as “Perform,” Linda had rated it as “Practice.” During the ensuing discussion, Brian understands where he needs
more practice before his performance is at the “Perform” level.
This approach to assessment has several key advantages. One is that it actively involves the learner in the assessment process and
establishes the habit of healthy reflection and self-assessment that is critical to being a safe pilot. Another is that these grades are not
self-esteem related, since they do not describe a recognized level of prestige (such as A+ or “Outstanding”), but rather a level o f
performance. The learner cannot flunk a lesson. Instead, he or she demonstrates a particular level of flight and SRM skills.
Both instructors and learners may initially be reluctant to use this method of assessment. Instructors may think it requires more time,
when in fact it is merely a more structured, effective, and collaborative version of a traditional postflight critique. Also, instructors
who learned in the more traditional assessment structure must be careful not to equate or force the dimensions of the rubric into the
traditional grading mold of A through F. One way to avoid this temptation is to remember that evaluation should be progressive: the
learner may achieve a new level of learning during each lesson. For example, in flight one, a task might be a “describe” item. By
flight three, it is a “practice” item, and by flight five, it is a “manage-decide” item.
The learner may be reluctant to self-assess if he or she has not had the chance to participate in such a process before. Therefore, the
instructor may need to teach the learner how to become an active participant in the collaborative assessment.
Choosing an Effective Assessment Method
When deciding how to assess learner progress, aviation instructors can follow a four-step process.