Trouble with testing technicians

Sept. 5, 2012

I’m delighted to be certified as a technician by two industry organizations — NATE (North American Technician Excellence) for HVAC and BPI (Building Performance Institute) for building energy analysis. In my state of Colorado, there is no HVAC licensing. These certifications are the next best thing. 

On the other hand, I’m dismayed. Paper and pencil tests are not a good way to recognize a competent technician. These tests require two skills that have absolutely nothing to do with being a technician or with real life. One of these irrelevant skills is the ability to read long complicated questions. The other is more a matter of luck than skill. It is choosing the correct answer out of four inadequate answers.

These tests have nothing to do with the primary competency of being a technician —using brain, hands and tools to actually do something. That something is to install or fix a piece of equipment or analyze a building. To try to create a parallel, how would you feel knowing that your surgeon had no hands-on examinations, just pencil and paper tests about thinking about surgery? Yikes! Yet that’s where our industry is today.

Except for occasional references to an equipment manual, our profession requires little reading. Yet the tests not only require lots of reading, but of material written in a way that makes it very difficult to comprehend. 

Of course it’s OK to expect technicians to be able to read. But there are various levels of readability. Everything written has a readability level. It is easy to get information from a short clear sentence. It is much harder, and sometimes impossible, to get that same information, especially if it is mixed with other distracting information separated by lots of commas, from a long complicated sentence — see how this sentence is an example of that?

Several readability measurements have been around for many years. You can go online and Google “readability” or the FOG index or the Flesh scale. But what it comes down to is that the longer the sentences and the more ideas in the sentence, the harder it is to get the meaning. Depending upon the instrument, the writing gets a number rating or a grade level score. 

What level do you think would be appropriate for an HVAC technician to read, eighth grade or 20+?  What level do you think the tests are written at? Here are some clues. The Reader’s Digest magazine is written at the eighth grade reading level. I have 18 years of education, including master’s degrees in English and in industrial education, and I can’t make sense of many of the test questions. I don’t mean that I don’t know the answers. I mean that it’s unclear what the question is even asking.

What does the ability to read at an extremely high level have to do with being an HVAC technician or building analyst? Absolutely nothing. And that is why many of our competent technicians fail. We know that competent technicians often do not read well.  And yet as an industry we let ourselves be bullied into the idea that if a technician can’t read difficult material, he must not know how to look at a furnace. We fail the technician rather than question the test.

But here’s what pains me the most. These tests have the power to lower the already often fragile self esteem of our technicians. Many of our best technicians languished in the back row of traditional school classrooms. They were great in shop class, and not so good with reading and math.

As a society we judge intelligence based upon reading and math abilities. But those are just two of the types of intelligence. Hands-on/mechanical is another recognized type of intelligence. Our industry is extremely high in that under-appreciated ability.

One of my pleasures as a Honeywell product rep was to introduce a new device to a technician.  His hands would be almost twitching to touch it. Once he got it, his finger tips found every connector, knob and spring. He might not even look at the device.  Information flowed directly from hands to brain. How do I know this? He was asking questions all along: “What’s this screw for?”  Why did they put these terminals here?”

On the other hand a non-technician, such as a purchasing agent, would leave the device on the table and keep his hands in his lap. His one question was the cost.

Experts say that a person is likely too gifted in only one or possibly two areas. We don’t expect a doctor to be able to fix a toilet. But we do ask a mechanic to pass a test written at a high reading level.

The only real test of technician competence is hands-on demonstration of the work. The testing industry claims that such testing would be too subjective (it seems to me that it’s pretty black and white — the furnace fires or not). They also claim it would be too labor intensive.  

The fact is that testing is currently under the control of the testing industry. Professional testers have a vested interest in doing what they have always done — writing questions, providing four possible answers, grading and collecting money. If a technician fails and has to re-test, then that’s more money. 

I wonder if those writing the tests are aware of the difference between a good question and a bad one. When I was in about the fourth grade the teacher let us kids try writing test questions. The ones we wrote were really, really hard. We thought the idea was to make kids answer them wrong. That’s what the NATE and BPI tests make me think of—kids who think the point is to make it really hard for other kids to pass.

Besides reading level, there is the subject matter of the questions. If there are 50 or 100 questions on the test on gas equipment, in today’s world should one of them really be how many millivolts are in a powerpile generator? 

Immediately after taking an exam recently, I talked with a couple other test takers. One was an award-winning tech school instructor and the other was an experienced contractor. All of us agreed that there was no correct answer for question 62. No answer was any better than the others. With the eenie-meenie-miney-moe technique, we had a 25% chance of guessing the right answer. That’s no way to test an HVAC technician’s ability to fix a furnace, boiler, or air conditioner, or to analyze a building’s energy efficiency.

Carol Fey is a technical trainer who has been in the HVAC industry for over 25 years. You can find her books and DVD at www.carolfey.com.  To see her adventures while a heating mechanic in Antarctica, go to www.carolfey.blogspot.com.

Voice your opinion!

To join the conversation, and become an exclusive member of Contractor, create an account today!