female hand using a feather scribe to write on paper by candelight

Why do writers of academic letters of reference put 20% of medical students in the top 5% of the class?: Thoughts on the weakness of letters of reference

| 21 January 2021

This guest blog deals with an activity familiar to all academic physicians: writing letters of reference for medical students.

graph red and yellow in the shape of a hill

If this were our Queen’s University Medical school class, which has 100 students/year, only 5 students could be said to be in the top 5% and 5 must be at the bottom 5% (and that’s just math). Admittedly one could run this assessment for many important aspects of doctoring (test results, compassion, work ethic, community engagement, creativity, courage, advocacy) and the evaluation of people would change with different people assuming the top 5% positions.

As with most things in life, honesty is the best policy (raising the question whether dishonesty is the second best policy). While the term dishonesty is harsh, there is a lot of hyperbole in most letters of reference. It’s hard to fit 20% of applicants into “the top 5% of trainees” and yet kindness or risk aversion favours this new math. Likewise, the vast majority of students are ranked as being above average (a mathematical impossibility).

When it comes to letter writing a thesaurus of superlatives is often used. It is relatively rare that weaknesses or areas for improvement are flagged. Communication of weaknesses and areas for improvement are often implied rather than stated. We are all familiar with the art of “reading between the lines”. It is often easier to write a lengthy and glowing reference that is superficial than to compose an honest appraisal of a student. The stakes are high for all involved. The medical student wants a positive letter. It is the ticket to securing the residency training position of their choice and a desirable residency position launches their postgraduate career. For the letter writer there are concerns about being too harsh (even if accurate) and damaging the chances of a trainee for success. There are also concerns about the repercussion of an honest but less than glowing letter should the trainee find its contents offensive (even confidential letters can be discovered). However, for the training program that will be responsible for the training of the medical student during a lengthy residency program it is unfair to blind them to challenges they might reasonably expect. False praise can blind a training program to the need for prospective intervention or remediation, which serves no one well. Patients expect us to honestly evaluate doctors in training. Whether in letters of reference or in the CBME evaluations that trainees will receive once they enter residency, honesty is critical. Honesty does not mean hostile and harsh; but it also does not mean saccharin and superficial. We will never improve doctors in training a without sober, honest evaluation.

To obtain a residency position after medical school students apply to the Canadian Resident Matching Service (CaRMS)(click here). These applications require letters of support from faculty members who know the student. As we enter into the CaRMS PGY-1 match application cycle, faculty will be asked to write reference letters to support applicants. With disruptions in clinical care and clerkship rotations due to the COVID-19 pandemic, getting to know applicants has become challenging. However, reference letters remain key components of the resident selection processes. As we begin to write these letters, it is interesting to reflect on their perceived and actual value in the selection process. It was such reflection that led Dr. Lawrence Hookey and Dr. Mala Joneja (pictured below) to pursue a deeper exploration of the value of these letters, moving beyond opinion and anecdote through the use a qualitative research approach.

photo of Dr. Hookey photo of Dr. Joneja

Dr. Lawrence Hookey Dr. Mala Joneja

I invited Drs Joneja and Hookey to summarize some of the research they recently published on letters of reference. I believe their findings can help guide us as we write these important letters. So, let’s here from Dr Lawrence Hookey (Chair of Gastroenterology) and Dr Mala Joneja (Chair of Rheumatology).

_______________________________

Our study published in BMC Medical Education last year (click here), examined 715 reference letters for applicants in the CaRMS subspecialty match. We looked to define essential components of a high-quality letter and to elucidate the relationship between the quality of the letter and the letter writer.

We demonstrated that key features of a high-quality letter included:

A description of the nature of the relationship between letter writer and applicant
Duration of the relationship
In Training Evaluation Report information
Research involvement
Comments on areas for improvement.

We also found that program directors wrote the highest quality letters for applicants, and that letters from elective rotations were of the poorest quality.

screenshot of research paper on reference letters

Dr. Lawrence Hookey

My interest in this field of research was borne of frustration- reading program applications first as a residency program committee (RPC member) and subsequently as program director. I grew tired of reading that the vast majority of residents were described as being in the “top 5%” (obviously mathematically impossible). Likewise it seemed unfair to “red flag” those listed in the “top 25%” as clearly, they in this system they must have fatal flaws to be ranked so poorly.

We have come (falsely I believe) to rely on the art of reading a reference letter, detecting what is actually meant by reading “between the lines”. The yield from this inference-based assessment method is overstated. I propose that inference-based letter reading is a skill that has been developed of necessity due to the unwillingness of referees to be frank and honest. The reasons for this are multifactorial, but I think the essence is that we are truly afraid of what we write will coming back to the trainee themselves.

Our research team addressed the question why academic referees write letters of reference and what they hope to see in such letters. What they hope to see in a letter of reference seems to be answers to a few key questions: is the person going to be a good colleague (someone we want to spend most days within the upcoming years) or will they take more than a reasonable amount of time and effort to get them to a level that would be considered competent? Naturally some trainees are going to take more effort to achieve competency or excellence. However, medical educators who are also busy clinicians are aware that taking on a trainee who likely requires remediation from day one is not an appealing prospect. Thus, an honest appraisal for a candidate’s abilities is important.

A particular incident led to my conversion from “expert complainer about reference letters” to novice medical education researcher. A trainee was involved in a critical incident during my program director years. I struggled to understand the situation and how certain decisions were made. I reviewed the trainee’s application file to see if there were clues to past behaviours, specifically communication or judgement issues. I found nothing there, just the usual “top 10%” and “would make an excellent GI trainee”. I was so upset by the situation that I decided to go a step further and contact one of the letter writers to ask if there had been any similar incidents in the past with this trainee. Initially, the individual was more than happy to talk, but when I outlined what had recently transpired, I was met with an awkward silence followed by, “I was afraid something like that may happen.” . They then recounted a mixed bag of emotions and I was appreciative of their candor, albeit delayed. They acknowledged having been frustrated with the individual and with a system rife with unhelpful and misleading letters. I was in shock that this trainee had been sent to another program with no warning or heads up. This was a disservice to both trainee and training program. A balanced but critical appraisal in the letter warning would have allowed for proactive corrective actions and could have accelerated plans to help the individual and the program. I became determined to delve further into why most letters are loaded with unhelpful cheerleading, lack true insight into who the person is as a physician, and fail to make explicit what challenges they will likely face on the road to becoming a competent or excellent independent practitioner.

Our research shows that there appear to be two camps of letter writers- advocates and appraisers. Those sitting strongly on the advocate side of the fence firmly believe their role is to help the trainee get their position of choice, and to burnish any application to achieve this goal, often to the point of omitting key deficiencies or challenges the trainee presents. These omissions are intentional because the referee knows that an honest but critical assessment of weaknesses or deficiencies (and we all have them) may be used against the trainee at RPC ranking committee meetings. In contrast, the appraisers give a real-world view of what the trainee is like to work with that often includes challenges faced (especially those overcome). Clearly this is not a black and white issue, and referees can be both advocates and appraisers. Nevertheless, the move to a pure advocate role has likely been driven not on the basis of the writer’s own personality but rather results from experience on selection committees. When one reviews 20 files, all of which are “top 5%” candidates and then encounters the file of someone who is merely a “top 20%” candidate (perhaps a candidate who met a challenge and then overcame it, eg. “improved presentation style to become more concise”, the RPC reader may ask, “why risk it?”. I have sat around these RPC tables and the concerns are always that when one weakness is acknowledged, what else isn’t being said (ie. Is this the tip of the iceberg?).

The project was started nearly a decade ago with the participation of Dr. Deepti Chopra, then a first-year medical student at Queen’s University. She is now an accredited gastroenterologist in practice; however, in the intervening decade the system of academic letters of reference hasn’t changed much. My own practice of writing and interpreting letters of reference has however markedly evolved, having been informed by what I learned in this research initiative. I will no longer write letters based on purely on elective rotations, explaining to the trainee that I don’t know them well enough and that the absence of a letter from an elective at Queen’s does not hurt their application in any way. I write honest letters, and I hope the RPC readers appreciate and recognize the difference in the content reflects a desire to be honest and maintain standards. I avoid any percentage comments, as they hold no weight with readers at this point. I am ready to defend any letter I write at this point, which I shamefully admit was not always the case in the past. In fact, I was recently asked by a trainee if I would be willing to change a letter I had written so that it could be used on future applications, essentially to more supportive and omit any challenges the trainee had faced. I declined, and explained why I couldn’t do so. These conversations are difficult and awkward, but if we refuse to have them we perpetuate the broken system we currently have.

Dr. Mala Joneja:

My interest in the study was the result of simple curiosity regarding the role and impact of the letter of reference (or letter of recommendation as it is also called), in medical education. I wondered what the basis was for the tremendous value given to this particular component of the application process in selection of residents. The study with Dr. Hookey looked at letters of reference in the Canadian medical subspecialty match; however, the results and teaching points from the study are applicable to the PGY-1 specialty match as well.

The study showed that letters from elective rotations are of the least value in resident selection because they were least likely to include critical key components. Core Internal Medicine program directors practice what they preach, in writing the best quality letters of reference for medical subspecialty match candidates. Their letters were more likely to have the key components listed. I believe the findings show us that if we examine a phenomenon in medical education closely, we may find evidence of relationships we knew existed before and we may at the same time find results that are surprising. Our study reinforced the value of the teacher-learner relationship while showing us that going for elective rotations to get a reference letter may not be such a good idea. We may also find a clue regarding how to improve a process that has been around for ages, and that is not likely to disappear anytime soon. In our case, we came across the potential value of asking program directors to guide all letter writers in writing high quality, honest and objective letters.

As a clinical faculty member, when I write a letter of reference for an applicant, I feel that I am saying that I vouch for this applicant and their abilities. You can imagine that even in ancient times, when someone wanted to become part of group, they would require someone to vouch for them. This stamp of approval from someone in authority or someone who has expertise is the essence of the letter of reference. Most letters of reference, with few exceptions, carry this minimal attestation of approval. It is the ranking of abilities and attributes of an applicant that makes the process of writing and reading these letters a more complex art. It has been stated that many reference letters sound very similar with few differentiating characteristics, and that ranking candidates according to these letters becomes an exercise in reading between the lines. Information that is not favourable toward the candidate may not be explicitly mentioned due to fear of retaliation, future confrontation or simply keeping “fingers crossed” that the candidate will do alright and deserves a chance.

Reading between the lines should not be necessary if a letter writer truly knows the applicant. The relationship between a supervisor writing the letter and student (or trainee) should be of sufficient depth that the granted stamp of approval has meaning. In order to show that, some amount of detail that allows the letter reader to separate that applicant from the others, should be included. This relationship of supervisor or mentor and learner, actually loses its meaning if letters are written half-heartedly, and without detail or critique. This perhaps explains why letters from elective rotations were found to have less value than were those from home institutions, as there is a much lower probability of a learner developing a strong relationship with an elective supervisor.

I think this result was somewhat unexpected though because there has been significant value placed on going on an elective rotation and getting a reference from a supervisor who is not from an applicant’s home institution. This perceived value of the elective rotation is not a standard reinforced by any rules in postgraduate training, but simply to bolster chances of being highly ranked by many programs. Perhaps it is the thought that someone who is not biased by any prior knowledge or relationship, has assessed this learner and found them competent. It turns out that this potential benefit is likely outweighed by the lower quality of these letters. Another potential explanation for the low quality of elective letters, is a lack of investment felt to be necessary or possible on the part of the elective supervisor. This then leads to a less strong teacher-learner relationship and a lower quality letter. However, the elective rotation is not an important part of clinical education simply because it affords the opportunity for a reference letter from another source. It is also potentially important for applicants to experience the learning environment at different institutions as they look for the residency program that is best for them. Unfortunately, the applicants in this year’s cycle will not have had such opportunities due to the COVID-19 pandemic. This might be some small reassurance to them that the letters from elective rotations are not found to be as valuable.

Even though letters of reference have well-known shortcomings, they are likely to continue to be part of the resident selection process and educators should work to improve their usefulness. One clue as to how to do is this comes from our study. We should be asking the program directors for their guidance and advice. Despite the fact that this would be another task for program directors, it would be worthwhile and benefit letter writers, letter readers and applicants. It would help us refine this traditional process to suit our current needs. What I have learned from studying the letter of reference is that it is up to me to make this a useful document for the application process.

So has this research changed my practice? Yes, my letter writing practice has evolved to include sufficient detail demonstrating that I know the learner well and that I am aware of their abilities. I routinely state a candidates strengths and where I have seen improvement, and I do not use percentages to differentiate groups of applicants. This strategy feels authentic to me as it also honours the relationship between teaching and learner, and conveys a true stamp of approval.

Here are some key messages for letter writes and applicants.

For letter writers:

Be authentic. Agree to write letters for those you feel you can give an honest appraisal and advocate for their success. Be prepared to turn down requests if you are unable to do this.
Be an advocate. If you are ready to vouch for the candidate, give them the best boost that you can and be as specific to the particular candidate as much as you can.
Be an appraiser. As a letter writer, programs are relying on your assessment of the capabilities of the candidate and your assessment of the applicants capabilities related to their future work is of great value to the letter reader. It is your expert opinion that is being asked for by programs.

For applicants:

Look for someone who will represent you authentically, someone that you have had a good working relationship with.
Look for an advocate, someone who knows the residency education applicant system and what it means to vouch for someone. Let them know of the outcome of your application. The letter writer would love to know what happened.
Assist them in advocating for you. Remind your referee of the time that you worked together and the cases that you saw together. Tell them why applying to this program is important to you.

Good luck to all those applying in the upcoming CaRMS PGY-1 residency match and to all those letter writers out there!

Dr. Joneja, Dr. Hookey and I look forward to your thoughts on this post.