Skip to main content
Reference Letter Signing Image

Reference letters for Resident Selection: Frustrations and Future Direction

Rachel Bentley, MSc Candidate, Translational Medicine

At the Medical Grand Rounds on October 3, the Department of Medicine was fortunate to have Dr. Lawrence Hookey, the current Chair of the Queen’s Gastroenterology Division and Medical Director of the Endoscopy Units at Kingston General and Hotel Dieu hospitals. Presenting alongside his collaborator Dr. Mala Joneja, Dr. Hookey’s presentation focused on the problems with the current system in which letters of recommendation are written for resident selection and some potential improvements to this process.

It is widely accepted that letters of recommendation are among the most important aspects of a resident’s application; many program directors say they would consider a weaker candidate with a well-crafted reference letter more favourably, and would consider a stronger candidate with a poorly-crafted reference letter less favourably.

As Dr. Joneja highlighted, the concept of vouching for someone has been around a long time and holds a lot of power. It is unlikely that reference letters will become a thing of the past as they provide a unique milieu for personalized comments and an opportunity to make comparisons among applicants. The difficulty that program directors are currently facing is that these letters are poor at predicting future performance, which is the role they are meant to play in the selection process.

As Dr. Hookey’s research has elucidated, the majority of letter writers see their role as being an advocate for the applicant, with a fear that any negative comment could damage the resident’s career. Dr. Hookey outlined how this reluctance to be negative in letters leads to a code of sorts, where letter writers will intentionally omit a certain element or hide a criticism in somewhat positive but not glowing terms, requiring program directors to read between the lines. Additionally, with a lack of standardization, the stock a program director puts in a certain letter of recommendation can depend on their knowledge of the letter writer more than the content of the letter itself. For example, if a certain letter writer gives high praise to even the most mediocre of applicants, the program director would likely not weigh this letter heavily when making a decision.

Dr. Hookey’s research has striven to address the evident disconnect that exists between what program directors want in a letter and what referees are writing. After surveying a number of program directors to assess what they want in a letter, Dr. Hookey’s group reviewed letters to determine how many letters qualified as high-quality letters based on the criteria from program director responses. Interestingly, despite a fair bit of consensus among program directors as to what constitutes a high-quality letter, few of the letters written by program directors met all the criteria of a high-quality letter.

One of the most important characteristics that a letter of recommendation can speak to is the applicant’s communication skills. Communication skills are integral in the medical field and the letter writers, who typically interact with the applicant regularly for an extended period of time, are in a much better position to evaluate the applicant’s typical behaviour than the program director, who likely interacts with the applicant only during the interview. Interestingly, in Dr. Hookey’s study, despite being explicitly asked to comment on the applicant’s communication skills, not all letters included such statements, with less than half commenting on the applicant’s interactions with patients.

Another important aspect of these letters that Dr. Hookey pointed out was that, without qualifiers or specific examples about the applicant’s qualities and skills, the letters can be too subjective and so do not provide much value to the program directors. Moving forward, these shortcomings need to be communicated to letter writers; program directors should perhaps agree on a more standardized format for reference letters so that all necessary aspects are included in a descriptive and useful way. The fact that there is no mechanism for feedback to the letter writers is detrimental to both sides, as the letter writers may not know that their current format is not helpful and the program directors are clearly not getting an output that is predictive of performance. Additionally, improving this process could optimize the selection process, and ultimately improve patient care.

It was a pleasure to have Dr. Hookey at our latest Medical Grand Rounds discussion, and the TMED graduate students appreciated the opportunity to share Dr. Hookey’s insights into the difficulties and future directions for the writing of reference letters.

Comments

Name
Matthew James

Tue, 10/08/2019 - 10:01

Great summary Rachel! You alluded in your post to program directors agreeing on a standardized format or "must-include" items in their reference letters. Dr. Hookey touched on this in his Grand Rounds Presentation, and we discussed this in our post rounds discussion.
Now that its been a few days since his talk, I'm curious what you think the most important characteristics or attributes that should be included in a useful reference letter?

Name
Matthew James

Thank you Matthew, that's an interesting question. I think that the most important characteristics to include in a useful reference letter should be comments on the applicant's communication skills and interactions with patients. I think interaction with patients is a very important area to assess and the reference letters provide a useful platform for the program directors to learn about the applicant's typical performance. Beyond this though, I think one of the most important things that needs to change in reference letters, as Dr. Hookey mentioned, is for letter writers to use concrete examples to support the statements they are making about the applicant. Since reference letters seem to be be overly positive and less distinct, putting a statement into context can make it much more useful to the program director and enhance their ability to compare applicants.
I would be interested to hear your thoughts on this as well. What characteristics do you think are the most important to include in reference letters to help program directors predict applicant performance?

Name
Rachel Bentley

I agree with you that communications skills and interactions with patients, with concrete examples is 100% essential in a strong reference letter as it allows the adjudicator to have a real understanding of the candidates capacity as a physician. Dr. Hookey mentioned that the physicians writing the reference letters don't often observe their residents interacting with patients first-hand on a regular basis, so the candidate providing some specific examples to the referee may be helpful.
I think that another important quality to discuss in a reference is the candidates interaction with the healthcare team, particularly how they work with the nurses and other professionals on a day-to-day basis. I think that a person's team work and interpersonal skills in the work place say a great deal about a candidates potential for success in a new work environment, and if I were reading a reference letter, I would be very interested to know these qualities about the applicant.

Name
Matthew James

Yes, excellent summary, Rachel! I think one of the essential characteristics that referees should include in their letters is a negative attribute or something the candidate can improve. Dr. Hookey mentioned how difficult this is for referees to include without hurting the applicants' application. Speaking to the experienced referees out there, how do you approach this situation?

Name
Sophia Linton

Name
Daniel Rivera

Tue, 10/08/2019 - 14:05

Hi Rachel,

You certainly covered the most important main points of Dr. Hookey's presentation and current research. One thing that continues to stick out to me is that a reference letter writer, depending on experience or additional knowledge, might be more skillful in the craft or 'code' of reference than another applicant's and potentially put the applicant they are writing for at an advantage. Theoretically, standardization could help ameliorate this, but I wonder whether you think that even with standardization, as it is currently proposed, the skill of the reference letter writer can be completely controlled for. As Dr. Hookey mentioned, whatever changes are made to improve the application process, there always seems to be an associated shift in courses or preparation strategy that puts some applicants at a greater advantage - do you think with standardization, as it has been proposed, this will also occur (i.e. people finding ways to become better at tackling standardized letters)? Theoretically, it seems as though it shouldn't, but what about in practice? Can standardization perfect in the world of reference writing?

Name
Daniel Rivera

Hi Daniel,
I definitely agree that standardization could be difficult to put into practice and I don't think that standardization could ever be perfect. I do, however, think that by having more stringent standards, and by requiring letter writers to support their claims with concrete examples, the letters would be more of an honest reflection of the applicant. This could also potentially reduce the scale of the advantage conferred to the writer more skilled in the craft or code. There would likely always be a slight advantage to the experienced letter writer but these honest letters would ideally make it a more even playing field.
What are your thoughts on the subject? Do you think there is a way to eliminate the advantage conferred to the more experienced or skilled letter writer?

Name
Rachel Bentley

Great summary Rachel. I agree that standardization of these reference letters will definitely help produce a more thorough and perhaps candid assessment of applicants. However, as Dr. Hookey mentioned, the fear of damaging an applicant's career with a somewhat mediocre or negative comment would most likely still have a strong influence on this process. Nonetheless, I believe that standardization may go a long way to reduce this "fear" slightly, particularly if it is mandatory to comment on an area of weakness of an applicant.

Name
Edwin Ocran

This is a great question and got me thinking about med school application references. As Matthew mentioned during the discussion, and something I've seen through helping some undergrads apply, OMSAS has applied a standardized form for references to fill out. This includes rating skills of the individual and justifying your rating with real-life examples. Although I do believe this will aid in better comparisons of individuals, since all these sets of skills are being ranked in each individual applying, I also see where there may be some faults. The problem of boosting a rating of individual in the fear of "ruining" an individuals chances of getting an interview, still could occur. This isn't really a problem with the standardized references but rather an issue of using reference letters in general, and being afraid to put your honest opinion down. So while I think standardizing will improve references slightly, there are still will be some issues surrounding it.

Name
Spencer Finn

Hi Daniel,
I think standardization of the reference letter and the reference letter process might be the way to go in the future! Firstly, I think a standardized methodology, whether that be a template, a universal language or a form of sorts, for referees to fill out will allow evaluators to have an objective evaluation, as opposed to interpreting and "reading through the lines" as Dr. Hookey mentioned. I don't think that people will be able to "beat the system" and trick a standardized system because rules will be universal. However, the honesty of the referee may come into question in a universal format if they choose to exaggerate. Perhaps even moving towards a Likert scale of rating the candidate in terms of quality will be an even more objective way!

Name
Quentin Tsang

Name
Anne Ellis

Tue, 10/08/2019 - 14:30

So, I'm going to go out on a limb here and admit I freely suggest to trainees who wish to have me write their letter of reference in draft form for me first, for me to edit and 'tone down' or 'build up' if necessary as I see fit. What did the TMED 800 class take away from this lecture to help you in your future 'draft requests' if and when they come your way?

Name
Anne Ellis

Name
Madison MacKinnon

Wed, 10/09/2019 - 10:14

In reply to by Anne Ellis (not verified)

Hi Dr. Ellis,
Personally, in my future "draft requests" I think I would definitely focus on having concrete examples to support the statements I make in my letter. I would even consider providing multiple examples if possible to show my letter writer I am honest in what I am stating, and in case they were not there to witness one them in person. I would also provide a critique of myself with an example of how I have improved. Dr. Hookey emphasized doing this in his talk, and I greatly agreed as I think that it shows the applicant is very self-aware and has a drive for self-improvement.

Name
Madison MacKinnon

Name
Thalia Hua

Wed, 10/09/2019 - 10:55

In reply to by Anne Ellis (not verified)

Something that I took away from the presentation is that reference letters are not revised versions of your CV/Resume! I have definitely done this in the past where I would provide my referees with activities solely from my CV and be satisfied with a product that compiled my achievements in a glorified manner. Although some activities may suggest certain character traits (for example volunteering = selfless or president = leadership), would it be a good idea to openly communicate to the referees what qualities you would want to be highlighted?

Name
Thalia Hua

Name
Spencer Finn

Tue, 10/08/2019 - 20:08

Really great summary! I thought I'd briefly bring up a topic that wasn't really covered in Dr. Hookey's discussion but one I noticed while reading his papers that stood out to me. In one of Dr. Hookey's papers, he mentions that if the committee choosing residents has a relationship with whoever wrote the reference letter, they are much more likely to think of the student in a favourable light. So I was wondering if anyone had any ideas on how this could be resolved with standardization? Maybe anonymous references?

Name
Spencer Finn

Yeah that's an interesting point Spencer. I think that part of the process should involve letters from people who you have a close relationship with (as they likely know you best) but it does seem like there should be requirements for letters from individuals anonymous to the applicant, who have had the chance to observe the individual but not necessarily develop any type of relationship.

Name
Austin Read

Name
Lawrence Hookey

Wed, 10/09/2019 - 06:37

Rachel,
I think you said it much better than I did! Excellent job facilitating the group discussion as well.

Name
Lawrence Hookey

Name
Reem Alzafiri

Sat, 10/12/2019 - 16:15

I enjoyed personally enjoyed this talk and am grateful that there is research surrounding the idea of what it means to have a good reference letter and how to approach a supervisor for said reference. As a student it's important that our letters are about showing our greatest strengths and what we would offer the program of our desire. I think one of the biggest struggles in obtaining a reference is the time and relationship built with supervisors; I understand they are busy people who have to sometimes oversee more than one student, so facilitating that professional relationship is not always easy, especially when you are in need of more than one letter. I believe that the best way to show that your student is qualified for the position they want is to prove by a set of examples of how this student is a smart individual, reliable, and professional. This would help set us apart from generic references that state the same ideas in a conventional manner. If more supervisors look at research such as Dr. Hookey, I believe the outcome of one's reference letter would be much highly regarded.
Thank you again for the unique perspective and discussion of this research.

Name
Reem Alzafiri

Name
Jay Kataria

Fri, 10/18/2019 - 00:13

Hi Rachel,

Well done with the summary of Dr. Hookey and Dr. Mala's MGR presentation! In respect to one of the papers given, I find it interesting how the phrase "I give my highest recommendation" was rated the most positive and the phrase, "showed improvement" was rated the most negative. In my opinion, the phrase, "showed improvement" displays the applicant grew over the course of the placement or job. In addition, "showed improvement" seems to be more quantifiable compared to "I give my highest recommendation". Which comment do you think is more valuable?

Name
Jay Kataria

Add new comment

Plain text

  • No HTML tags allowed.
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.