Why are Situational Judgement Tests the best way forward?
In order to ensure that medical students are selected in to the Foundation Programme in the fairest possible way, a number of different selection methods were considered as part of an option appraisal. When comparing the different options, it became clear that Situational Judgement Tests (SJTs) are the fairest, most reliable and practical way forward. This is because students will take the SJT in exam conditions and so everyone will have an equal chance to do well. There is also research evidence to support the use of SJTs and it is expected that a sufficient number of questions can be developed in order to use SJTs in the long term.
What do SJTs assess?
SJTs are a test of aptitude and are designed to assess the professional attributes expected of a Foundation doctor. There are two question formats:
1. Rank five possible responses in the most appropriate order
2. Select the three most appropriate responses for the situation
Different scenarios lend themselves to different response formats so using two different formats allows a range of situations to be tested.
Students must answer what they ‘should’ do in the scenario described, not what they ‘would’ do. This is because research into SJT shows that questions asking a candidate what they ‘would’ do are more susceptible to coaching.
How are the SJT questions written?
The SJT tests a number of different attributes, which were identified during a job analysis of the FY1 role, including team working and professionalism. The attributes form the basis of the SJT items, which are written by subject matter experts who work closely with Foundation doctors. This ensures that the scenarios presented are an accurate reflection of what FY1s encounter in their role. The items are then reviewed by other subject experts including FY1 and FY2 doctors, to ensure they are both realistic and fair.
How fair are SJTs?
The SJT will be invigilated, meaning that students will have a fair chance to do well without the possibility that some are receiving outside help. The items have also been designed to reduce the ability for coaching. In order to ensure students feel prepared for the SJT, prior to taking the ‘live’ exam, they will have access to example questions and answers to help familiarise themselves with the format.
The answer keys allow for the elements of subjectivity in the ranking scenarios, with points awarded for near misses. This means it is possible to score highly, without getting all of the answers in exactly the right order. However, if students put the best answer as the least appropriate or vice-versa, they would not get points for this.
In addition, research has shown that generally, scores are less influenced by ethnicity than tests of cognitive ability. Tests into the effects of group differences on performance in this SJT will be carried out at all stages.
Have SJTs been tried and tested?
Research evidence suggests that SJTs are able to predict performance in the role, as well as showing higher validity over other methods. They have also been shown to be reliable.
They are currently used for selection into GP training and are increasingly being piloted and used in other specialty selection processes. Evidence suggests that within medical selection, SJTs are a reliable and valid method of selection.
SJT items were initially piloted at four medical schools, involving over 450 medical students, in autumn 2010. Fifteen further pilots involving over 1,000 medical students took place at both UK and non-UK medical schools in spring 2011. The results show good levels of reliability and the SJT was able to differentiate between candidates.
How will you ensure the SJTs remains the fairest method possible?
Creating a system that can be used into the future is a really important consideration. ‘White space’ questions cannot continue to be used as there are limited ways to ask these types of questions. SJTs allow for a range of scenarios to be presented and a vast amount of questions can be created.
The SJT can be refreshed every year with new items to help increase the longevity of the test and in order to ensure the SJT remains valid, ongoing work with take place – for example, studies which assess whether performance on the test is related to future performance as a doctor.
References
1. Ployhart, R.E., & Erhart, M.G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11, 1-16
2. Lievens, F., Buyse, T., & Sackett, P. R. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 443-452
3. Patterson, F., Archer, V., Zibarras, L., Coan, P., Kerrin, M., Paul, .N. (submitted) Using situational judgement tests for selection into medical education and training: A review of the research evidence. Medical Education
4. Lievens, F., & Coetsier, P. (2002). Situational Tests in Student Selection: An examination of predictive validity, adverse impact and construct validity, International Journal of Selection and Assessment, 10, 245-255
5. Patterson, F., Baron, H., Carr, V, Plint, S, & Lane, P (2009). Evaluation of three short-listing methodologies for selection into postgraduate training in general practice, Medical Education 43, 50-57
6. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740
7. Patterson, F., Carr, V., Zibarras, L., Burr, B., Berkin, L., Plint, S., Irish, B., & Gregory, S. (2009). New machine-marked tests for selection into core medical training: evidence from two validation studies