ANGIO Mentor Validation & Publication

The following abstract was presented in the prize session of the European Society for Vascular Surgery (ESVS) Annual Meeting  September 20 - 23, 2007 in Madrid, Spain

Experienced endovascular interventionalists objectively improve their skills by attending carotid artery stent training courses.

S. Neequaye, I. Van Herzeele, R. Aggarwal, M. Hamady, A. Darzi, T. Cleveland, P. Gaines,  N. Cheshire

Department of Biosurgery and Surgical Technology, Imperial College London, U.K.

Objective Carotid artery stenting is an advanced endovascular intervention with a steep learning curve. Its success is tempered by the need to ensure that all interventionalists possess technical proficiency prior to performing cases on patients. Virtual reality simulation has been proposed as a means to train and objectively assess technical performance without risks to patient safety. The aim of this study was, using a commercially available simulator, to objectively assess psychomotor skills acquisition of experienced interventionalists attending a two-day carotid artery stent course.
Methods Eleven experienced endovascular physicians from several medical disciplines with minimal experience in carotid artery stenting participated in this study. During two identical two-day courses, cognitive and technical skills were trained using didactic sessions, case reviews, supervised VR simulation and live-cases. Pre- and post-course skills were assessed through performance on the same carotid artery stent procedure using the quantitative and qualitative assessment parameters derived from the simulator. The Wilcoxon signed ranks test was used to compare performance across the two groups.
Results Significant differences were noted between pre- and post-course performance for total procedure time (medians 36 vs. 20 minutes, p=0.005), fluoroscopic time (20 vs. 11 minutes, p= 0.016) and delivery-retrieval time of the embolic protection device (12 vs. 9 minutes, p= 0.007). Advancement of the guiding catheter or guiding sheath without a leading wire occurred to a greater extent pre-course versus post-course (199 vs. 152 millimetres, p= 0.050) as did spasm of the internal carotid artery (4 vs. 2, p= 0.049).
Conclusions This study has objectively proven a benefit for experienced interventionalists to attend carotid artery stenting courses for skills acquisition measured by a virtual reality simulator. In addition to providing participants with an insight into their skills, these data can be used to objectively audit course efficacy.

Analysis of simulated angiographic procedures. Part 2: extracting efficiency data from audio and video recordings.

Duncan JR, Kline B, Glaiberman CB.
Mallinckrodt Institute of Radiology, Washington University School of Medicine, 510 S. Kingshighway Blvd., St. Louis, MO 63110, USA.
J Vasc Interv Radiol. 2007 Apr;18(4):535-44

PURPOSE: To create and test methods of extracting efficiency data from recordings of simulated renal stent procedures. MATERIALS AND METHODS: Task analysis was performed and used to design a standardized testing protocol. Five experienced angiographers then performed 16 renal stent simulations using the Simbionix AngioMentor angiographic simulator. Audio and video recordings of these simulations were captured from multiple vantage points. The recordings were synchronized and compiled. A series of efficiency metrics (procedure time, contrast volume, and tool use) were then extracted from the recordings. The intraobserver and interobserver variability of these individual metrics was also assessed. The metrics were converted to costs and aggregated to determine the fixed and variable costs of a procedure segment or the entire procedure. RESULTS: Task analysis and pilot testing led to a standardized testing protocol suitable for performance assessment. Task analysis also identified seven checkpoints that divided the renal stent simulations into six segments. Efficiency metrics for these different segments were extracted from the recordings and showed excellent intra- and interobserver correlations. Analysis of the individual and aggregated efficiency metrics demonstrated large differences between segments as well as between different angiographers. These differences persisted when efficiency was expressed as either total or variable costs. CONCLUSIONS: Task analysis facilitated both protocol development and data analysis. Efficiency metrics were readily extracted from recordings of simulated procedures. Aggregating the metrics and dividing the procedure into segments revealed potential insights that could be easily overlooked because the simulator currently does not attempt to aggregate the metrics and only provides data derived from the entire procedure. The data indicate that analysis of simulated angiographic procedures will be a powerful method of assessing performance in interventional radiology.

The use of interventional cardiovascular simulation to evaluate operator performance: The carotid Assessment of operator performance by the Simbionix carotid StEnting Simulator Study (ASSESS)

Giora Weisz, Jacque Devaud, Stephen Ramee, Mark Reisman, William Gray

Cardiovascular Research Foundation, and Center for Interventional Vascular Simulation, New-York Presbyterian Hospital, Columbia University, New York, NY

Journal of the Society for Simulation in Healthcare 2007, Volume2, Issue 1

Background: The percutaneous endovascular carotid stenting procedure has only recently been approved by the FDA, and interventional operators from multiple disciplines (Cardiology, Radiology, Vascular Surgery) are getting training in this field. The professional societies of these disciplines are looking for novel ways to assess the competency of an operator, before they grant authorization to perform this complex procedure.
Simulators have been suggested as tools to evaluate operators in specific procedures and operative techniques. The virtual reality machines can be used to capture numerous factors that translate into the success and safety of the procedure. These include selection of devices, accuracy of manual performance, reaction to interactive problems, use of adjuvant pharmacotherapy, management of hemodynamic changes, and prevention and resolution of complications. All of these can be automatically evaluated and scored by a complex software designed virtual reality simulator. The simulators that are used in Interventional Cardiology and endovascular therapy have not been validated yet as such assessment tool.

Objective: The main objective of the ASSESS study is to validate the simulator-based metrics as a tool to discriminate between different levels of operators.

Methods: We used the Angio-Mentor carotid Stenting simulator system (Simbionix). The simulators had been loaded with predefined cases demonstrating increasing complexity for stenting intervention.
Total of 34 operators (10 experts, 12 intermediate level, and 12 novice; based on prior reported experience) were asked each to do three cases of carotid stenting on the simulator. All the operators had the same cases that were selected to demonstrate increased complexity to judge anatomy, correct performance of the technical steps, use of wires, catheters, embolic protection device, balloons, and stents, adjuvant pharmacotherapy, and management of interactive hemodynamic response. During the procedure, multiple features of all of the above elements were automatically captured by the simulator. Each step and parameter captured had a pre-defined score. Scoring was based on penalty points for mal-performance, as pre-defined by a panel of independent experts in carotid stenting, prior to conducting the study.

Results: The study has been completed, and all the data that were captured is being analyzed. Full results will be available at the time of the presentation

Conclusions: Being able to demonstrate the ability of a hi-tech, virtual reality simulator to differentiate between different levels of interventional operators, may enlighten the simulator as a potential objective “examination tool”, to evaluate the performance and competence of an interventional operator. These will justify the use of Interventional Cardiovascular Simulation for certification and credentialing