Instrument-Tissue Interaction Detection Framework for Surgical Video Understanding

Wenjun Lin, Yan Hu, Huazhu Fu, Mingming Yang, Chin Boon Chng, Ryo Kawasaki, Cheekong Chui, Jiang Liu

Research output: Journal PublicationArticlepeer-review

Abstract

Instrument-tissue interaction detection task, which helps understand surgical activities, is vital for constructing computer-assisted surgery systems but with many challenges. Firstly, most models represent instrument-tissue interaction in a coarse-grained way which only focuses on classification and lacks the ability to automatically detect instruments and tissues. Secondly, existing works do not fully consider relations between intra-and inter-frame of instruments and tissues. In the paper, we propose to represent instrument-tissue interaction as {instrument class, instrument bounding box, tissue class, tissue bounding box, action class} quintuple and present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. We also propose a Spatial Corresponding Attention (SCA) Layer to incorporate features of proposals between adjacent frames through spatial encoding. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance. For evaluation, we build a cataract surgery video (PhacoQ) dataset and a cholecystectomy surgery video (CholecQ) dataset. Experimental results demonstrate the promising performance of our model, which outperforms other state-of-the-art models on both datasets.

Original languageEnglish
Pages (from-to)2803-2813
Number of pages11
JournalIEEE Transactions on Medical Imaging
Volume43
Issue number8
DOIs
Publication statusPublished - 2024
Externally publishedYes

Keywords

  • Instrument-tissue interaction detection
  • surgical scene understanding
  • surgical video

ASJC Scopus subject areas

  • Software
  • Radiological and Ultrasound Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Instrument-Tissue Interaction Detection Framework for Surgical Video Understanding'. Together they form a unique fingerprint.

Cite this