Researcher (Assistant Professor) in Computer Vision at the Istituto Italiano di Tecnologia (IIT). Stuart's research focus is on Visual Reasoning to understand the layout of visual content from Iconography (e.g. Sketches) to 3D Scene understanding and their implications on methods of interaction. He is a PI on the MEMEX RIA EU H2020 project for increasing social inclusion with Cultural Heritage and Co-PI on the RePAIR EU FET H2020 project for the reconstruction of frescoes. Stuart has previously held PostDoc positions at IIT, University College London (UCL) and the University of Surrey. Also, at the University of Surrey, Stuart was awarded his PhD on visual information retrieval for sketches. Stuart continues to hold an honorary position at UCL and UCL Digital Humanities. He also regularly organises workshops and conferences, most recently the Vision for Art (VISART) Workshop at ECCV'20 and British Machine Conference (BMVC) 2021.

Stuart James

Research Scientist
Developer
Blogger
Runner
Climber
Drummer

 

 

 

Research interests

My research activities fit broadly into Spatial Reasoning — how we can reason about the layout of objects in space in both 2D and 3D to provide insight or retrieve relevant information. My research has a keen interest on varied data types including those from the Humanities such as Art and Cultural Heritage.


Exploring using Depth and Knowledge to answer questions specifically related to the layout of a 3D scene from a 2D perspective.

Visual Question & Answering

Detection, Representation and Reasoning on simplified representations or symbols such as Sketch, Line, Hatching, Motifs or icons.

Abstract & Iconography Reasoning

Identifying and retrieving relevant knowledge held within Knowledge Graphs to support Computer Vision tasks such as Visual Question and Answering or reasoning on location.

Knowledge Retrieval & Reasoning

Reconstructing the semantic relational structure of the scene using geometry and knowledge. Providing advanced interaction for questioning and reasoning.

Scene Graph

Principally on layout of content in 2D or 3D and how to make decisions that influence about a path or option linked with Visual Question and Answering

Planning & Reasoning

We have explored using sketches to search collections of videos using Visual Storyboarding to express the sequence of events in the target clip.

Sketch based Retrieval

We are using sequences to retrieve information providing a broader context than a one-off search. We have demonstrated through Free-Hand storyboarding and storey synthesis.

Visual Narratives and Stories

Within VR we explored the use free-hand sketching in an Immersive Environment (VR) with multiple modalities for the task of retrieval.

Interaction in Virtual Reality

Providing storytelling experiences overlaying information of surrounding Cultural Heritage and the stories of the particpants in the MEMEX Project.

Interaction in Augmented Reality

Cultural Heritage & Digital Humanities

Assistive Technologies

Robotics

Research Group & Collaborators

Research Topic: Causality

Supervisor with Dr Jacopo Cavazza (IIT) and Dr Alessio Del Bue (IIT)

Davide Talon

PhD Student

Research Topic: Localising ourselves and objects for Social Good

Collaborator with Dr Alessio Del Bue (IIT)

Dr Matteo Taiana

PostDoc Collaborator

Research Topic: Optimising camera localisation in urban scenes

Collaborator with Dr Alessio Del Bue (IIT)

Dr Matteo Toso

PostDoc Collaborator

Research Topic: RePAIR Fresco 3D reconstruction and assembly

Collaborator with Dr Alessio Del Bue (IIT)

Dr Theodore Tsesmelis

PostDoc Collaborator

Research Topic: Knowlege Graph guided retrieval and localisation

Collaborator with Dr Alessio Del Bue (IIT)

Àlex Solé Gómez

Research Fellow

Research Topic: Immersive spatial interaction within Virtual Reality

Collaborator with Prof. Anthony Steed (UCL)

Dr Daniele Giunchi

External Collaborator

All associates in research lines PAVIS and VGM at Istituto Italiano di Tecnologia

Latest Blog Post

03 Dec 2020 . research . Three great videos about the pilot locations of MEMEX Comments

Snapshot of MEMEX Project news

We recently published three videos on the pilot locations, Paris, Barcelona and Spain, of the MEMEX Project. This challenging collaboration effort came together during the second wave of COVID so a massive effort was needed by the Michael Culture Association, NOHO Limited, Fundacio Interarts per a la Cooperacio Cultural Internacional (InterArts), Mapas das Ideias and Dédale.

Read More

Previous posts

newsfeed

Latest Publication

2022 Writing with (Digital) Scissors: Designing a Text Editing Tool for Assisted Storytelling using Crowd-Generated Content

Digital Storytelling can exploit numerous technologies and sources of information to support the creation, refinement and enhancement of a narrative. Research on text editing tools has created novel interactions that support authors in different stages of the creative process, such as the inclusion of crowd-generated content for writing. While these interactions have the potential to change workflows, integration of these in a way that is useful and matches users’ needs is unclear. In order to investigate the space of Assisted Storytelling, we designed and conducted a study to analyze how users write and edit a story about Cultural Heritage using an auxiliary source like Wikipedia. Through a diffractive analysis of stories, creative processes, and social and cultural contexts, we reflect and derive implications for design. These were applied to develop an AI-supported text editing tool using crowd-sourced content from Wikipedia and Wikidata.

Accepted at International Conference on Interactive Digital Storytelling (ICIDS 2022) in Santa Cruz, USA.

See full publication index

Contact

To find out more about our research you can find me at...

Pattern Analysis and Computer Vision (PAVIS)

Center for Human Technologies - Istituto Italiano di Tecnologia
Via Enrico Melen, 83, 16152 Genova GE