Researcher (Assistant Professor) in Computer Vision and Machine Learning for Cultural Heritage following a Postdoctoral Researcher position at Istituto Italiano di Tecnologia (IIT) working on 3D scene understanding with Dr Alessio Del Bue. Previously Stuart was a currently a Research Associate at University College London (UCL). At UCL Stuart's research focused on cultural heritage domain including fine-detail 3D reconstruction of large bas-relief art work belonging to the British Museum. In additional looking at the British Library book illustration dataset, creating algorithms for segmentation of cross-hatch imagery. Previous to his time at UCL, Stuart completed a post-doc at University of Surrey looking at social media implications across different lifespan transitions; utilising multi-label classification and clustering through Deep Learning methods. Also at the University of Surrey, Stuart completed his PhD which explored how free-hand sketched storyboards can be used for retrieval and synthesis of video. Developing novel algorithms for video understanding and representation. Throughout his research career he has maintained a strong software engineering ethic and demonstrated this through prototype research systems.
Starting a research group in the interplay between Visual Artificial Intelligence and Cultural Heritage, bringing across techniques for Reconstruction, Scene Understanding and Question and Answering to enrich the abilities of both scholars and the general public. We have Postdoctoral Researcher posts available and PhDs in association with Ca' Foscari University of Venice. If you are interested in collaboration, feel free to get in touch (please note we are not a services group).
Working under Dr Alessio Del Bue
Exploring Visual Question and Answering in relation to geometry for scene understanding. More to come about this project as it is published.
Working under Prof. Tim Weyrich
Continued work on Cultural Heritage Vision and Graphics applications.
Working under Prof. Tim Weyrich
This project explores two different aspects of Digital Heritage analysis of artefacts and printed illustrations. Firstly the 3D reconstruction of artefacts, specifical bas-relief, provides challenges to traditionally computer vision methods. By utilising and developing new combinations of multiview photometric stereo and structure from motion we aim to reconstruct British Museum assets. Secondly exploring the analysis of illustrations of historical books provided by the British Library we develop an approach to segment sparse line structures with learnt shading styles.
Working under Dr John Collomosse
An EPSRC funded project exploring the digital effect on key life transition points, through Social Media. The project required the development of algorithms for classification and clustering utilising both Image and Text. Additional explored data purification of large noisy social media datasets using Genetic Algorithms. As well as a structured manifold mapping techniques for presentation of user data through a 2D game interface.
Working under Dr John Collomosse
Funding for research in to Visual Information Retrieval.
Working under Mr Michael Anthony
Providing Computer Support for a Small Business. Demonstrated through Developing reliable systems with strong backup and recovery policies. Work at JCS Technology involved setting up servers and network infrastructure and support for a variety of platforms Window, Linux and bespoke platforms. The role provided the opportunity to work within a budget and make key decisions on the day to day operating of the business.
Media retrieval has been dominated by text-based queries utilising meta-data tags, but such queries are cumbersome to describe the appearance and in the case of video temporal information. We propose methods using sketch as an intuitive way to describe and search such media content. Sketch based Video retrieval has traditionally applied complex model fitting, in contrast, we explore representations suitable for index structure to achieve sublinear query time. Which also makes it possible to get the user in the loop through relevance feedback. Secondly, we propose Sketch based Human Pose Retrieval (SBHPR), a method of finding humans postures within videos using stickman depictions. Developing a manifold based retrieval method and learning a domain adaptation to improve precision on new videos. Finally, we extended the SBHPR method to a storyboard allowing a sequence of pose and action labels (run, jump) to be intertwined. This is demonstrated for video segment retrieval and synthesis of a new video, by extending the motion graph technique.
Dissertation - Fluid dynamics Simulation interacting with rigid body objects using Smoothed Particle Hydrodynamics based on Mullers algorithm. Other notable projects - Sony PSP Student Development Kit
Mohamed Dahy Elkhouly, Stuart James, Alessio Del Bue
University of Adelaide | Adelaide, Australia
In this talk, we look at understanding the relationships between objects within a 3D scene. Firstly, we present our latest paper on using multi-view information to construct a scene graph of objects guided by the layout of ellipsoids. Our ellipsoid nodes coupled with object nodes act as proxies allowing relationships 'same-set', 'part-of', 'same-plane' and 'support' to be inferred by message passing over the graph. We build an architecture that can support such geometric nodes, object nodes and relational nodes merged using within an RNN framework. Secondly, we show how a question about the layout of a scene can be directly answered using RGBD. Using a depth branch guided by region proposals, inferred from the RGB, we show how encoding the relationships between regions provides the necessary support to improve answer prediction. We evaluate over new datasets designed for the VQA depth problem.
University College London | London, UK
Humans have an innate ability to communicate visually; the earliest forms of communication were cave drawings, and children can communicate visual descriptions of scenes through drawings well before they can write. Drawings and sketches offer an intuitive and efficient means for communicating visual concepts. Today, society faces a deluge of digital visual content driven by a surge in the generation of video on social media and the online availability of video archives. Mobile devices are emerging as the dominant platform for consuming this content, with Cisco predicting that by 2018 over 80% of mobile traffic will be video. Sketch offers a familiar and expressive modality for interacting with video on the touch-screens commonly present on such devices. This presentation contributes several new algorithms for searching and manipulating video using free-hand sketches. We propose the Visual Narrative (VN); a storyboarded sequence of one or more actions in the form of sketch that collectively describe an event. We show that VNs can be used to both efficiently search video repositories, and to synthesise video clips. First, we describe a sketch based video retrieval (SBVR) system that fuses multiple modalities (shape, colour, semantics, and motion) in order to find relevant video clips. An efficient multi-modal video descriptor is proposed enabling the search of hundreds of videos in milliseconds. This contrasts with prior SBVR that lacks an efficient index representation, and take minutes or hours to search similar datasets. This contribution not only makes SBVR practical at interactive speeds, but also enables user-refinement of results through relevance feedback to resolve sketch ambiguity, including the relative priority of the different VN modalities. Second, we present the first algorithm for sketch based pose retrieval. A pictographic representation (stick-men) is used to specify a desired human pose within the VN, and similar poses found within a video dataset. We use archival dance performance footage from the UK National Resource Centre for Dance (UK-NRCD), containing diverse examples of human pose. We investigate appropriate descriptors for sketch and video, and propose a novel manifold learning technique for mapping between the two descriptor spaces and so performing sketched pose retrieval. We show that domain adaptation can be applied to boost the performance of this system through a novel piece-wise feature-space warping technique. Third, we present a graph representation for VNs comprising multiple actions. We focus on the extension of our pose retrieval system to a sequence of poses interspersed with actions (e.g. jump, twirl). We show that our graph representation can be used for two applications: 1) to retrieve sequences of video comprising multiple actions; 2) to synthesise new video sequences by retrieving and concatenating video fragments from archival footage.
University of Surrey | Surrey, Portugal
University of Surrey | Surrey, UK
INESC-ID | Lisbon, Portugal
University of Manchester | Manchester, UK
Additional Honers - BMVA Summer School Poster Competition Runner Up Award