Progress in 2007 and 2006

Current approaches to the retrieval of digital cultural objects are limited because they are keyword-based and cannot deal with high-level abstract concepts. This project aims to integrate the content-based approach of computer vision with the descriptive-based, metadata approach of information to provide more effective techniques for the indexing and retrieval of cultural objects.

We have developed a hierarchical representation scheme for artworks with semantic structures. This consists of three levels of information: low (basic metadata, visual features, textual attributes); medium (relationships: spatial, temporal, grouping, categorical, associative); and high (abstract concepts: semantics, context, symbolism, etc.). We have also designed a broad system for semantics and context-based indexing and retrieval of artworks. The following components have been completed:

• a method for detecting features and object categories (e.g., 'bird', 'people') from an image using deformable templates;
• a module extension of the Loculus metadata wrapper (developed in the Standards and Metadata Project) to cover the representation scheme 'Artefact' for cultural objects;
• an Expression Language for cultural objects, to allow effective representation, information parsing, and communication between users and systems;
• a web-based gallery of traditional Vietnamese woodcuts to facilitate the gathering of user tags and guiding of tag contribution (the gallery also serves as a demonstration showcase for the technologies as they are developed. There is also a Chinese version); and
• a scheme to generate high-level metadata (semantics, context, symbolism) from a combination of basic metadata, visual features and annotations using an inference engine.

Initial discussions were held with the East China Normal University in Wuhan and the Central China Normal University in Shanghai on potential collaboration opportunities.

Binh Pham spent three weeks at the Smithsonian Institutions in Washington D.C. (Freer & Sackler Galleries, Smithsonian American Art Museum, American Art Archives, Smithsonian Library) presenting this work and holding discussions with curators, librarians and technical staff on trends and obstacles in the management of digital artwork and cultural objects.

Rob Smith (Postdoctoral Fellow) resigned in July 2007 to take up a position in industry. Binh Pham took two months of long-service leave in August and September. Dr. Jinglan Zhang (a Lecturer in Faculty of IT) and a part-time Research Assistant joined the research team in July.

Progress in 2006
A semantic and context-based representation model for digital artworks has been developed, which integrates content-based indexing approach used in computer science with the description-based approach used in information science. We have also developed a web-based prototype to demonstrate this new integrated approach to image indexing and retrieval.

Computer vision techniques have been constructed for robust object category detection using a newly developed augmented deformable shape template.

A broad system design for an indexing and retrieval system has been completed, which uses a Bayesian network, machine trained classifiers, deformable templates and user-defined heuristics. We are currently building each of the components that will provide a separate source of image data for use by the system.

In August 2006 at the iCi symposium, we demonstrated our web-based art gallery to facilitate the gathering of user tags where prompting strategies were experimented with to guide the tag solicitation. This work will be the basis of one module in the indexing and retrieval system.