Content-Based Semantic Retrieval of Video Documents, Application to Navigation, Research and Automatic Content Abstraction.

2007-2009 (project director Lect. Dr. Ing. Bogdan IONESCU)

 CNCSIS project RP-2

Global results overview (more details about the techniques are provided in the extended version of the site, i.e. on the previous page click on the desired language/flag)

Scientific results:

 

- developing methods for the temporal segmentation of image sequences (see phase 2008.1.2.B.), i.e. cut detection (histogram-based, see phase 2008-1.1.A.), fade detection (statistics of pixel intensities, see phase 2008-1.1.B.) and dissolve detection (shape analysis of the number of fading-in and fading-out pixels, see phase 2008-1.2.A.);

 

- developing methods for the semantic/symbolic characterization of the motion content (motion estimation techniques, parameterization of the motion information, symbolic characterization of the camera movement, see phase 2008-1.3.A.-C.);

 

- developing methods for the semantic/symbolic characterization of the color content (global color distribution, parameterization of the color information, color perception, see phase 2008-2.1.A.-B.);

 

- developing methods for the semantic/symbolic characterization of the temporal structure (parameterization of the temporal information, determining the action content, semantic characterization of the temporal information, see phase 2008-2.2.A.-B.);

 

- tackling automatic movie genre classification (clustering and classification techniques, detecting animated movies, see phase 2009-1.1.);

 

- developing methods for the construction of automatic video summaries (content-based adaptive summary, compact summary, see phase 2009-1.3.);

 

- developing methods for the construction of automatic video skims and trailers (see phase 2009-1.3.);

 

- extensive state-of-the-art of the literature in the tackled domains.
Text Box: Technical results: During the project, we have developed a software solution for content-based video indexing, which is composed of several modules, thus: - the Movie Analyser application integrates all the video analysis and processing techniques developed during this project. This is a practical tool for future research, experimental validation as well as for accessing movie contents. Within the overall indexing system, its role is to process offline the movies and to extract content parameters for further use in the system (see the figure below). - the Movie Classification Interface is a graphic tool for developing automatic movie genre classification techniques. It uses a SVM classification technique. It allows the user to specify the feature space and to analyze the classification results on a preselected video database. The input data are the parameters extracted with the Movie Analyzer (see figure below). - the MovieGlobe Navigation System represents the heart of the video indexing system. It retrieves the movie features from the Movie Analyser and from the Classification Interface in order to represent movie contents in a 3D virtual world. The system allows the user to fly and to explore movie contents in a unique 3D world of movies. One of its major features represents the Java client-server architecture which allows remote user access through a simple Internet browser. Also, its modular architecture allows future improvements and upgrades (see figure below).  The content-based video indexing system (for more images click on pictures).

International collaborations:

 

During the project, we have initiated several new collaboration with foreign laboratories, as well as with private companies from the multimedia or video processing fields, while pursuing the existing ones, thus:

 

- the collaboration with laboratory LISTIC - Laboratoire d’Informatique, Systèmes, Traitement de l’Information et de la Connaissance, Polytech’Savoie, University of Savoie, Annecy-France (Prof. Patrick Lambert, Prof. Philippe Bolon), which provided support for implementing the project;

 

- the collaboration with the company CITIA - Cite de l'Image en Mouvement, Annecy-France (owner of the International Animated Film Festival of Annecy, Assoc. Prof. Daniel Bouillot), which provided access to several animated movies from the “Animaquid” indexing system;

 

- the collaboration with laboratory VIVIA - Video Image Vision and Autonomous Systems, University of Ottawa, Canada (Assoc. Prof. Robert Laganiere), which provided support for implementing the project;

 

- the collaboration with the company CERIMES – Centre de Ressources et d’Information sur les Multimédias pour L’Enseignement Supérieur al Ministère Enseignement Supérieur et Recherche Français, Vanves-France (Mr. Emmanuel Quillet), which provided full access to a database of documentary movies;

 

- two experience exchanges at the laboratory LISTIC, Polytech’Savoie, Annecy-France; cooperation between the CNCSIS RP-2 project and the Rhône-Alpes regional project LIMA – Loisir et IMAges;

 

- attending with laboratory LISTIC, Polytech’Savoie and laboratory VIVA, University of Ottawa to the Trecvid 2008 international video benchmarking campaign where the proposed video skimming method took a middle position in the list.

 

- experience exchange at the laboratory VIVA – Video Image Vision and Autonomous Systems, University of Ottawa, Canada; cooperation between the CNCSIS RP-2 project and the project SSHRC - Image, Text, Sound & Technology (ITST) – “Media-ting the Bible: Searching for Scripture in Digital Image and Sound”.

 

- initiating a contact with laboratory DISCOVER Lab - Distributed & Collaborative Virtual Environments Research Laboratory, University of Ottawa, Canada, (Prof. Emil Petriu, head of the laboratory), which works in the fields of virtual reality, artificial intelligence and human-machine interaction;

 

- initiating a collaboration with the company Eboo Solutions, Annecy-France (Dr. Ing. Johan Tailland and Dr. Ing. Michel Cintract), which develops solutions for smart video surveillance applications;

 

- experience exchange and initiating a collaboration with the CNRS laboratory LI - Laboratoire d'Informatique, Université de Marne-la-Vallée, Paris (Prof. Jean-Christophe Pesquet, CNRS senior researcher Caroline Chaux), which works in the fields of signal processing;

Dissemination of the results:

 

- publishing an international journal paper (ISI, INSPEC): B. Ionescu, D. Coquin, P. Lambert, V. Buzuloiu „A Fuzzy Color-Based Approach for Understanding Animated Movies Content in the Indexing Task”, Eurasip Journal on Image and Video Processing, special issue “Color in Image and Video Processing”, doi:10.1155/2008/849625, 2008 (17 pages).

 

- giving the thematic seminar  “Introduction to Video Indexing and Its Applications” and presenting the project achievements so far, laboratory VIVA, University of Ottawa, Canada, http://www.site.uottawa.ca/ research/viva/seminars.html, 2008 (50 minutes).

 

- publishing an international conference paper (ISI): R. Laganiere, R. Bacco, A. Hocevar, P. Lambert, G. Pays, B. Ionescu, "Video Summarization from Spatio-Temporal Features", ACM International Conference on Multimedia, Trecvid BBC Rushes Summarization Workshop, Vancouver, Canada, 27 October - 1 November, 2008 (5 pages).

 

- contributed with an article to the magazine „The Story of My Return” (presenting the context of the RP-2 project), CNCSIS, conference CNCSIS 10 - The National Conference of the Research, 15-17 may, Brasov, 2008.

 

- publishing an international journal paper (ISI): B. Ionescu, L. Ott, P. Lambert, D. Coquin, A. Pacureanu, V. Buzuloiu, “Tackling Action-Based Video Abstraction of Animated Movies for Video Browsing”, submitted in May 2009 to SPIE JEI - International Journal on Electronic Imaging (accepted paper, 15 pages).

 

- publishing an international book chapter: B. Ionescu, P. Lambert, D. Coquin, A. Marin, C. Vertan, “Analyzing Animated Movie Contents for Automatic Video Indexing”, submitted in July 2009 to “Machine Learning Techniques for Adaptive Multimedia Retrieval: Technologies Applications and Perspectives”, IGI – Global (accepted paper, 28 pages).

 

- publishing a national volume: B. Ionescu, “Analysis and Processing of Video Sequences – Automatic Content-based Indexing”, “Editura Tehnica Bucuresti” Publishing House, ISBN 978-973-31-2354-5, 2009 (270 pages in Romanian).

 

- publishing an international conference paper (ISI):  B. Ionescu, A. Păcureanu, P. Lambert, C. Vertan, “Highlighting Action Content in Animated Movies”, IEEE ISSCS -  International Symposium on Signals, Circuits and Systems, special session "Action Recognition and Tracking in Multimodal Video", 9-10 July, Iaşi, Romania, 2009 (4 pages).

 

- publishing an international conference paper (ISI): C. Vertan, M. Zamfir, B. Ionescu, “Non-Linear Color Image Processing by Color to Plane Polygon Association”, IEEE ISSCS -  International Symposium on Signals, Circuits and Systems, 9-10 July, Iaşi, Romania, 2009 (4 pages).

 

- including the project thematic with student projects and diploma projects at the University “Politehnica” of Bucharest, thus promoting the video indexing field, which is still at its beginnings in Romania.

 

- including the video indexing domain to the new Master formations at the Faculty of Electronics, Telecommunications and Information Technology, Department of Applied Electronics and Information Engineering.

· Scientific results

· Technical results

· Infrastructure

· International collaborations

· Dissemination of the results

Infrastructure:

 

With the project funds we have created new research and teaching infrastructures and we have improved the existing ones from the laboratory LAPI - The Image Processing and Analysis Laboratory, Faculty of Electronics, Telecommunication and Information Technology, University “Politehnica” of Bucharest. We list here the most significant contributions:

 

- creating a new laboratory focusing on video indexing (room B144); complete restoration of the rooms B135A, B138, B139, B141, B143 and B144; air conditioning the rooms B144, B135A and B143; installing new office and laboratory furniture in rooms B135A, B138, B139, B143 and B144.

 

- installing video projection systems in laboratories B135A and B138; purchasing portable video projection systems; installing whiteboard magnetic tables in the laboratories as well as in the offices;

 

- purchasing IT infrastructure (high-end workstations, one graphic workstation with quad Xeon, laptop and mini-laptop systems, peripheral devices, storage devices, WiFi infrastructure, UPS, etc.);