Transfer Learning for Image Aesthetic Assessment
The project aims to develop transfer learning methods for deep
convolutional networks. In this framework, besides the main database,
which allows supervised learning, we will consider a second database,
larger in volume, that contains examples either without annotations or
with labels, but for another, related, problem. The transfer learning
involves extracting useful information from the second database to
increase performance on the primary set, in a simultaneous process.
For the development of the transfer learning algorithm, the
characteristics of the convolutional networks to form descriptors on
the intermediate layers, respectively to be sequentially trained are
exploited. The central algorithmic principle is the partitioning by
clustering (if the data does not have labels) or by classifying the
secondary data set, where the descriptors of data are extracted on an
intermediate layer. The network is required to create descriptors that
provide wide margins between the data of one group and the centroid
of another group. The groups can be associated with the classes from
the primary database and the method of calculating the wide margin
will be explored. Secondly, for the harmonization of the two databases
we will use regularization by injecting a random perturbation, in an
annealed process, into the global gradient.
From an application point of view, we aim to estimate the aesthetic
value of a photograph. The aesthetic value aggregates, consciously or
unconsciously, how pleasing an image looks. For high performance, in
this direction we will exploit the previously developed transfer
learning algorithm. The application is useful in commercial area to
offer customers visual quality.
Objectives and achievements
The project aimed to work on two directions:
- [O1]. Transfer Learning and CNN layer structuring
- [O2]. Image Aesthetics which is to estimate automatically the aesthetic value of an image.
The directions were aimed to be distinct conceptually, but also to communicate between them.
[O1]. On the
transfer learning direction - the objective was to develop the individual concepts and
combine them in robust algorithms that are able to efficiently transfer information from a secondary database
to a primary subject one. The individual concepts identified at the proposal stage were:
- [O1.1] Large margin . This means to ask that the data from the large unlabeled set
enforces large margins between pseudo-classes using on the encodings (taken from some
layer before the decisional one) to cluster according to classes and the distance
between a point and other cluster centroids should be used.
Achievements
- - We have showed that this idea is feasible and useful for more difficult
(from a machine learning point of view) problems such as face expression.
The resulting algorithm was named Margin-Mix and published in ref [1]
[here].
A variation, also on on two databases, this time even with distinct annotations on the two databases
is proposed for publication in ref [R4] [here]
- - We have showed that this idea is also useful for training in supervised manner a single database
which is more difficult. The algorithm was named Soft-Mix and proposed for publication in [R3] [here]
- [O1.2] Annealed regularization which is to injecting random perturbation into the overall gradient
in a two database problem, one is able to cope with a bias between the two databases.
Achievements
- - We have showed the feasibility of this idea in various experiments in either aesthetic evaluation
[4] [here] or in [6] [here] in a single database scenario.
- - The idea of injecting randomization in two database was shown to work
for other more difficult problems such as face expression is proposed for publication in [R2] [here]
- [O1.3] Semi-supervised learning via clustering loss this is a derivation of objective [O1.1] with direct application in semisupervised learning
Achievements
- - The concept is strong enough to boost significantly the performance of small networks as showed in [7]
which is [here]
[O2]. Image Aesthetics. The objective here, was given an image to provides scores from
aesthetic point of view. Two approaches are taken commonly: one is to give a holistic and unique grade to the
overall image; the other is to asses specifically various aesthetic elements.
Achievements
Both approaches have been attempted. In a structured way achievements are:
- - Performance on the holistic grade are in [4] [here] and respectively
[R3] [here]
- - Performance on the attributes are in [5] [here]
- The available databases did not unify the two approaches. To work on that, we have used many observers to annotate a set of images with attribute.
Those images had already holistic annotation. The process of anotation finished in June 2022 and results are still under analysis.
The results will be made public as A-AVA database in the Download section.