A Multi Criteria Decision Making Based Approach for Semantic Image Annotation

International Journal of Computer-Aided Technologies (IJCAx) Vol.2,No.1,January 2015
17
A Multi Criteria Decision Making Based Approach
for Semantic Image Annotation
Hengame Deljooi and Somayye Jafarali Jassbi
Department of Computer Engineering, Science and Research Branch, Islamic Azad
University, Tehran, Iran
ABSTRACT
Automatic image annotation has emerged as an important research topic due to its potential application on
both image understanding and web image search. This paper presents a model, which integrates visual
topics and regional contexts to automatic image annotation. Regional contexts model the relationship
between the regions, while visual topics provide the global distribution of topics over an image. Previous
image annotation methods neglected the relationship between the regions in an image, while these regions
are exactly explanation of the image semantics, therefore considering the relationship between them are
helpful to annotate the images. Regional contexts and visual topics are learned by PLSA (Probability
Latent Semantic Analysis) from the training data. The proposed model incorporates these two types of
information by MCDM (Multi Criteria Decision Making) approach based on WSM (Weighted Sum
Method). Experiments conducted on the 5k Corel dataset demonstrate the effectiveness of the proposed
model.
KEYWORDS
Automatic Image Annotation, Regional Contexts, Visual Topics, PLSA, Multi Criteria Decision Making
1.INTRODUCTION
With the exponential growth of the Web image, how to retrieval and manage them presents a
significant challenge. Content-based Image Retrieval (CBIR) methods compute relevance
between images based on the visual similarity of the low-level features [1]-[3]. However, the
problem is the so-called “semantic gap” [4] between low-level features and high-level semantic
concepts. Therefore, CBIR technique is seldom adopted in practice.
Current commercial image search engines utilize the surrounding textual descriptions of images
on the hosting web pages to index web images. These search engines assume the surrounding
textual descriptions of images are related to the semantic content of web images. Consequently,
the textural descriptions can be used as approximate annotations of web image [3]. On the other
hand, the indexing method has many drawbacks. Firstly, there are too many irrelevant words in
the surrounding textual descriptions, which results in low image search precision rate. Secondly,
the surrounding text does not seem to fully describe the semantic content of Web images, which
results in low image search recall rate. Finally, it cannot be used to retrieval for web images
without textual descriptions.
To overcome these shortcomings, Automatic Image Annotation (AIA) has attracted a great deal
of attention in recent years.

18
1.2. Related Works
Existing automatic image annotation methods can be classified into four categories: Vector Space
Models [5], [6], Classification Methods [7]-[10], Graph-based Methods [11]-[14] and Statistical
Models [15]-[21].
The Vector Space Models: The vector space model [5], [6] framework is a popular technique in
information retrieval especially in text retrieval. Generally, documents are represented as vectors,
each of which contains the occurrences of keywords within the document in question. The length
of the vectors is equal to the vocabulary size. These approaches treat images as documents, and
build visual terms which are analogous to keywords, from the image feature descriptors. Some of
the methods in this category include: SvdCos method, Saliency-based semantic propagation, and
cross-language latent semantic indexing based approach.
Classification Methods: Classification approaches [7]-[10] for image annotation view the
process of attaching keywords to images as that of classifying images to a number of pre-defined
groups, each of which is characterized by a concept or keyword. These approaches pose image
annotation as a supervised classification problem. Specifically, each keyword is viewed as a
unique class. Binary classifiers for each class or a multiclass classifier is trained independently to
predict the annotations of new images. Given an un-annotated image, the classification algorithms
find its membership and annotate it with the corresponding keyword. Multiple annotations can be
generated by assuming an image belongs to multiple classes. Some of the works in this category
include: image linguistic indexing, image annotation using SVM, multiple instance learning
approaches, non-negative Matrix factorization based approaches.
Graph Based Methods: Recently, the graph-based methods [11]-[14] have achieved much
success in the image and video analysis domain including image annotation. How to build a
similarity graph is very important in graph learning. A good graph should reflect a deep
understanding of the data structure and help to mine potential knowledge as much as possible.
Some works in this category include: image annotation via graph learning, image annotation
refinement via graph based learning.
Statistical Models: The basic idea of statistical techniques [15]-[21] is to estimate the
probabilities of documents related to the query submitted by the user. Documents are then ranked
according to their probabilities. The main goal of statistical techniques is to learn a model from
the joint distribution of visual and textual features on the training data and predicting the missing
textual features for a new image.
Among the image annotation models classification approaches are the oldest in this domain. But
statistical models and graph based approaches have more chance. The proposed approach in this
paper follows statistical based models. So we will review the statistical models that are proposed
for automatic image annotation.
The co-occurrence model proposed by Mori et al. [16] is perhaps one of the first attempts at
automatic image annotation. They first divide images into rectangular tiles of the same size, and
calculate a feature descriptor of color and texture for each tile. All the descriptors are clustered
into a number of groups, each of which is represented by the centroid. Each tile inherits the whole
set of labels from the original image. Then, they estimate the probability of a keyword W related
to a cluster C by the co-occurrence of the keyword and the image tiles within the cluster.
Duygulu et al. [17] proposed a machine translation model for automatic image annotation. They
argued that region based image annotation is more interesting because global annotation does not

19
give information on which part of the image is related to which keyword. In their point of view,
the process of attaching keywords to image regions is analogous to the translation of one form of
representation (image regions; French) to another form (words; English). Indeed, Their Machine
translation model applies one of the classical statistical machine translation models to translate
from the set of keywords of an image to the set of blobs that generated by clustering the image
features.
Jeon et al. [18] proposed Cross Media Relevance Model (CMRM) for automatic image
annotation. CMRM benefits the joint distribution of keywords and blobs. Each image describe by
visual features (blobs) as well as textual keywords. In CMRM the probability of observing a set
of blobs with a set of keywords estimated.
Lavrenko et al. [19] proposed a Continues-space Relevance Model (CRM) that improves CMRM
by using continuous probability density functions to estimate the probability of observing a region
given an image.
Feng et al. [20] modified the above model and proposed Multiple Bernoulli Relevance Model
(MBRM) such that the probability of observing labels given an image was modeled as a multiple-
Bernoulli distribution. In MBRM images simply divided into rectangular tiles instead of applying
automatic segmentation algorithms.
There exist two problems in above algorithms. First, most existing algorithms have taken one of
two approaches, using regional or global features exclusively. Second, conventional approaches
consider each word separately without the textual context relations. As a result the textual context
relations among annotation words have been ignored. By textual context we refer to co-
occurrence relationship among words. Wang et al. [21] proposed a model for image annotation
that combines global, regional, and contextual features by an extended CMRM. They incorporate
the three kinds of information to describe image semantics to annotate images by estimating their
joint probability.
1.2.Our Approach
There exists an important problem in all of the statistical methods, that all the explained methods
utilize direct distribution of regions for AIA. However considering the relationship between
regions, that each of them is an explanation of a word, helps us to improve the final annotation
words. So we obtain the relationship between regions and instead of using the direct distribution
of regions, we utilize the distribution of topics between regions for AIA.
To address the above problem we propose a novel approach for AIA. Typical CMRM only takes
regional features to describe an image, the extended CMRM incorporates both global and
regional features, as well as textual context to annotate images. In the approach that we propose,
the visual topics have described as a global distribution vector of topics in the image, moreover
we consider the obtained relationship between regions and model the regional contexts as a
distribution of topics between regions. Both the visual topics and regional contexts are learned by
a PLSA approach [22] from the training data. Then we integrate these two kinds of information
by MCDM approach based on WSM. Generally our method includes three steps as follows:
1. Model the regional contexts or the relationship between the regions as a distribution of
topics between regions for generating the annotation related to image regions content.
2. Utilize the visual topics as a global distribution vector of topics to consider the overall
image content.

20
3. Combining the extracted two kinds of information from the image using the WSM in
MCDM algorithm.
The framework is shown in Fig. 1. The images are represented by their regions and collection of
patches. Our proposed model is learned from the training data based on the two kinds of
information, regional contexts ( )
B I and visual topics ( )
H I .
1.3. Paper Outline
The reminder of this paper organized as follows: Extended CMRM is described in section 2. Our
proposed model is introduced in section 3. In section 4 experimental results are presented and
finally concluding remarks are explained in section 5.
2. Extended CMRM
The conventional CMRM [18] considers only one representation of images, i.e. a bag of blobs. To
deal with images which are not suitable to be represented as a bag of blobs, we need to consider
other representation as well. So, ECMRM [21] has proposed to use visual topics as another
representation. This new image representation is combined with the visual blobs representation.
Moreover the typical CMRM annotates keywords individually without considering the joint
distribution of different keywords. To solve this problem, textual context is proposed. To annotate
an image with multiple keywords, first annotate the image with textual contexts and then
compose the keywords from the distribution of keywords under each textual context. The learning
of textual context is based on the PLSA approach [22]. PLSA is proposed to automatically learn
topics from text documents. Similar to learn topics from text documents, we can also learn visual
topics from images.
The ECMRM method annotates a test image I by estimating the joint probability of textual
contexts C, its visual blobs (regions) ( )
1 2
, ,..., m
R b b b
= obtained by image segmentation [23] and the
visual topics distribution ( )
H I , which J represents training image and τ is the training set:
( )
( ) ( ) ( )
( )
1
, , , ,..., ,
m
J
P C R H I P J P C b b H I J
∈τ
= ∑ (1)
Comparing ECMRM with CMRM, there are two points of difference to elaborate. First, the
original CMRM in annotates an image using only the regional features ( )
1 2
, ,..., m
R b b b
= . However,
ECMRM uses both the regional features R and the global features ( )
H I which represent the
global distribution of visual topics in image I. This suggests the ECMRM model combines the
global features and regional features. Second, CMRM predicts the probability of a single word W
directly, while ECMRM predicts the probability of a textual context C. This indicates ECMRM
does not assume the mutual independence between words given the image. Thus, the extended
CMRM incorporates the textual context from the training data.
Textual contexts C, image blobs and visual topics distribution are independent. So that
( )
( )
1
, ,..., ,
m
P C b b H I J can be simplified as:
( )
( ) ( ) ( )
( ) ( )
1
1
, ,..., ,
m
m i
i
P C b b H I J P C J P H I J P b J
=
= ∏ (2)

21
( )
P b J is estimated as the same as that in CMRM. ( )
P C I is available after learning textual
contexts on the manual annotations. ( )
( )
P H I J is defined as the Kullback-Leibler divergence
between the visual topic distribution of I and J:
Figure 1. The proposed approach to automatic image annotation using regional contexts and visual topics.
( )
( ) ( ) ( )
( ) ( )
( )
( )
1
log
Q
i
kl i
i
i
P q I
P H I J D H I H J P q I
P q J
=
= = ∑ (3)
In which kl
D is the Kullback-Leibler divergence between two distributions. From the Bayesian
theory, we know that:
( )
( )
( )
( )
( )
( )
1
, ,..., ,
, m
P C b b H I
P C I
P C I
P I P I
= = (4)
Therefore, normalization on ( )
( )
1
, ,..., ,
m
P C b b H I will give the conditional distribution of textual
contexts ( )
P C I . The conditional keyword distribution ( )
j
P W I of I is obtained by fusing the
keyword distribution of the entire textual contexts:
( ) ( ) ( )
S
j j i i
i
P W I P W C P C I
= ∑ (5)

22
3.The Proposed Scheme
The notable point is that the relationship between regions is not considered in the ECMRM
method. We know that each of these regions is explanation of a word and the semantic
relationship between these regions is more accurate than the semantic relationship between words
(textual contexts), which are assigned to the image by human. Because these regions have
represented the exact content of an image, finally the generated keywords are related to the image
content.
3.1.Learning Regional Contexts from Images
Similar to learning topics from text document in the ECMRM method, we can also learn regional
contexts from a collection of images. Firstly, according to ECMRM method the regions are
obtained from image segmentation based on objects [23]. We should represent an image as a
collection of words, so consider an image as a document and assume the objects in an image as
words in this document (we consider each region as a word). Then apply the PLSA approach
developed by Hoffman to reach the relationship between regions and instead of the direct
distribution of regions in an ECMRM, we utilize the distribution of topics between them to AIA.
We call these topics, which have described the relationship between regions, regional context.
PLSA approach [22] behaves as follows:
Suppose we are given a set of text documents { }
1 2
, ,..., n
D d d d
= each of which is represented by a
term frequency vector:
( ) ( ) ( )
1 2
, , , ,..., ,
i i i i m
d n d w n d w n d w
 
=   (6)
In which ( )
,
i j
n d w is the number of occurrence of word j
w in document i
d , and m is the
vocabulary size. PLSA assumes that each word in a document is generated by a specific hidden
topic k
Z , where k
Z I
∈ and I is the vocabulary of hidden topics. Since k
Z is a hidden variable, the
conditional probability of a word j
w given document i
d is a marginalization over the topics:
( ) ( ) ( )
,
K
j i j k i k i
k
P w d P w Z d P Z d
= ∑ (7)
In which K is the number of hidden topics, ( )
,
j k i
P w Z d is the conditional probability of a word
j
w given topic k
Z and the document i
d , ( )
k i
P Z d is the conditional probability of topic k
Z
given i
d . Furthermore, PLSA assumes that the conditional probability of generating a word by a
specific topic is independent from the document:
( ) ( )
,
j k i j k
P w Z d P w Z
= (8)
Therefore (7) can be simplified as:
( ) ( ) ( )
K
j i j k k i
k
P w d P w Z P Z d
= ∑ (9)

23
The model parameters ( )
j k
P w Z and ( )
k i
P Z d can be learned by an EM algorithm [24].
3.2. Learning Visual Topics from Images
Similar to learning topics from text documents, we can also learn visual topics from images. The
notable point is representing an image as a bag of words [25], like to the vector representation of
text documents. We partition an image by a regular grid and take it as an unordered set of image
patches. Then we extract a Gabor descriptor [26]-[28] and vector quantizes each image patch by
clustering a subset of patches from the training images, which has proved effective for object
recognition [28]. Therefore, we develop the orientation histogram [29] from multi-scale Gabor
features because Gabor features are better representation than simple gradient features. We call
the cluster centers as visual words. Then we can transform an image into a bag of visual words by
assigning a visual word label to each image patch. We have the bag of visual word representation,
apply the PLSA to learn a set of visual topics.
There is a difference between image regions obtained by image segmentation and image patches.
Image patches that grouped in a topic do not have spatial agglomeration and visual consistency
although image segmentation groups pixels by its visual property and spatial location. Fig. 2
illustrates the image regions, image patches and the distributions of regional context and visual
topics of an image. Visual topics and regional contexts (topics between regions obtained by image
segmentation) focus on different aspect of an image, so they are complementary to each other and
a combination of them is expected to achieve better performance.

24
Figure 2. Illusrtration of two types of information: (A) Original image, (B) Image regions
haveobtained by segmentation, (C) Distribution of regional contexts, (D) Image patches that the
patches with the same topic are indicated by the same color, (E) Distribution of visual topics
3.3. Combining Regional Contexts and Visual Topics
Our method annotates a test image I by combining the regional contexts (the distribution of topics
between regions) ( )
B I and the visual topics distribution ( )
H I and considering a given word (W):
( ) ( )
( ) ( ) ( ) ( )
( )
, , , ,
J
P W B I H I P J P W B I H I J
∈τ
= ∑ (10)
Comparing (10) with (1) there is one point of difference to elaborate: ECMRM in (1) annotates an
image uses both the regional features R and the global features ( )
H I , means that it uses the direct
distribution of regions. But our method in (10) considers to the relationship between the regions
and it uses the distribution of topics between regions as a regional contexts ( )
B I . Because the
semantic relationship between regions is more accurate than the relationship between words, we
have ignored the textual context in (1) and instead of it we utilize regional contexts. So we utilize
both the regional contexts and visual topics together. These two kinds of information extract
different words, so combining them are useful to create related words and consistent to image
content.
However the conventional methods suppose the conditionally independence between the regional
information and visual words, but we cannot consider this assumption. It is obvious that to obtain
the regional information we cannot utilize the image directly and we should use the image
features domain. Subsequently regional information is dependent to the visual words, because the
visual words are in this domain too. This dependency stems from that one of the important
extracted features from the image needed for regional representation is texture which cannot be
independent from the visual words.
To solve this problem we explain the distribution of test image keywords generally as below
equation:
( ) ( ) j
J
p w I p w J
τ
α
∈
= ∑ (11)
This equation is similar to (10), but we utilize j
α instead of considering the distribution of ( )
B I
and ( )
H I . According to (11) the annotation of a test image is equal to the sum of multiply the
distribution of training image keywords in their coefficient ( j
α ). This coefficient calculates the
similarity of a test image I with the training images, considering to (10) in our proposed model
j
α is the conditional joint distribution of the regional contexts and visual topics. Indeed j
α
define the contribution of training image keywords to create the distribution of test image
keywords. We can understand easily that if one training image is more similar to the test image I,
the distribution of their keywords is similar too, and the contribution of that training image or j
α
should be more. As regards ( )
B I and ( )
H I are dependent and to simplify (10) and calculate j
α or
the similarity between the test image and training images we use the MCDM (Multi Criteria
Decision Making) approach. MCDM is the selection of the best action from a set of alternatives,
each of which is evaluated against multiple criteria.

25
Among the MCDM methods, we utilize the WSM (Weighted Sum Method).
Inputs to WSM methods are as follows:
• WSM assumes that we have N alternatives (options) and M attributes (criteria) and we
have the score of each option with respect to each criterion. In our model the alternatives
are training images and the criteria are regional contexts and visual topics. Indeed to
determine the criteria we should introduce some measure to calculate the similarity. We
obtain these measures by investigating the various experimental results. In the different
experiment, we fetch up that ( )
( )2
p B I
 
 
 
and ( )
( )
p B I
 
  increase the precision and
( )
( ) ( )
( )
*
p B I p H I
 
  has an influence to boost the recall, so we specify these measures:
( )
( ) ( )
( ) ( )
( ) ( )
( )
2
, ,
p B I J p B I J p B I J p H I J
 
 
 
(12)
• Let xij score of option i with respect to criterion j, we have the m×n matrix. Then we
construct normalized decision matrix. This step transforms various attribute dimensions
into non-dimensional attributes, which allows comparisons across criteria. rij is the
normalized rating of the of the th
i alternative for the th
j criterion.
• Construct the weighted normalized decision matrix. Assume we have a set of weights for
each criterion w j for j=1,...,n . These weights express the relative importance of it in
respect to the others. Considering the 3 suggested criteria in our model, we need to define
3 weights for them:
6
1-
7 7
a
a a
 
∗
 
 
(13)
• Calculate the i
S , i
S is the overall score of th
i alternative.
1
for i=1,2,...,N
M
i j ij
j
S w r
=
= ∑ (14)
Note that in this method the alternative with the highest score is selected as the preferred one.
Considering to the suggested criteria, we need to calculate the ( )
( )
P B I J and ( )
( )
P H I J . These
two items are defined as the Kullback-Leibler divergence between the distribution image I and
image J, according to (3) ( )
( )
P H I J is available:
( )
( ) ( ) ( )
( ) ( )
( )
( )
1
log
Q
i
kl i
i
i
P q I
P B I J D B I B J P q I
P q J
=
= =∑ (15)
4.Experimental Results and Discussion
In this section we will discuss details of the dataset and also show experimental results. The test
bed and evaluation metrics are introduced in Section 4.1 and Section 4.2, respectively. The
experimental results with compared to other annotation models are presented in Section 4.3.

26
4.1. Test Bed
We have used a set of Corel images consists of 5000 images to evaluate the proposed model. The
image database consists of 50 main topics and each class includes 100 images. COREL-5K was
first collected by Duygulu et al. [17] and it has been extensively used by other researchers, so it
has been known as a de facto standard dataset in image annotation researches. Each image is also
associated with 1-5 keywords. Therefore, there are 374 keywords in the dataset.
We partition the whole data set into a training set and a test set, 4500 training images and 500 test
images. For the region features, we use the JSEG algorithm [23] to segment each image into 1--
11 regions. Image regions with area less than 1/25 of the whole image are discarded. In average
there are five image regions per image. Each image region is represented by a 36-D feature
vector. For the dense grid, we sample 13×13 pixels image patches without overlapping. The
average number of image patches per image is around 550. The image regions are clustered into
500 image blobs, similarly the Gabor descriptor of image patches are clustered into 500 centered.
We experiment on different number of visual topics (V), textual context (T) and regional contexts
(R) and achieve the best number of them that are shown in results.
4.2. Evaluation Metrics
For each method, we take the top five words as the final annotation. We use precision, recall and
1
F metrics, to evaluate the quality of our approach. Precision and recall are calculated on the basis
of each keyword in the vocabulary. Then the values are averaged over the keywords in the
vocabulary.
Precision is defined as the number of images correctly predicted with a given keyword called r,
divided by the total number of images predicted with the keyword called n. Recall is defined as
the number of images correctly predicted with a given keyword, divided by the total number of
images having this keyword in its ground-truth called N.
( )
Pr
r
ecision W
n
= (16)
( )
Re
r
call W
N
= (17)
( )
1
2 Pr Re
Pr Re
ecision call
F
ecision call
× ×
=
+
(18)
We also use keyword number with recall>0 to show the diversity of correct words that can be
predicted by the automatic image annotation model. The measure is important because a model
can achieve high precision and recall values by only performing well on a small number of
common words.
4.3. Comparative Analysis
The results of experiments are shown in Table 1. Comparing the results in a table, the
improvement of ECMRM [21] which incorporates the regional, global and contextual features to
original CMRM [18] is obvious and improved F1value from 0.1115 to 0.3270. But in the
proposed approach, we combine the regional contexts and visual topics, which has achieved the
best performance and improved F1value from 0.3270 to 0.4665. In addition precision, recall and
keyword number with recall>0 measure have been significantly improved. In summery using the

27
regional contexts (neglect the textual context), significantly improve the performance, because the
regions or objects in an image are exactly the explanation of image content. Also each of these
regions acts as one word, so the relationship between the regions is more accurate than textual
contexts to create annotation keywords. Textual contexts have some noises, because the keywords
of training images are created manually and by human, consequently they have inconsistence. It
is obvious that incorporating this information with visual topics help us to achieve the best
performance, because these two kinds of information focus on different aspects of an image,
extract different words and they are complementary to each other. Fig. 3 shows F1 value of
sample words in CMRM, ECMRM and the proposed approach. The difference between the
F1value of our method and ECMRM of sample words in Fig. 3 shows the noises in the manual
annotation and textual contexts of ECMRM method. We can observe that considering the
relationship between regions is helpful to obtain the annotation keywords related to image content
and increase the F1value. Some test images with the annotations generated by the CMRM,
ECMRM and the proposed approach are demonstrated in Fig. 4. It is observed that the proposed
approach can yield better annotations, which describe image semantics, than ECMRM.
Table 1. The average precision, recall, F1 and keywords with recall>0 values of experiments
The Proposed
Approach
ECMRM [21]
CMRM [18]
Models
0.4942
0.3050
0.1288
Precision
0.4417
0.3524
0.0983
Recall
0.4665
0.3270
0.1115
F1
138
129
66
Keywords with
recall>0

28
Figure 3. Some sample words and their F1 value in CMRM, ECMRM and the proposed approach
Image
Ground
Truth
Clouds, Sea,
Sun, Tree
Gate, Grass,
Temple, Tree
Cars, Formula,
Tracks, Wall
Jet, Mountain,
Plane
Bear, Polar,
Snow, Tundra
Cat, Forest,
Grass, Tiger
CMRM
Buildings,
Sunset, Skyline,
City, Water
People,
Buildings, Flag,
Parade, Sky
People, Water,
Cars, Tree, Sky
Sky, Plane, Jet,
Birds,
Albatross
Water, Sky,
Bear, Tree,
Plane
Water, Tree,
Grass, Garden,
Plants
ECMRM
Buildings,
Sunset, Water,
People, Skyline
Water, Sky,
People,
Buildings, Boats
People, Cars,
Water, Tracks,
Street
Sky, Plane, Jet,
Birds, People
Bear, Polar,
Tree, Water,
Snow
Grass, Plants,
Cat, Garden,
Tree
Proposed
Approach
Water, Tree,
Sky, Sunset,
Skyline
Grass, Tree,
Temple, Sky,
People
Cars, People,
Tracks, Street,
Race
Sky, Plane, Jet,
Tree, Clouds
Bear, Polar,
Snow, Ice,
Water
Cat, Grass,
Plants, Forest,
Garden
Figure 4. Some sample words and their F1 value in CMRM, ECMRM and the proposed approach

29
5.CONCLUSIONS
In this paper, we have proposed a method for AIA which is improved the result of ECMRM
method. Instead of using the direct distribution of regions in ECMRM, we utilize the distribution
of topics between regions. Moreover regional contexts are more accurate than textual contexts or
the relationship between keywords in ECMRM. The proposed approach combines the regional
contexts and visual topics for automatic image annotation, as for the dependence between these
two kinds of information in the image, we use the MCDM approach based on WSM to integrate
them. To obtain global distribution of topics, visual topics are learned from the training images by
a PLSA approach. Furthermore, PLSA has used to model the regional contexts or the topics
between regions. The proposed method is tested on a 5000 Corel data set and the results show
that utilization of regional contexts or considering the relationship between regions improves the
performance significantly and its incorporation with the visual topics leads to the best
performance.
REFERENCES
1] A. Tousch, S. Herbin, J. Audlibert, (2012) “Semantic Hierarchies for Image Annotation: A Survey,”
Elsevier Ltd., Pattern Recognition, vol. 40, pp. 333-345.
[2] D. Zhang, Md.M. Islam, G. Lu, (2011) “A review on automatic image annotation techniques,” Elsevier
Ltd., Pattern Recognition, vol. 45.
[3] R. Datta, D. Joshi, J. Li, J.Z. Wang, (2008) “Image retrieval: ideas, influences, and trends of the new
age,” ACM Comput. Surv, vol. 40, no. 2, Article 5.
[4] Y. Liu, D. Zhang, G. Lu, W. Ma, (2007) “A survey of content-based image retrieval with high-level
semantics,” Elsevier Ltd., Pattern Recognition, vol. 40, pp. 262-282.
[5] .Y. Pan, H.J. Yang, P. Duygulu, Ch. Faloutsos, (2004) “Automatic image captioning,” In Proceedings
of the IEEE International Conference on Multimedia and Expo (ICME), pp. 1987-1990.
[6] .S. Hare, P.H. Lewis, (2005) “Saliency-based models of image content and their application to auto-
annotation by semantic propagation,” In Proceedings of Multimedia and the Semantic Web / European
Semantic Web Conference.
[7] C. Cusano, G. Ciocca, R. Schettini, (2003) “Image Annotation Using Svm,” In Proceedings of Internet
Imaging IV, SPIE 5304, vol. 5304, pp. 330-338.
[8] E. Chang, K. Goh, G. Sychay, G. Wu, (2003) “CBSA: content based soft annotation for multimodal
image retrieval using Bayes point machines,” In Proceedings of IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), pp. 26-38.
[9] J. Li, J. Wang, (2003) “Automatic linguistic indexing of pictures by a statistical modeling approach,”
IEEE Trans. on PAMI, 25(19), pp. 1075-1088.
[10] A. Fakhari, A. M. Eftekhari Moghadam, (2013) “Combination of classification and regression in
decision tree for multi-labeling image annotation and retrieval,” Elsevier Ltd., Applied Soft
Computing, Vol. 13, pp. 1292-1302.
[11]J. Tang, H. Li, G. Qi, T. Chua, (2010) “Image annotation by graph-based inference with integrated
multiple/single instance representations,” IEEE Trans. on Multimedia, vol. 12, no. 2.
[12]J. Liu, M. Li, Q. Liu, H. Lu, S. Ma, (2009) “Image annotation via graph learning,” Elsevier Ltd.,
Pattern Recognition, vol. 42, pp. 218-228.
[13]J. Liu, B. Wang, H. Lu, S. Ma, (2008) “A graph-based image annotation framework,” Elsevier Ltd.,
Pattern Recognition Letters, vol. 29, pp. 407-415.
[14]J. Liu, M. Li, W. Ma, Q. Liu, H. Lu, (2006) "An Adaptive Graph Model For Automatic Image
Annotation," In Proceedings of the 8th ACM international workshop on Multimedia information
retrieval.
[15]G. Carneiro, A.B. Chan, P.J. Moreno, N. Vasconcelos, (2007) “Supervised learning of semantic classes
for image annotation and retrieval,” IEEE Trans. on PAMI, vol. 29, no. 3.
[16]Y. Mori, H. Takahashi, R. Oka, (1999) “Image-to-word transformation based on dividing and vector
quantizing images with words,” In Proceedings of First International Workshop Multimedia Intelligent
Storage and Retrieval Management.

30
[17]P. Duygulu, K. Barnard, J. Freitas, D. Forsyth, (2002) “Object recognition as machine translation:
learning a lexicon for a fixed image vocabulary,” In Proceedings of the 7th European Conference on
Computer Vision, vol. 2353, pp. 97-112.
[18]J. Jeon, V. Lavrenko, R. Manmatha, (2003) “Automatic image annotation and retrieval using Cross-
Media Relevance Model,” In Proceedings of the 26th annual international ACM SIGIR, pp. 119-126.
[19]V. Lavrenko, R. Manmatha, J. Jeon, (2003) “A model for learning the semantics of pictures,” In
Proceedings of Advance in Neutral Information Processing.
[20]S. Feng, R. Manmatha, V. Laverenko, (2004) “Multiple Bernoulli Relevance Models for image and
video annotation,” In IEEE Computer Society Conference on Computer Vision and Pattern
Recognition(CVPR), pp. 1002-1009.
[21]Y. Wang, T. Mei, Sh. Gong, X.Sh. Hua, (2009) “Combining global, regional and contextual features
for automatic image annotation,” Elsevier Ltd., Pattern Recognition, vol. 42, pp. 259-266.
[22]T. Hofmann, (2001) “Unsupervised learning by probabilistic latent semantic analysis,” Machine
Learning, vol. 42, pp. 177-196.
[23]Y. Deng, B.S. Manjunath, (2001) "Unsupervised Segmentation of Color-Texture Regions in Images
and Video," IEEE Trans. on PAMI, Vol. 23, No. 8, pp. 800-810.
[24]A.P. Dempster, N.M. Laird, D.B. Rubin, (2007) "Maximum Likelihood from Incomplete Data via the
EM Algorithm," Journal of the Royal Statistical Society, Vol. 39, No. 1, pp. 1-38.
[25]G. Csurka, Ch.R. Dance, L. Fan, J. Willamowski, C. Bray, (2004) "Visual categorization with bags of
keypoints," In Proceedings of ECCV Workshop on Statistical Learning in Computer Vision, pp. 1-16.
[26]K. Hotta, (2008) “Scene classification based on multi-resolution orientation histogram of Gabor
features,” In ICVS'08 Proceedings of the 6th international conference on Computer vision systems,
vol. 5008, pp. 291-301.
[27]K. Hotta, (2009) “Scene classification based on local autocorrelation of similarities with subspaces,” In
16th IEEE International Conference on Image Processing (ICIP), pp. 2053-2056.
[28]L. Fei Fei, P. Perona, (2005) “A Bayesian hierarchical model for learning natural scene categories,” In
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(CVPR'05), vol. 2, pp. 524-531.
[29]S. Lazebnik, C. Schmid, J. Ponce, (2006) "Beyond bags of features: Spatial pyramid matching for
recognizing natural scene categories," In Proceedings of IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pp. 2169-2178.
Authors
Hengame Deljooi received her B.Sc. in hardware computer engineering from Azad
University of Qazvin and the M.Sc. degree in computer architecture from Azad
University of Qazvin in 2008 and 2012 respectively, she is currently working toward
the Ph.D. degree in Computer architecture at I slamic Azad University, Science and
Research Branch. Her research intrests include Computer Architecture, Image
Processing and Image Retrieval.
Somayyeh Jafarali Jassbi received her B.Sc. in hardware computer engineering from
Azad University of Tehran, South Branch in 2003. She also received the M.Sc. and
Ph.D. degree in computer architecture from the Islamic Azad University, Science and
Research Branch in 2007 and 2010 respectively. She is c urrently assistant professor in
department of computer engineering of Science and Research Branch of Islamic Azad
University. Her research intrests are Information Technology, Computer Arithmetic,
Network Security and Cryptography.

A Multi Criteria Decision Making Based Approach for Semantic Image Annotation

Recommended

More Related Content

Similar to A Multi Criteria Decision Making Based Approach for Semantic Image Annotation (20)

More from ijcax (20)

Recently uploaded (20)

A Multi Criteria Decision Making Based Approach for Semantic Image Annotation