Relation between the Kantorovich-Wasserstein metric and the Kullback-Leibler divergence

Conference paper


Belavkin, R. 2018. Relation between the Kantorovich-Wasserstein metric and the Kullback-Leibler divergence. IGAIA IV 2016: Information Geometry and Its Applications. Liblice, Czech Republic 12 - 17 Jun 2016 Springer International Publishing. https://doi.org/10.1007/978-3-319-97798-0_15
TypeConference paper
TitleRelation between the Kantorovich-Wasserstein metric and the Kullback-Leibler divergence
AuthorsBelavkin, R.
Abstract

We discuss a relation between the Kantorovich-Wasserstein (KW) metric and the Kullback-Leibler (KL) divergence. The former is defined using the optimal transport problem (OTP) in the Kantorovich formulation. The latter is used to define entropy and mutual information, which appear in variational problems to find optimal channel (OCP) from the rate distortion and the value of information theories. We show that OTP is equivalent to OCP with one additional constraint fixing the output measure, and therefore OCP with constraints on the KL-divergence gives a lower bound on the KW-metric. The dual formulation of OTP allows us to explore the relation between the KL-divergence and the KW-metric using decomposition of the former based on the law of cosines. This way we show the link between two divergences using the variational and geometric principles.

Research GroupArtificial Intelligence group
ConferenceIGAIA IV 2016: Information Geometry and Its Applications
ISSN2194-1009
ISBN
Hardcover9783319977973
PublisherSpringer International Publishing
Publication dates
Online04 Nov 2018
Print01 Jan 2018
Publication process dates
Deposited11 Jan 2019
Accepted15 Apr 2016
Output statusPublished
Accepted author manuscript
Copyright Statement

The final authenticated version is available online at https://doi.org/10.1007/978-3-319-97798-0_15

Additional information

Paper published as:
Belavkin R.V. (2018) Relation Between the Kantorovich–Wasserstein Metric and the Kullback–Leibler Divergence. In: Ay N., Gibilisco P., Matúš F. (eds) Information Geometry and Its Applications. IGAIA IV 2016. Springer Proceedings in Mathematics & Statistics, vol 252. Springer, Cham

Digital Object Identifier (DOI)https://doi.org/10.1007/978-3-319-97798-0_15
LanguageEnglish
Book titleInformation Geometry and Its Applications
Permalink -

https://repository.mdx.ac.uk/item/881wy

Download files


Accepted author manuscript
  • 60
    total views
  • 22
    total downloads
  • 0
    views this month
  • 0
    downloads this month

Export as

Related outputs

An evolutionary approach to automated class-specific data augmentation for image classification
Marc, S., Belavkin, R., Windridge, D. and Gao, X. 2024. An evolutionary approach to automated class-specific data augmentation for image classification. Moosaei, H., Hladík, M. and Pardalos, P. (ed.) 6th International Conference on the Dynamics of Information Systems. Prague, Czech Republic 03 - 06 Dec 2023 Springer. pp. 170–185 https://doi.org/10.1007/978-3-031-50320-7_12
Value of information in the mean-square case and its application to the analysis of financial time-series forecast
Belavkin, R., Pardalos, P. and Principe, J. 2023. Value of information in the mean-square case and its application to the analysis of financial time-series forecast. Simos, D., Rasskazova, V., Archetti, F., Kotsireas, I. and Pardalos, P. (ed.) 16th International Conference on Learning and Intelligent Optimization. Milos Island, Greece 05 - 10 Jun 2022 Cham Springer. https://doi.org/10.1007/978-3-031-24866-5_39
Value of information in the binary case and confusion matrix
Belavkin, R., Pardalos, P. and Principe, J. 2022. Value of information in the binary case and confusion matrix. 41st International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering. Paris, France 18 - 22 Jul 2022 MDPI AG. pp. 1-9 https://doi.org/10.3390/psf2022005008
Environmental pleiotropy and demographic history direct adaptation under antibiotic selection
Gifford, D., Krašovec, R., Aston, E., Belavkin, R., Channon, A. and Knight, C. 2018. Environmental pleiotropy and demographic history direct adaptation under antibiotic selection. Heredity. 121 (5), pp. 438-448. https://doi.org/10.1038/s41437-018-0137-3
Opposing effects of final population density and stress on Escherichia coli mutation rate
Krašovec, R., Richards, H., Gifford, D., Belavkin, R., Channon, A., Aston, E., McBain, A. and Knight, C. 2018. Opposing effects of final population density and stress on Escherichia coli mutation rate. The ISME journal. 12 (12), pp. 2981-2987. https://doi.org/10.1038/s41396-018-0237-3
Critical mutation rate has an exponential dependence on population size for eukaryotic-length genomes with crossover
Aston, E., Channon, A., Belavkin, R., Gifford, D., Krašovec, R. and Knight, C. 2017. Critical mutation rate has an exponential dependence on population size for eukaryotic-length genomes with crossover. Scientific Reports. 7 (1). https://doi.org/10.1038/s41598-017-14628-x
Spontaneous mutation rate is a plastic trait associated with population density across domains of life
Krašovec, R., Richards, H., Gifford, D., Hatcher, C., Faulkner, K., Belavkin, R., Channon, A., Aston, E., McBain, A. and Knight, C. 2017. Spontaneous mutation rate is a plastic trait associated with population density across domains of life. PLOS Biology. 15 (8). https://doi.org/10.1371/journal.pbio.2002731
Monotonicity of fitness landscapes and mutation rate control
Belavkin, R., Channon, A., Aston, E., Aston, J., Krašovec, R. and Knight, C. 2016. Monotonicity of fitness landscapes and mutation rate control. Journal of Mathematical Biology. 73 (6-7), pp. 1491-1524. https://doi.org/10.1007/s00285-016-0995-3
Asymmetric topologies on statistical manifolds
Belavkin, R. 2015. Asymmetric topologies on statistical manifolds. 2nd International Conference on Geometric Science of Information. Palaiseau, France 28 - 30 Oct 2015 Springer. https://doi.org/10.1007/978-3-319-25040-3_23
Local semantic indexing for resource discovery on overlay network using mobile agents
Singh, M., Cheng, X. and Belavkin, R. 2014. Local semantic indexing for resource discovery on overlay network using mobile agents. International Journal of Computational Intelligence Systems. 7 (3), pp. 432-455. https://doi.org/10.1080/18756891.2013.856257
Where antibiotic resistance mutations meet quorum-sensing
Krašovec, R., Belavkin, R., Aston, J., Channon, A., Aston, E., Rash, B., Kadirvel, M., Forbes, S. and Knight, C. 2014. Where antibiotic resistance mutations meet quorum-sensing. Microbial Cell. 1 (7), pp. 250-252. https://doi.org/10.15698/mic2014.07.158
On variational definition of quantum entropy
Belavkin, R. 2014. On variational definition of quantum entropy. 34th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering (MaxEnt 2014). Clos Lucé, Amboise, France 21 - 26 Sep 2014 American Institute of Physics. https://doi.org/10.1063/1.4905979
Asymmetry of risk and value of information
Belavkin, R. 2014. Asymmetry of risk and value of information. in: Vogiatzis, C., Walteros, J. and Pardalos, P. (ed.) Dynamics of Information Systems: Computational and Mathematical Challenges Springer.
Mutation rate plasticity in rifampicin resistance depends on Escherichia coli cell–cell interactions
Krašovec, R., Belavkin, R., Aston, J., Channon, A., Aston, E., Rash, B., Kadirvel, M., Forbes, S. and Knight, C. 2014. Mutation rate plasticity in rifampicin resistance depends on Escherichia coli cell–cell interactions. Nature Communications. 5, pp. 1-8. https://doi.org/10.1038/ncomms4742
Minimum of information distance criterion for optimal control of mutation rate in evolutionary systems
Belavkin, R. 2013. Minimum of information distance criterion for optimal control of mutation rate in evolutionary systems. in: Accardi, L., Freudenberg, W. and Ohya, M. (ed.) Quantum Bio-Informatics V : Proceedings of the Quantum Bio-Informatics 2011. Tokyo University of Science, Japan, 7 – 12 March 2011 World Scientific.
Law of cosines and Shannon-Pythagorean theorem for quantum information
Belavkin, R. 2013. Law of cosines and Shannon-Pythagorean theorem for quantum information. in: Nielsen, F. and Barbaresco, F. (ed.) Geometric Science of Information : First International Conference, GSI 2013, Paris, France, August 28-30, 2013. Proceedings Berlin Springer.
Optimal measures and Markov transition kernels
Belavkin, R. 2013. Optimal measures and Markov transition kernels. Journal of Global Optimization. 55 (2), pp. 387-416. https://doi.org/10.1007/s10898-012-9851-1
Critical mutation rate has an exponential dependence on population size
Channon, A., Aston, E., Day, C., Belavkin, R. and Knight, C. 2011. Critical mutation rate has an exponential dependence on population size. Lenaerts, T., Giacobini, M., Bersini, H., Bourgine, P., Dorigo, M. and Doursat, R. (ed.) ECAL 2011: The 11th European Conference on Artificial Life. Paris, France 08 - 12 Aug 2011 The MIT Press. pp. 117-124 https://doi.org/10.7551/978-0-262-29714-1-ch021
Dynamics of information and optimal control of mutation in evolutionary systems
Belavkin, R. 2012. Dynamics of information and optimal control of mutation in evolutionary systems. in: Sorokin, A., Murphey, R., Thai, M. and Pardalos, P. (ed.) Dynamics of Information Systems: Mathematical Foundations New York Springer.
Maximal connectivity and constraints in the human brain
Belavkin, R. 2012. Maximal connectivity and constraints in the human brain. in: Pardalos, P., Coleman, T. and Xanthopoulos, P. (ed.) Optimization and data analysis in biomedical informatics New York Springer.
On evolution of an information dynamic system and its generating operator
Belavkin, R. 2012. On evolution of an information dynamic system and its generating operator. Optimization Letters. 6 (5), pp. 827-840. https://doi.org/10.1007/s11590-011-0325-z
Theory and practice of optimal mutation rate control in Hamming spaces of DNA sequences
Belavkin, R., Channon, A., Aston, E., Aston, J. and Knight, C. 2011. Theory and practice of optimal mutation rate control in Hamming spaces of DNA sequences. Lenaerts, T., Giacobini, M., Bersini, H., Bourgine, P., Dorigo, M. and Doursat, R. (ed.) ECAL 2011: The 11th European Conference on Artificial Life. Paris, France 08 - 12 Aug 2011 The MIT Press. pp. 85-92 https://doi.org/10.7551/978-0-262-29714-1-ch017
Mutation and optimal search of sequences in nested Hamming spaces
Belavkin, R. 2011. Mutation and optimal search of sequences in nested Hamming spaces. IEEE Information Theory Workshop (ITW). Paraty, Brazil 16 - 20 Oct 2011 IEEE. pp. 90-94 https://doi.org/10.1109/ITW.2011.6089592
Conflict resolution and learning probability matching in a neural cell-assembly architecture
Belavkin, R. and Huyck, C. 2011. Conflict resolution and learning probability matching in a neural cell-assembly architecture. Cognitive Systems Research. 12 (2), pp. 93-101. https://doi.org/10.1016/j.cogsys.2010.08.003
Towards a theory of decision-making without paradoxes.
Belavkin, R. 2006. Towards a theory of decision-making without paradoxes. Fum, D., Missier, F. and Stocco, A. (ed.) Proceedings of the Seventh International Conference on Cognitive Modeling. Trieste, Italy 05 - 08 Apr 2006 Trieste, Italy Edizioni Goliardiche. pp. 38-43
Emergence of rules in cell assemblies of fLIF neurons.
Belavkin, R. and Huyck, C. 2008. Emergence of rules in cell assemblies of fLIF neurons. The 18th European Conference on Artificial Intelligence. University of Patras, Greece 21 - 25 Jul 2008
A model of probability matching in a two-choice task based on stochastic control of learning in neural cell-assemblies.
Belavkin, R. and Huyck, C. 2009. A model of probability matching in a two-choice task based on stochastic control of learning in neural cell-assemblies. 9th International conference on cognitive modelling {ICCM 2009]. University of Manchester 24 - 26 Jul 2009
Resource discovery using mobile agents
Singh, M., Cheng, X. and Belavkin, R. 2010. Resource discovery using mobile agents. Frontier of Computer Science and Technology (FCST), 2010 Fifth International Conference. Changchun, Jilin Province 18 - 22 Aug 2010 IEEE. pp. 72 -77 https://doi.org/10.1109/FCST.2010.93
Utility and value of information in cognitive science, biology and quantum theory.
Belavkin, R. 2010. Utility and value of information in cognitive science, biology and quantum theory. Accardi, L., Freudenberg, W. and Ohya, M. (ed.) Quantum Bio-Informatics III. Tokyo, Japan 11 - 14 Mar 2009 London World Scientific. pp. 33-47 https://doi.org/10.1142/9789814304061_0004
Information trajectory of optimal learning
Belavkin, R. 2010. Information trajectory of optimal learning. in: Hirsch, M., Pardalos, P. and Murphey, R. (ed.) Dynamics of Information Systems: Theory and Applications Springer.
Learning behaviour patterns of classroom and distance students using flexible learning resources.
Dimitrova, M., Belavkin, R., Milankovic-Atkinson, M., Sadler, C. and Murphy, A. 2003. Learning behaviour patterns of classroom and distance students using flexible learning resources. in: Lee, K. and Mitchell, K. (ed.) International conference on computers in education 2003: a conference of the Asia-Pacific chapter of the association for the advancement of computing in education (AACE). Hong Kong ICCE.
Bounds of optimal learning
Belavkin, R. 2009. Bounds of optimal learning. 2009 IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning. Nashville, TN, USA 30 Mar - 02 Apr 2009 IEEE. pp. 199-204 https://doi.org/10.1109/ADPRL.2009.4927545
Counting with neurons: rule application with nets of fatiguing leaking integrate and fire neurons.
Huyck, C. and Belavkin, R. 2006. Counting with neurons: rule application with nets of fatiguing leaking integrate and fire neurons. 7th International Conference on Cognitive Modelling. Trieste, Italy pp. 142-147
The use of entropy for analysis and control of cognitive models
Belavkin, R. and Ritter, F. 2003. The use of entropy for analysis and control of cognitive models. The Fifth International Conference on Cognitive Modelling. Bamberg, Germany 2003 pp. 21-26
Conflict resolution by random estimated costs
Belavkin, R. 2003. Conflict resolution by random estimated costs. 17th European Simulation Multiconference. Nottingham UK pp. 105-110
On relation between emotion and entropy
Belavkin, R. 2004. On relation between emotion and entropy. AISB'04 Symposium on Emotion, Cognition and Affective Computing. Leeds UK pp. 1-8
OPTIMIST: A new conflict resolution algorithm for ACT-R.
Belavkin, R. and Ritter, F. 2004. OPTIMIST: A new conflict resolution algorithm for ACT-R. Sixth International Conference on Cognitive Modelling. Mahwah, NJ pp. 40-45
Entropy and information in models of learning behaviour
Belavkin, R. 2005. Entropy and information in models of learning behaviour. AISB Quarterly. 119, pp. 5-5.
Towards a theory of decision-making with paradoxes.
Belavkin, R. 2006. Towards a theory of decision-making with paradoxes. Proceedings of the Seventh International Conference on Cognitive Modelling. Trieste, Italy 2006 pp. 38-43
The duality of utility and information in optimally learning systems
Belavkin, R. 2008. The duality of utility and information in optimally learning systems. 7th IEEE International Conference on Cybernetic Intelligent Systems. London, UK 09 - 10 Sep 2008 London IEEE. https://doi.org/10.1109/UKRICIS.2008.4798960
Do neural models scale up to a human brain?
Belavkin, R. 2007. Do neural models scale up to a human brain? International Joint Conference on Neural Networks (IJCNN 2007). Orlando, Florida 12 - 17 Aug 2007 IEEE. pp. 2312-2317 https://doi.org/10.1109/IJCNN.2007.4371319
Acting irrationally to improve performance in stochastic worlds
Belavkin, R. 2005. Acting irrationally to improve performance in stochastic worlds. Bramer, M., Coenen, F. and Allen, T. (ed.) 25th SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence. Cambridge, UK 2005 Springer. pp. 305-316 https://doi.org/10.1007/978-1-84628-226-3_23