Application of multimodal approach for identifying similarities in multi-dimensional datasets with usage example

Olga Perl, Ivan Perl


Increasingly, one can find research from various fields devoted to working with different formats of data, tasks, research perspectives or views on the object of research. Often these studies use the term "multimodal", which at the same time varies greatly from region to region. Thus, researchers from different subject areas are faced with the same task: choosing an approach and processing multimodal data. The problem with this task lies in the narrow area of applicability of the proposed solutions. Then the very concept of multimodality needs formalization. At the same time, it becomes necessary to allocate data structures to work with this concept. It is also important to describe the approaches for analyzing data in the selected data structures. This article proposes a definition of the concept of multimodal data, describes 4 structures for working with it, and also proposes the method for identifying the most similar multimodal objects. In addition, the given structures are illustrated by examples. The method for finding similar multimodal objects is supplemented with modification descriptions so that it can be applied to all 4 multimodal data structures. The article also demonstrates the application of the method on a general example with a description of data structures - the study of cities according to the characteristics of the population, climate and the number of universities. The example of the study is for illustrative purposes only, however, it can be used for further research after verification by appropriate specialists. The article provides methods for configuring the method and recommendations for working with them. Calculated object similarity (coherence power) is a way to define a multidimensional metric over a complex data structure. At the end of the article, directions for further research are given, which are already being carried out by the authors at the present time.

Full Text:

PDF (Russian)


L. A. Nguyen, "Multimodal logic programming," Theoretical Computer Science, vol. 360, no. 1, pp. 247-288, 2006.

R. Dockins, A. W. Appel and A. Hobor, "Multimodal Separation Logic for Reasoning About Operational Semantics," in Proceedings of the 24th Conference on the Mathematical Foundations of Programming Semantics (MFPS XXIV), 2008.

B. Karimi and M. Bashiri, "Designing a Multi-commodity multimodal splittable supply chain network by logistic hubs for intelligent manufacturing," in Procedia Manufacturing, Columbus, OH, 2018.

K. Atchaneeyasakul, D. S. Liebeskind, R. Jahan, S. Starkman, L. Sharma, B. Yoo, J. Avelar, N. Rao, J. Hinman, G. Duckwiler, M. Nour, V. Szeder, S. Tateshima, G. Colby, M. B. Hosseini, R. Raychev, D. Kim and J. L. Saver, "Efficient Multimodal MRI Evaluation for Endovascular Thrombectomy of Anterior Circulation Large Vessel Occlusion," Journal of Stroke and Cerebrovascular Diseases, vol. 29, no. 12, p. 105271, 2020.

J. D. Tward, T. Schlomm, S. Bardot, D. J. Canter, T. Scroggins, S. J. Freedland, L. Lenz, D. D. Flake, T. Cohen, M. K. Brawer, S. Stone and J. Bishoff, "{Personalizing Localized Prostate Cancer: Validation of a Combined Clinical Cell-cycle Risk (CCR) Score Threshold for Prognosticating Benefit From Multimodality Therapy," Clinical Genitourinary Cancer, 2021.

J.-E. Peng, "The roles of multimodal pedagogic effects and classroom environment in willingness to communicate in English," System, vol. 82, pp. 161-173, 2019.

S. Philippe, A. D. Souchet, P. Lameras, P. Petridis, J. Caporal, G. Coldeboeuf and H. Duzan, "Multimodal teaching, learning and training in virtual reality: a review and case study," Virtual Reality & Intelligent Hardware, vol. 2, no. 5, pp. 421-442, 2020.

L. C. O. Tiong, S. T. Kim and Y. M. Ro, "Multimodal facial biometrics recognition: Dual-stream convolutional neural networks with multi-feature fusion layers," Image and Vision Computing, vol. 102, p. 103977, 2020.

W. Zhang, J. Yu, W. Zhao and C. Ran, "DMRFNet: Deep Multimodal Reasoning and Fusion for Visual Question Answering and explanation generation," Information Fusion, vol. 72, pp. 70-79, 2021.

W. Zhang, J. Yu, Y. Wang and W. Wang, "Multimodal deep fusion for image question answering," Knowledge-Based Systems, vol. 212, p. 106639, 2021.

O. Kalyonova and I. Perl, "Revealing of entities interconnections in system dynamics modelling process by applying multimodal data analysis paradigm," in 21st Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 2017.

G. Zhu, J. Wang, Z. Ren, Z. Lin and C. Zhang, "Object-Oriented Dynamics Learning through Multi-Level Abstraction," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 6989-6998, 04 2020.

K. Wenzel and H. Reinhardt, "Mathematical computations for linked data applications with openmath," in CEUR Workshop Proceedings, 2012.

J. Guo, J. Xu, Z. He and W. Liao, "Research on risk propagation method of multimodal transport network under uncertainty," Physica A: Statistical Mechanics and its Applications, vol. 563, p. 125494, 2021.

Spisok gorodov Rossii s naseleniem bolee 100 tysjach zhitelej, Vikipedija, 2021., Vuzy po gorodam Rossii, 2021.


  • There are currently no refbacks.

Abava  Absolutech Convergent 2020

ISSN: 2307-8162