A multilevel analysis of characters, consonant-clusters and syllables in Konkani, Hindi, Marathi, and Nepali using Jaccard, Cosine, and hybrid similarity measures

Abstract

The character set of a language in a writing system is based on the language and script. Devanagari is one of the scripts used by 10 out of 22 officially recognized Indian languages. Devanagari consists of various characters, and not all the characters are used by all languages. For instance, the character ळ (/ɭə/) is quite common in Konkani and Marathi, whereas it does not feature in Hindi. This highlights a specific feature of the writing system, with unique language and script combination. The study investigates the extent to which writing systems overlap and diverge at multiple representational levels: characters, consonant-clusters and syllables. This study also provides comparison and contrast between languages namely Konkani, Hindi, Marathi, and Nepali based on their characters, consonant-clusters and syllables in the written text. The comparison is performed using three similarity measures: binary Jaccard similarity which shows the overlap of characters, consonant-clusters and syllables; Cosine similarity captures the frequency based overlap; and Hybrid similarity combines both similarity measures. The results show the highest similarity score between Konkani and Marathi, with binary Jaccard, Cosine, and Hybrid similarity scores of 0.40, 0.97, and 0.69, respectively. The results also show that similarity between two languages is highest at the character level, followed by syllable and consonant-cluster level. The decreasing similarity across these levels indicates grapholinguistic differentiation as graphic structures become increasingly complex and integrated. These findings provide important insights into how languages belonging to the same language family (Indo-Aryan) and using the same script can differ in their usage of characters, consonant-clusters, and syllables. This study should guide in adapting rich resource languages (like Hindi and Marathi) tools for low-resource languages like Konkani.