State-of-the-art word embedding methods represent a word with a single vector and presume a linear vector space, which does not easily incorporate nonlinearity that is necessary to, for example, represent polysemeous words (those having multiple meanings). We study alternative mathematical representation of language, such as the use of nonlinear functions and fields. We have proposed one formulation called FIRE (Reference) which beats BERT in an evaluation of counting the number of word senses.

For mathematical models of language, their potential, limitations, and ways of improvement are investigated in terms of whether they reproduce the complex properties of language. The nature of linguistic structure is studied in terms of its relation to scaling properties of language. (Reference) (Reference)

A generative model is a mathematical formulation that generates a sample similar to real data. Many such models have been proposed using machine learning methods, including deep learning. Study of a good model serves to characterize the nature of a system and also to clarify the potential of machine learning. We study various time series models including classical Markov models, grammatical models, Simon processes, random walks on a network, neural models, autoencoders, and adversarial methods. The fundamental properties of generative models are studied in terms of whether they can generate samples resembling real data. (Reference)

Templates are multi-word expressions with slots, such as "Starting at __ on __ " or "regard _ as _", that appear frequently in text and also in data from sources such as Twitter. Automatic extraction of these template expressions is a challenging problem that is related to grammar inference. We propose automatic template extraction by using a binary decision diagram (BDD), which is mathematically equivalent to a minimal deterministic finite-state automaton (DFA). We have studied a basic formulation and currently seek a larger application to extract patterns from social networking service (SNS) data through additional use of deep learning methods. (Reference and its link to arxiv)

Various metrics are considered in terms of whether they characterize different kinds of data. For example, in the case of natural language, metrics that specify the author, language, or genre have been studied. One such metric is Yule's K, which is equivalent to Renyi's second-order (plug-in) entropy. Yule's K computes a value that does not depend on the data size but only on the data kind. We explore such metrics among various statistics related to scaling properties of real data and compare different kinds of data such as music, programming language sources, and natural language. (Reference)

How grammatically complex are adults' utterances as compared with those of children? How is a literary text more structurally complex than a Wikipedia source? How can such complexity be compared with that of a music performance or a programming languages source? In the linguistic domain, one existing, formal way to consider such questions is through the Chomsky hierarchy, which formulates different complexity levels of grammar through constraints put on rewriting rules. While the hierarchy provides qualitative categorization, it cannot serve for quantitatively comparing the structural complexity of time series. We investigate a new way to quantify structural complexity by using metrics based on scaling properties. (Reference)

Real instances of social systems have a bursty character, meaning that events occur in a clustered manner. For example, the figure on the right shows how rare events occur over time in texts (the first indicates rarer events than the second; the second, rarer than the third). This clustering phenomenon indicates how a sequence has long memory and thus exhibits self-similarity. We study methods for nonnumerical time series to quantify the degree of clustering and examine different self-similarity degrees across various systems. (Reference)

The theory of econophysics reveals the scaling properties of price, which explains why market crashes much more frequently than expected. A challenge of financial market modeling is to characterize the risks implied by extreme events, such as the COVID-19 crisis, which are rare and unable to well capture from the limited history of price data. News texts, on the other hand, are biased to such rare events and can thus be a desirable information source in addition to price history. We have studied the use of textual data for quantifying such extreme risks and its potential application. (Reference)

The bitcoin price crash at the beginning of 2018 was caused by various social factors. The influence of news wire stories and social media was especially crucial because of the combination of both credible and fake information being mixed and expanded on social media. We accumulate financial data including various stock and bitcoin prices and analyze the influence of the communication structure and textual data. (Reference)

We explore the complexity underlying human symbolic sequences via entropy rate estimation. Consider the number of possibilities for a time series of length n, with a parameter h, as 2^hn. For a random binary series consisting of half ones and half zeros, h=1. For the 26 characters in English, however, the number of possibilities is not 26ⁿ, because of various constraints such as "q" being followed only by "u". Shannon computed a value of h=1.3, but the question of acquiring a true h for human language is difficult to answer and remains unsolved: in fact, it is unknown whether h is even positive. Therefore, we study ways to compute the upper bound of h for various kinds of data, including music, programs, and market data, in addition to natural language. (Reference, Reference)