A Flexible Generative AI Framework for High-dimensional Data Analysis

Abstract

Analyzing high-dimensional data has been hindered by the challenges posed by the "curse of dimensionality." In response, we introduce a versatile computational framework known as Encoding Generative Modeling (EGM). EGM is designed to address fundamental problems in statistics and machine learning, such as density estimation, clustering and causal inference. The core concept of EGM involves the learning of a pair of functions facilitating bidirectional transformations between a low-dimensional latent space and the high-dimensional observation data space. Harnessing the capabilities of generative AI, we are able to effectively learn these transformation functions. One distinguishing feature of EGM lies in its adaptability. By introducing different structure or geometry to the latent space of the EGM framework, we demonstrate its efficacy in modeling a diverse set of statistical and machine learning problems. Our numeric results consistently showcase the superior performance of EGM across a variety of statistical and machine learning tasks. Besides, we also provide the theoretical foundations of the EGM framework under different settings. To summarize, the EGM framework emerges as a highly flexible tool for the analysis of high-dimensional data. It offers a novel perspective for the development of deep learning-based estimates, providing innovative solutions to a wide range of statistical and machine learning problems.