Data classification based on the local intrinsic dimension.

One of the founding paradigms of machine learning is that a small number of variables is often sufficient to describe high-dimensional data. The minimum number of variables required is called the intrinsic dimension (ID) of the data. Contrary tocommon intuition, there are cases where the ID varies within the same data set. This fact has been highlighted in technicaldiscussions, but seldom exploited to analyze large data sets and obtain insight into their structure. Here we develop a robustapproach to discriminate regions with different local IDs and classify the points accordingly. Our approach is computationallyefficient, and can be proficiently used even on large data sets. We find that many real-world data sets contain regions withwidely heterogeneous dimensions. These regions host points differing in core properties: folded vs unfolded configurations in aprotein molecular dynamics trajectory, active vs non-active regions in brain imaging data, and firms with different financial risk incompany balance sheets. A simple topological feature, the local ID, is thus sufficient to achieve an unsupervised classificationof high-dimensional data, complementary to the one given by clustering algorithms.