Everyone knows that there is currently a great deal of discussion about “Big Data”. It turns out, though, that complexity of the data is as much of a difficulty in making use of the data as is the sheer size. We have made enormous strides in handling size of data, to a point where the ability to handle size is a commodity. Complexity is a tougher nut to crack. In fact, relatively small data sets can often exhibit complexity making them difficult to analyze with traditional approaches. Although people understand intuitively that complexity is a real problem in data analysis, it is not an easy notion to define. In many cases, we “know it when we see it.” We have previously touched upon the methods for measuring shape in the post entitled Measuring Shape. What we will do in this post is to illustrate complexity through examples, and to draw conclusions about how one might deal with it. As an example of a simple data set, consider the data set below. Most of us would regard this as a simple data set, since we have seen many examples that look exactly like this, and because we are very familiar with the geometric primitive with which it can be approximated, a straight line. There are many algebraic methods that can deal with data like this, notably linear regression. It works very well when the data is well approximated by a line, plane, or other linear object. Unfortunately, not all data cooperates by fitting along a line or plane. Here is an example of another kind of data set that occurs frequently. In this case, the data does not fit along any particular line, and so linear regression methods will not work well. The data exhibits a certain kind of complexity, namely that it breaks into three distinct groups, but there is not a simple algebraic model that explains this data set. Statisticians have developed a great variety of methods for performing this kind of decomposition, under the heading of cluster analysis. If this was the only kind of complexity that occurs in data, we might decide to develop a toolkit consisting of (a) linear methods and (b) cluster analysis, and conclude that such a toolkit might solve all our data analysis problems. However, we would then be confronted with a data set that looks as follows. We note that it is not well approximated by a line, and does not naturally decompose into clusters. It is complex in its own particular way, namely that it exhibits loopy or circular behavior. This kind of behavior often occurs in data sets in which periodic or recurrent behavior is present, and it is therefore a valuable kind of feature to be able to understand and detect. Yet another kind of complexity is the one that occurs in the data set below. This is a distinct type that does not fit under any of preceding three types. The salient feature here is that the set has a central core, with three “flares” emerging from it. In some cases, the central core represents typical behaviors and the tips of the flares are extreme behaviors of some kind. Consider the performance of an aircraft engine for example. It demonstrates this type of behavior in the three phases of operation – takeoff, landing and drag during flight. For the first two cases above, there are mathematical pipelines, linear regression in the first case and cluster analysis in the second, which act as organizing principles for the data. Both of these can be viewed as approximation by a fixed shape, a line in the first case and a finite discrete set of points in the second case. Both methods are extremely useful. For the latter two cases, however, there are not explicit mathematical pipelines. One might imagine developing pipelines for each of these two types, but there will be other templates that arise which don’t fit under any of the four above. Below is the classic example of complex data. Complex data doesn’t have one or even two of these shapes – complex data has every one of those shapes – simultaneously. This data set, which delivered breakthrough insights into Type II diabetes is colored by glucose level. Those segments in red are distinct types of Type II diabetes. Note, however, the complexity of the data – it has loops, flares, clusters and if you looked within anyone of these segments would likely find some easily regressable data. Traditional techniques simply could not identify all of the features of this example of complex data. A better approach would be to develop a more generic organizing principle that can accommodate all possible types, rather than individual approaches that deal with specific types one by one. The Topological Data Analysis methodology takes the point of view that the shape of data is such an organizing principle, where one can assign to every data set, a topological network, which represents its shape. Once this is done, we find that we can operate on these networks in many very useful ways, many of which are discussed in some of our other blog posts. Everyone knows that there is currently a great deal of discussion about “Big Data”. It turns out, though, that complexity of the data is as much of a difficulty in making use of the data as is the sheer size. We have made enormous strides in handling size of data, to a point where the ability to handle size is a commodity. Complexity is a tougher nut to crack. In fact, relatively small data sets can often exhibit complexity making them difficult to analyze with traditional approaches. Although people understand intuitively that complexity is a real problem in data analysis, it is not an easy notion to define. In many cases, we “know it when we see it.” We have previously touched upon the methods for measuring shape in the post entitled Measuring Shape. What we will do in this post is to illustrate complexity through examples, and to draw conclusions about how one might deal with it. As an example of a simple data set, consider the data set below. Most of us would regard this as a simple data set, since we have seen many examples that look exactly like this, and because we are very familiar with the geometric primitive with which it can be approximated, a straight line. There are many algebraic methods that can deal with data like this, notably linear regression. It works very well when the data is well approximated by a line, plane, or other linear object. Unfortunately, not all data cooperates by fitting along a line or plane. Here is an example of another kind of data set that occurs frequently. In this case, the data does not fit along any particular line, and so linear regression methods will not work well. The data exhibits a certain kind of complexity, namely that it breaks into three distinct groups, but there is not a simple algebraic model that explains this data set. Statisticians have developed a great variety of methods for performing this kind of decomposition, under the heading of cluster analysis. If this was the only kind of complexity that occurs in data, we might decide to develop a toolkit consisting of (a) linear methods and (b) cluster analysis, and conclude that such a toolkit might solve all our data analysis problems. However, we would then be confronted with a data set that looks as follows. We note that it is not well approximated by a line, and does not naturally decompose into clusters. It is complex in its own particular way, namely that it exhibits loopy or circular behavior. This kind of behavior often occurs in data sets in which periodic or recurrent behavior is present, and it is therefore a valuable kind of feature to be able to understand and detect. Yet another kind of complexity is the one that occurs in the data set below. This is a distinct type that does not fit under any of preceding three types. The salient feature here is that the set has a central core, with three “flares” emerging from it. In some cases, the central core represents typical behaviors and the tips of the flares are extreme behaviors of some kind. Consider the performance of an aircraft engine for example. It demonstrates this type of behavior in the three phases of operation – takeoff, landing and drag during flight. For the first two cases above, there are mathematical pipelines, linear regression in the first case and cluster analysis in the second, which act as organizing principles for the data. Both of these can be viewed as approximation by a fixed shape, a line in the first case and a finite discrete set of points in the second case. Both methods are extremely useful. For the latter two cases, however, there are not explicit mathematical pipelines. One might imagine developing pipelines for each of these two types, but there will be other templates that arise which don’t fit under any of the four above. Below is the classic example of complex data. Complex data doesn’t have one or even two of these shapes – complex data has every one of those shapes – simultaneously. This data set, which delivered breakthrough insights into Type II diabetes is colored by glucose level. Those segments in red are distinct types of Type II diabetes. Note, however, the complexity of the data – it has loops, flares, clusters and if you looked within anyone of these segments would likely find some easily regressable data. Traditional techniques simply could not identify all of the features of this example of complex data. A better approach would be to develop a more generic organizing principle that can accommodate all possible types, rather than individual approaches that deal with specific types one by one. The Topological Data Analysis methodology takes the point of view that the shape of data is such an organizing principle, where one can assign to every data set, a topological network, which represents its shape. Once this is done, we find that we can operate on these networks in many very useful ways, many of which are discussed in some of our other blog posts.