Multivariate mutual information
In information theory there have been various attempts over the years to extend the definition of mutual information to more than two random variables. The expression and study of multivariate higher-degree mutual-information was achieved in two seemingly independent works: McGill who called these functions “interaction information”, and Hu Kuo Ting who also first proved the possible negativity of mutual-information for degrees higher than 2 and justified algebraically the intuitive correspondence to Venn diagrams .
Definition
The conditional mutual information can be used to inductively define a multivariate mutual information in a set- or measure-theoretic sense in the context of information diagrams. In this sense we define the multivariate mutual information as follows:where
This definition is identical to that of interaction information except for a change in sign in the case of an odd number of random variables.
Alternatively, the multivariate mutual information may be defined in measure-theoretic terms as the intersection of the individual entropies :
Defining, the set-theoretic identity which corresponds to the measure-theoretic statement, allows the above to be rewritten as:
which is identical to the first definition.
Properties
Multi-variate information and conditional multi-variate information can be decomposed into a sum of entropies.Multivariate statistical independence
The multivariate mutual-information functions generalize the pairwise independence case that states that if and only if, to arbitrary numerous variable. n variables are mutually independent if and only if the mutual information functions vanish with . In this sense, the can be used as a refined statistical independence criterion.Synergy and redundancy
The multivariate mutual information may be positive, negative or zero. The positivity corresponds to relations generalizing the pairwise correlations, nullity corresponds to a refined notion of independence, and negativity detects high dimensional "emergent" relations and clusterized datapoints ). For the simplest case of three variables X, Y, and Z, knowing, say, X yields a certain amount of information about Z. This information is just the mutual information . Likewise, knowing Y will also yield a certain amount of information about Z, that being the mutual information . The amount of information about Z which is yielded by knowing both X and Y together is the information that is mutual to Z and the X,Y pair, written and it may be greater than, equal to, or less than the sum of the two mutual information, this difference being the multivariate mutual information:. In the case where the sum of the two mutual information is greater than, the multivariate mutual information will be positive. In this case, some of the information about Z provided by knowing X is also provided by knowing Y, causing their sum to be greater than the information about Z from knowing both together. That is to say, there is a "redundancy" in the information about Z provided by the X and Y variables. In the case where the sum of the mutual information is less than, the multivariate mutual information will be negative. In this case, knowing both X and Y together provides more information about Z than the sum of the information yielded by knowing either one alone. That is to say, there is a "synergy" in the information about Z provided by the X and Y variables. The above explanation is intended to give an intuitive understanding of the multivariate mutual information, but it obscures the fact that it does not depend upon which variable is the subject and which other two are being thought of as the source of information. For 3 variables, Brenner et al. applied multivariate mutual information to neural coding and called its negativity "synergy" and Watkinson et al. applied it to genetic expressionExample of positive multivariate mutual information (redundancy)
Positive MMI is typical of common-cause structures. For example, clouds cause rain and also block the sun; therefore, the correlation between rain and darkness is partly accounted for by the presence of clouds,. The result is positive MMI.Examples of negative multivariate mutual information (synergy)
The case of negative MMI is infamously non-intuitive. A prototypical example of negative has as the output of an XOR gate to which and are the independent random inputs. In this case will be zero, but will be positive since once output is known, the value on input completely determines the value on input. Since, the result is negative MMI. It may seem that this example relies on a peculiar ordering of to obtain the positive interaction, but the symmetry of the definition for indicates that the same positive interaction information results regardless of which variable we consider as the interloper or conditioning variable. For example, input and output are also independent until input is fixed, at which time they are totally dependent.This situation is an instance where fixing the common effect of causes and induces a dependency among the causes that did not formerly exist. This behavior is colloquially referred to as explaining away and is thoroughly discussed in the Bayesian Network literature. Pearl's example is auto diagnostics: A car's engine can fail to start due either to a dead battery or due to a blocked fuel pump. Ordinarily, we assume that battery death and fuel pump blockage are independent events, because of the essential modularity of such automotive systems. Thus, in the absence of other information, knowing whether or not the battery is dead gives us no information about whether or not the fuel pump is blocked. However, if we happen to know that the car fails to start, this information induces a dependency between the two causes battery death and fuel blockage. Thus, knowing that the car fails to start, if an inspection shows the battery to be in good health, we conclude the fuel pump is blocked.
Battery death and fuel blockage are thus dependent, conditional on their common effect car starting. The obvious directionality in the common-effect graph belies a deep informational symmetry: If conditioning on a common effect
increases the dependency between its two parent causes, then conditioning on one of the causes must create the same increase in dependency between the second cause and the common effect. In Pearl's automotive example, if conditioning on car starts induces bits of dependency between the two causes battery dead and fuel blocked, then conditioning on
fuel blocked must induce bits of dependency between battery dead and car starts. This may seem odd because battery dead and car starts are governed by the implication battery dead car doesn't start. However, these variables are still not totally correlated because the converse is not true. Conditioning on fuel blocked removes the major alternate cause of failure to start, and strengthens the converse relation and therefore the association between battery dead and car starts.