Statistics

linear_mixed_model(y, x[, z_t, k, p_path, …]) Initialize a linear mixed model from a matrix table.
linear_mixed_regression_rows(entry_expr, model) For each row, test an input variable for association using a linear mixed model.
linear_regression_rows(y, x, covariates[, …]) For each row, test an input variable for association with response variables using linear regression.
logistic_regression_rows(test, y, x, covariates) For each row, test an input variable for association with a binary response variable using logistic regression.
poisson_regression_rows(test, y, x, covariates) For each row, test an input variable for association with a count response variable using Poisson regression.
pca(entry_expr[, k, compute_loadings]) Run principal component analysis (PCA) on numeric columns derived from a matrix table.
row_correlation(entry_expr[, block_size]) Computes the correlation matrix between row vectors.
hail.methods.linear_mixed_model(y, x, z_t=None, k=None, p_path=None, overwrite=False, standardize=True, mean_impute=True)[source]

Initialize a linear mixed model from a matrix table.

Examples

Initialize a model using three fixed effects (including intercept) and genetic marker random effects:

>>> marker_ds = dataset.filter_rows(dataset.use_as_marker)
>>> model, _ = hl.linear_mixed_model(
...     y=marker_ds.pheno.height,
...     x=[1, marker_ds.pheno.age, marker_ds.pheno.is_female],
...     z_t=marker_ds.GT.n_alt_alleles(),
...     p_path='output/p.bm')

Fit the model and examine \(h^2\):

>>> model.fit()
>>> model.h_sq

Sanity-check the normalized likelihood of \(h^2\) over the percentile grid:

>>> import matplotlib.pyplot as plt                     
>>> plt.plot(range(101), model.h_sq_normalized_lkhd())  

For this value of \(h^2\), test each variant for association:

>>> result_table = hl.linear_mixed_regression_rows(dataset.GT.n_alt_alleles(), model)

Alternatively, one can define a full-rank model using a pre-computed kinship matrix \(K\) in ndarray form. When \(K\) is the realized relationship matrix defined by the genetic markers, we obtain the same model as above with \(P\) written as a block matrix but returned as an ndarray:

>>> rrm = hl.realized_relationship_matrix(marker_ds.GT).to_numpy()
>>> model, p = hl.linear_mixed_model(
...     y=dataset.pheno.height,
...     x=[1, dataset.pheno.age, dataset.pheno.is_female],
...     k=rrm,
...     p_path='output/p.bm',
...     overwrite=True)

Notes

See LinearMixedModel for details on the model and notation.

Exactly one of z_t and k must be set.

If z_t is set, the model is low-rank if the number of samples \(n\) exceeds the number of random effects \(m\). At least one dimension must be less than or equal to 46300. If standardize is true, each random effect is first standardized to have mean 0 and variance \(\frac{1}{m}\), so that the diagonal values of the kinship matrix K = ZZ^T are 1.0 in expectation. This kinship matrix corresponds to the realized_relationship_matrix() in genetics. See LinearMixedModel.from_random_effects() and BlockMatrix.svd() for more details.

If k is set, the model is full-rank. For correct results, the indices of k must be aligned with columns of the source of y. Set p_path if you plan to use the model in linear_mixed_regression_rows(). k must be positive semi-definite; symmetry is not checked as only the lower triangle is used. See LinearMixedModel.from_kinship() for more details.

Missing, nan, or infinite values in y or x will raise an error. If set, z_t may only have missing values if mean_impute is true, in which case missing values of are set to the row mean. We recommend setting mean_impute to false if you expect no missing values, both for performance and as a sanity check.

Warning

If the rows of the matrix table have been filtered to a small fraction, then MatrixTable.repartition() before this method to improve performance.

Parameters:
  • y (Float64Expression) – Column-indexed expression for the observations (rows of \(y\)). Must have no missing values.
  • x (list of Float64Expression) – Non-empty list of column-indexed expressions for the fixed effects (rows of \(X\)). Each expression must have the same source as y or no source (e.g., the intercept 1.0). Must have no missing values.
  • z_t (Float64Expression, optional) – Entry-indexed expression for each mixed effect. These values are row-standardized to variance \(1 / m\) to form the entries of \(Z^T\). If mean_impute is false, must have no missing values. Exactly one of z_t and k must be set.
  • k (ndarray, optional) – Kinship matrix \(K\). Exactly one of z_t and k must be set.
  • p_path (str, optional) – Path at which to write the projection \(P\) as a block matrix. Required if z_t is set.
  • overwrite (bool) – If True, overwrite an existing file at p_path.
  • standardize (bool) – If True, standardize z_t by row to mean 0 and variance \(\frac{1}{m}\).
  • mean_impute (bool) – If True, mean-impute missing values of z_t by row.
Returns:

  • model (LinearMixedModel) – Linear mixed model ready to be fit.
  • p (ndarray or BlockMatrix) – Matrix \(P\) whose rows are the eigenvectors of \(K\). The type is block matrix if the model is low rank (i.e., if z_t is set and \(n > m\)).

hail.methods.linear_mixed_regression_rows(entry_expr, model, pa_t_path=None, a_t_path=None, mean_impute=True, partition_size=None, pass_through=())[source]

For each row, test an input variable for association using a linear mixed model.

Examples

See the example in linear_mixed_model() and section below on efficiently testing multiple responses or sets of fixed effects.

Notes

See LinearMixedModel for details on the model and notation.

This method packages up several steps for convenience:

  1. Read the transformation \(P\) from model.p_path.
  2. Write entry_expr at a_t_path as the block matrix \(A^T\) with block size that of \(P\). The parallelism is n_rows / block_size.
  3. Multiply and write \(A^T P^T\) at pa_t_path. The parallelism is the number of blocks in \((PA)^T\), which equals (n_rows / block_size) * (model.r / block_size).
  4. Compute regression results per row with LinearMixedModel.fit_alternatives(). The parallelism is n_rows / partition_size.

If pa_t_path and a_t_path are not set, temporary files are used.

entry_expr may only have missing values if mean_impute is true, in which case missing values of are set to the row mean. We recommend setting mean_impute to false if you expect no missing values, both for performance and as a sanity check.

Efficiently varying the response or set of fixed effects

Computing \(K\), \(P\), \(S\), \(A^T\), and especially the product \((PA)^T\) may require significant compute when \(n\) and/or \(m\) is large. However these quantities are all independent of the response \(y\) or fixed effects \(X\)! And with the model diagonalized, Step 4 above is fast and scalable.

So having run linear mixed regression once, we can compute \(h^2\) and regression statistics for another response or set of fixed effects on the same samples at the roughly the speed of linear_regression_rows().

For example, having collected another y and x as ndarrays, one can construct a new linear mixed model directly.

Supposing the model is full-rank and p is an ndarray:

>>> model = hl.stats.LinearMixedModel(p @ y, p @ x, s)      
>>> model.fit()                                    
>>> result_ht = model.fit_alternatives(pa_t_path)  

Supposing the model is low-rank and p is a block matrix:

>>> p = BlockMatrix.read(p_path)                             
>>> py, px = (p @ y).to_numpy(), (p @ x).to_numpy()          
>>> model = LinearMixedModel(py, px, s, y, x)                
>>> model.fit()                                              
>>> result_ht = model.fit_alternatives(pa_t_path, a_t_path)  

In either case, one can easily loop through many responses or conditional analyses. To join results back to the matrix table:

>>> dataset = dataset.add_row_index()                                    
>>> dataset = dataset.annotate_rows(lmmreg=result_ht[dataset.row_idx]])  

Warning

For correct results, the column-index of entry_expr must correspond to the sample index of the model. This will be true, for example, if model was created with linear_mixed_model() using (a possibly row-filtered version of) the source of entry_expr, or if y and x were collected to arrays from this source. Hail will raise an error if the number of columns does not match model.n, but will not detect, for example, permuted samples.

The warning on BlockMatrix.write_from_entry_expr() applies to this method when the number of samples is large.

Note

Use the pass_through parameter to include additional row fields from matrix table underlying entry_expr. For example, to include an “rsid” field, set` pass_through=[‘rsid’]`` or pass_through=[mt.rsid].

Parameters:
  • entry_expr (Float64Expression) – Entry-indexed expression for input variable. If mean_impute is false, must have no missing values.
  • model (LinearMixedModel) – Fit linear mixed model with path_p set.
  • pa_t_path (str, optional) – Path at which to store the transpose of \(PA\). If not set, a temporary file is used.
  • a_t_path (str, optional) – Path at which to store the transpose of \(A\). If not set, a temporary file is used.
  • mean_impute (bool) – Mean-impute missing values of entry_expr by row.
  • partition_size (int) – Number of rows to process per partition. Default given by block size of \(P\).
  • pass_through (list of str or Expression) – Additional row fields to include in the resulting table.
Returns:

Table

hail.methods.linear_regression_rows(y, x, covariates, block_size=16, pass_through=()) → hail.table.Table[source]

For each row, test an input variable for association with response variables using linear regression.

Examples

>>> result_ht = hl.linear_regression_rows(
...     y=dataset.pheno.height,
...     x=dataset.GT.n_alt_alleles(),
...     covariates=[1, dataset.pheno.age, dataset.pheno.is_female])

Warning

As in the example, the intercept covariate 1 must be included explicitly if desired.

Warning

If y is a single value or a list, linear_regression_rows() considers the same set of columns (i.e., samples, points) for every response variable and row, namely those columns for which all response variables and covariates are defined.

If y is a list of lists, then each inner list is treated as an independent group, subsetting columns for missingness separately.

Notes

With the default root and y a single expression, the following row-indexed fields are added.

  • <row key fields> (Any) – Row key fields.
  • <pass_through fields> (Any) – Row fields in pass_through.
  • n (tint32) – Number of columns used.
  • sum_x (tfloat64) – Sum of input values x.
  • y_transpose_x (tfloat64) – Dot product of response vector y with the input vector x.
  • beta (tfloat64) – Fit effect coefficient of x, \(\hat\beta_1\) below.
  • standard_error (tfloat64) – Estimated standard error, \(\widehat{\mathrm{se}}_1\).
  • t_stat (tfloat64) – \(t\)-statistic, equal to \(\hat\beta_1 / \widehat{\mathrm{se}}_1\).
  • p_value (tfloat64) – \(p\)-value.

If y is a list of expressions, then the last five fields instead have type tarray of tfloat64, with corresponding indexing of the list and each array.

If y is a list of lists of expressions, then n and sum_x are of type array<float64>, and the last five fields are of type array<array<float64>>. Index into these arrays with a[index_in_outer_list, index_in_inner_list]. For example, if y=[[a], [b, c]] then the p-value for b is p_value[1][0].

In the statistical genetics example above, the input variable x encodes genotype as the number of alternate alleles (0, 1, or 2). For each variant (row), genotype is tested for association with height controlling for age and sex, by fitting the linear regression model:

\[\mathrm{height} = \beta_0 + \beta_1 \, \mathrm{genotype} + \beta_2 \, \mathrm{age} + \beta_3 \, \mathrm{is\_female} + \varepsilon, \quad \varepsilon \sim \mathrm{N}(0, \sigma^2)\]

Boolean covariates like \(\mathrm{is\_female}\) are encoded as 1 for True and 0 for False. The null model sets \(\beta_1 = 0\).

The standard least-squares linear regression model is derived in Section 3.2 of The Elements of Statistical Learning, 2nd Edition. See equation 3.12 for the t-statistic which follows the t-distribution with \(n - k - 1\) degrees of freedom, under the null hypothesis of no effect, with \(n\) samples and \(k\) covariates in addition to x.

Note

Use the pass_through parameter to include additional row fields from matrix table underlying x. For example, to include an “rsid” field, set pass_through=['rsid'] or pass_through=[mt.rsid].

Parameters:
  • y (Float64Expression or list of Float64Expression) – One or more column-indexed response expressions.
  • x (Float64Expression) – Entry-indexed expression for input variable.
  • covariates (list of Float64Expression) – List of column-indexed covariate expressions.
  • block_size (int) – Number of row regressions to perform simultaneously per core. Larger blocks require more memory but may improve performance.
  • pass_through (list of str or Expression) – Additional row fields to include in the resulting table.
Returns:

Table

hail.methods.logistic_regression_rows(test, y, x, covariates, pass_through=()) → hail.table.Table[source]

For each row, test an input variable for association with a binary response variable using logistic regression.

Examples

Run the logistic regression Wald test per variant using a Boolean phenotype, intercept and two covariates stored in column-indexed fields:

>>> result_ht = hl.logistic_regression_rows(
...     test='wald',
...     y=dataset.pheno.is_case,
...     x=dataset.GT.n_alt_alleles(),
...     covariates=[1, dataset.pheno.age, dataset.pheno.is_female])

Warning

logistic_regression_rows() considers the same set of columns (i.e., samples, points) for every row, namely those columns for which all covariates are defined. For each row, missing values of x are mean-imputed over these columns. As in the example, the intercept covariate 1 must be included explicitly if desired.

Notes

This method performs, for each row, a significance test of the input variable in predicting a binary (case-control) response variable based on the logistic regression model. The response variable type must either be numeric (with all present values 0 or 1) or Boolean, in which case true and false are coded as 1 and 0, respectively.

Hail supports the Wald test (‘wald’), likelihood ratio test (‘lrt’), Rao score test (‘score’), and Firth test (‘firth’). Hail only includes columns for which the response variable and all covariates are defined. For each row, Hail imputes missing input values as the mean of the non-missing values.

The example above considers a model of the form

\[\mathrm{Prob}(\mathrm{is_case}) = \mathrm{sigmoid}(\beta_0 + \beta_1 \, \mathrm{gt} + \beta_2 \, \mathrm{age} + \beta_3 \, \mathrm{is\_female} + \varepsilon), \quad \varepsilon \sim \mathrm{N}(0, \sigma^2)\]

where \(\mathrm{sigmoid}\) is the sigmoid function, the genotype \(\mathrm{gt}\) is coded as 0 for HomRef, 1 for Het, and 2 for HomVar, and the Boolean covariate \(\mathrm{is\_female}\) is coded as for True (female) and 0 for False (male). The null model sets \(\beta_1 = 0\).

The structure of the emitted row field depends on the test statistic as shown in the tables below.

Test Field Type Value
Wald beta float64 fit effect coefficient, \(\hat\beta_1\)
Wald standard_error float64 estimated standard error, \(\widehat{\mathrm{se}}\)
Wald z_stat float64 Wald \(z\)-statistic, equal to \(\hat\beta_1 / \widehat{\mathrm{se}}\)
Wald p_value float64 Wald p-value testing \(\beta_1 = 0\)
LRT, Firth beta float64 fit effect coefficient, \(\hat\beta_1\)
LRT, Firth chi_sq_stat float64 deviance statistic
LRT, Firth p_value float64 LRT / Firth p-value testing \(\beta_1 = 0\)
Score chi_sq_stat float64 score statistic
Score p_value float64 score p-value testing \(\beta_1 = 0\)

For the Wald and likelihood ratio tests, Hail fits the logistic model for each row using Newton iteration and only emits the above fields when the maximum likelihood estimate of the coefficients converges. The Firth test uses a modified form of Newton iteration. To help diagnose convergence issues, Hail also emits three fields which summarize the iterative fitting process:

Test Field Type Value
Wald, LRT, Firth fit.n_iterations int32 number of iterations until convergence, explosion, or reaching the max (25 for Wald, LRT; 100 for Firth)
Wald, LRT, Firth fit.converged bool True if iteration converged
Wald, LRT, Firth fit.exploded bool True if iteration exploded

We consider iteration to have converged when every coordinate of \(\beta\) changes by less than \(10^{-6}\). For Wald and LRT, up to 25 iterations are attempted; in testing we find 4 or 5 iterations nearly always suffice. Convergence may also fail due to explosion, which refers to low-level numerical linear algebra exceptions caused by manipulating ill-conditioned matrices. Explosion may result from (nearly) linearly dependent covariates or complete separation.

A more common situation in genetics is quasi-complete seperation, e.g. variants that are observed only in cases (or controls). Such variants inevitably arise when testing millions of variants with very low minor allele count. The maximum likelihood estimate of \(\beta\) under logistic regression is then undefined but convergence may still occur after a large number of iterations due to a very flat likelihood surface. In testing, we find that such variants produce a secondary bump from 10 to 15 iterations in the histogram of number of iterations per variant. We also find that this faux convergence produces large standard errors and large (insignificant) p-values. To not miss such variants, consider using Firth logistic regression, linear regression, or group-based tests.

Here’s a concrete illustration of quasi-complete seperation in R. Suppose we have 2010 samples distributed as follows for a particular variant:

Status HomRef Het HomVar
Case 1000 10 0
Control 1000 0 0

The following R code fits the (standard) logistic, Firth logistic, and linear regression models to this data, where x is genotype, y is phenotype, and logistf is from the logistf package:

x <- c(rep(0,1000), rep(1,1000), rep(1,10)
y <- c(rep(0,1000), rep(0,1000), rep(1,10))
logfit <- glm(y ~ x, family=binomial())
firthfit <- logistf(y ~ x)
linfit <- lm(y ~ x)

The resulting p-values for the genotype coefficient are 0.991, 0.00085, and 0.0016, respectively. The erroneous value 0.991 is due to quasi-complete separation. Moving one of the 10 hets from case to control eliminates this quasi-complete separation; the p-values from R are then 0.0373, 0.0111, and 0.0116, respectively, as expected for a less significant association.

The Firth test reduces bias from small counts and resolves the issue of separation by penalizing maximum likelihood estimation by the Jeffrey’s invariant prior. This test is slower, as both the null and full model must be fit per variant, and convergence of the modified Newton method is linear rather than quadratic. For Firth, 100 iterations are attempted for the null model and, if that is successful, for the full model as well. In testing we find 20 iterations nearly always suffices. If the null model fails to converge, then the logreg.fit fields reflect the null model; otherwise, they reflect the full model.

See Recommended joint and meta-analysis strategies for case-control association testing of single low-count variants for an empirical comparison of the logistic Wald, LRT, score, and Firth tests. The theoretical foundations of the Wald, likelihood ratio, and score tests may be found in Chapter 3 of Gesine Reinert’s notes Statistical Theory. Firth introduced his approach in Bias reduction of maximum likelihood estimates, 1993. Heinze and Schemper further analyze Firth’s approach in A solution to the problem of separation in logistic regression, 2002.

Hail’s logistic regression tests correspond to the b.wald, b.lrt, and b.score tests in EPACTS. For each variant, Hail imputes missing input values as the mean of non-missing input values, whereas EPACTS subsets to those samples with called genotypes. Hence, Hail and EPACTS results will currently only agree for variants with no missing genotypes.

Note

Use the pass_through parameter to include additional row fields from matrix table underlying x. For example, to include an “rsid” field, set pass_through=['rsid'] or pass_through=[mt.rsid].

Parameters:
  • test ({‘wald’, ‘lrt’, ‘score’, ‘firth’}) – Statistical test.
  • y (Float64Expression) – Column-indexed response expression. All non-missing values must evaluate to 0 or 1. Note that a BooleanExpression will be implicitly converted to a Float64Expression with this property.
  • x (Float64Expression) – Entry-indexed expression for input variable.
  • covariates (list of Float64Expression) – Non-empty list of column-indexed covariate expressions.
  • pass_through (list of str or Expression) – Additional row fields to include in the resulting table.
Returns:

Table

hail.methods.poisson_regression_rows(test, y, x, covariates, pass_through=()) → hail.table.Table[source]

For each row, test an input variable for association with a count response variable using Poisson regression.

Notes

See logistic_regression_rows() for more info on statistical tests of general linear models.

Note

Use the pass_through parameter to include additional row fields from matrix table underlying x. For example, to include an “rsid” field, set pass_through=['rsid'] or pass_through=[mt.rsid].

Parameters:
  • y (Float64Expression) – Column-indexed response expression. All non-missing values must evaluate to a non-negative integer.
  • x (Float64Expression) – Entry-indexed expression for input variable.
  • covariates (list of Float64Expression) – Non-empty list of column-indexed covariate expressions.
  • pass_through (list of str or Expression) – Additional row fields to include in the resulting table.
Returns:

Table

hail.methods.pca(entry_expr, k=10, compute_loadings=False) → Tuple[[List[float], hail.table.Table], hail.table.Table][source]

Run principal component analysis (PCA) on numeric columns derived from a matrix table.

Examples

For a matrix table with variant rows, sample columns, and genotype entries, compute the top 2 PC sample scores and eigenvalues of the matrix of 0s and 1s encoding missingness of genotype calls.

>>> eigenvalues, scores, _ = hl.pca(hl.int(hl.is_defined(dataset.GT)),
...                                 k=2)

Warning

This method does not automatically mean-center or normalize each column. If desired, such transformations should be incorporated in entry_expr.

Hail will return an error if entry_expr evaluates to missing, nan, or infinity on any entry.

Notes

PCA is run on the columns of the numeric matrix obtained by evaluating entry_expr on each entry of the matrix table, or equivalently on the rows of the transposed numeric matrix \(M\) referenced below.

PCA computes the SVD

\[M = USV^T\]

where columns of \(U\) are left singular vectors (orthonormal in \(\mathbb{R}^n\)), columns of \(V\) are right singular vectors (orthonormal in \(\mathbb{R}^m\)), and \(S=\mathrm{diag}(s_1, s_2, \ldots)\) with ordered singular values \(s_1 \ge s_2 \ge \cdots \ge 0\). Typically one computes only the first \(k\) singular vectors and values, yielding the best rank \(k\) approximation \(U_k S_k V_k^T\) of \(M\); the truncations \(U_k\), \(S_k\) and \(V_k\) are \(n \times k\), \(k \times k\) and \(m \times k\) respectively.

From the perspective of the rows of \(M\) as samples (data points), \(V_k\) contains the loadings for the first \(k\) PCs while \(MV_k = U_k S_k\) contains the first \(k\) PC scores of each sample. The loadings represent a new basis of features while the scores represent the projected data on those features. The eigenvalues of the Gramian \(MM^T\) are the squares of the singular values \(s_1^2, s_2^2, \ldots\), which represent the variances carried by the respective PCs. By default, Hail only computes the loadings if the loadings parameter is specified.

Scores are stored in a Table with the column key of the matrix table as key and a field scores of type array<float64> containing the principal component scores.

Loadings are stored in a Table with the row key of the matrix table as key and a field loadings of type array<float64> containing the principal component loadings.

The eigenvalues are returned in descending order, with scores and loadings given the corresponding array order.

Parameters:
  • entry_expr (Expression) – Numeric expression for matrix entries.
  • k (int) – Number of principal components.
  • compute_loadings (bool) – If True, compute row loadings.
Returns:

(list of float, Table, Table) – List of eigenvalues, table with column scores, table with row loadings.

hail.methods.row_correlation(entry_expr, block_size=None) → hail.linalg.blockmatrix.BlockMatrix[source]

Computes the correlation matrix between row vectors.

Examples

Consider the following dataset with three variants and four samples:

>>> data = [{'v': '1:1:A:C', 's': 'a', 'GT': hl.Call([0, 0])},
...         {'v': '1:1:A:C', 's': 'b', 'GT': hl.Call([0, 0])},
...         {'v': '1:1:A:C', 's': 'c', 'GT': hl.Call([0, 1])},
...         {'v': '1:1:A:C', 's': 'd', 'GT': hl.Call([1, 1])},
...         {'v': '1:2:G:T', 's': 'a', 'GT': hl.Call([0, 1])},
...         {'v': '1:2:G:T', 's': 'b', 'GT': hl.Call([1, 1])},
...         {'v': '1:2:G:T', 's': 'c', 'GT': hl.Call([0, 1])},
...         {'v': '1:2:G:T', 's': 'd', 'GT': hl.Call([0, 0])},
...         {'v': '1:3:C:G', 's': 'a', 'GT': hl.Call([0, 1])},
...         {'v': '1:3:C:G', 's': 'b', 'GT': hl.Call([0, 0])},
...         {'v': '1:3:C:G', 's': 'c', 'GT': hl.Call([1, 1])},
...         {'v': '1:3:C:G', 's': 'd', 'GT': hl.null(hl.tcall)}]
>>> ht = hl.Table.parallelize(data, hl.dtype('struct{v: str, s: str, GT: call}'))
>>> mt = ht.to_matrix_table(row_key=['v'], col_key=['s'])

Compute genotype correlation between all pairs of variants:

>>> ld = hl.row_correlation(mt.GT.n_alt_alleles())
>>> ld.to_numpy()
array([[ 1.        , -0.85280287,  0.42640143],
       [-0.85280287,  1.        , -0.5       ],
       [ 0.42640143, -0.5       ,  1.        ]])

Compute genotype correlation between consecutively-indexed variants:

>>> ld.sparsify_band(lower=0, upper=1).to_numpy()
array([[ 1.        , -0.85280287,  0.        ],
       [ 0.        ,  1.        , -0.5       ],
       [ 0.        ,  0.        ,  1.        ]])

Warning

Rows with a constant value (i.e., zero variance) will result nan correlation values. To avoid this, first check that all rows vary or filter out constant rows (for example, with the help of aggregators.stats()).

Notes

In this method, each row of entries is regarded as a vector with elements defined by entry_expr and missing values mean-imputed per row. The (i, j) element of the resulting block matrix is the correlation between rows i and j (as 0-indexed by order in the matrix table; see add_row_index()).

The correlation of two vectors is defined as the Pearson correlation coeffecient between the corresponding empirical distributions of elements, or equivalently as the cosine of the angle between the vectors.

This method has two stages:

  • writing the row-normalized block matrix to a temporary file on persistent disk with BlockMatrix.from_entry_expr(). The parallelism is n_rows / block_size.
  • reading and multiplying this block matrix by its transpose. The parallelism is (n_rows / block_size)^2 if all blocks are computed.

Warning

See all warnings on BlockMatrix.from_entry_expr(). In particular, for large matrices, it may be preferable to run the two stages separately, saving the row-normalized block matrix to a file on external storage with BlockMatrix.write_from_entry_expr().

The resulting number of matrix elements is the square of the number of rows in the matrix table, so computing the full matrix may be infeasible. For example, ten million rows would produce 800TB of float64 values. The block-sparse representation on BlockMatrix may be used to work efficiently with regions of such matrices, as in the second example above and ld_matrix().

To prevent excessive re-computation, be sure to write and read the (possibly block-sparsified) result before multiplication by another matrix.

Parameters:
Returns:

BlockMatrix – Correlation matrix between row vectors. Row and column indices correspond to matrix table row index.