ArrayNumericExpression

class hail.expr.ArrayNumericExpression[source]

Expression of type tarray with a numeric type.

Numeric arrays support arithmetic both with scalar values and other arrays. Arithmetic between two numeric arrays requires that the length of each array is identical, and will apply the operation positionally (a1 * a2 will multiply the first element of a1 by the first element of a2, the second element of a1 by the second element of a2, and so on). Arithmetic with a scalar will apply the operation to each element of the array.

>>> a1 = hl.literal([0, 1, 2, 3, 4, 5])

>>> a2 = hl.literal([1, -1, 1, -1, 1, -1])

Attributes

dtype

The data type of the expression.

Methods

__add__(other)[source]

Positionally add an array or a scalar.

Examples

>>> hl.eval(a1 + 5)
[5, 6, 7, 8, 9, 10]

>>> hl.eval(a1 + a2)
[1, 0, 3, 2, 5, 4]

Parameters:: other (NumericExpression or ArrayNumericExpression) – Value or array to add.
Returns:: ArrayNumericExpression – Array of positional sums.

__eq__(other)

Returns True if the two expressions are equal.

Examples

>>> x = hl.literal(5)
>>> y = hl.literal(5)
>>> z = hl.literal(1)

>>> hl.eval(x == y)
True

>>> hl.eval(x == z)
False

Notes

This method will fail with an error if the two expressions are not of comparable types.

Parameters:: other (Expression) – Expression for equality comparison.
Returns:: BooleanExpression – True if the two expressions are equal.

__floordiv__(other)[source]

Positionally divide by an array or a scalar using floor division.

Examples

>>> hl.eval(a1 // 2)
[0, 0, 1, 1, 2, 2]

Parameters:: other (NumericExpression or ArrayNumericExpression)
Returns:: ArrayNumericExpression

__ge__(other): Return self>=value.

__getitem__(item)

Index into or slice the array.

Examples

Index with a single integer:

>>> hl.eval(names[1])
'Bob'

>>> hl.eval(names[-1])
'Charlie'

Slicing is also supported:

>>> hl.eval(names[1:])
['Bob', 'Charlie']

Parameters:: item (slice or Expression of type tint32) – Index or slice.
Returns:: Expression – Element or array slice.

__gt__(other): Return self>value.

__le__(other): Return self<=value.

__lt__(other): Return self<value.

__mod__(other)[source]

Positionally compute the left modulo the right.

Examples

>>> hl.eval(a1 % 2)
[0, 1, 0, 1, 0, 1]

Parameters:: other (NumericExpression or ArrayNumericExpression)
Returns:: ArrayNumericExpression

__mul__(other)[source]

Positionally multiply by an array or a scalar.

Examples

>>> hl.eval(a2 * 5)
[5, -5, 5, -5, 5, -5]

>>> hl.eval(a1 * a2)
[0, -1, 2, -3, 4, -5]

Parameters:: other (NumericExpression or ArrayNumericExpression) – Value or array to multiply by.
Returns:: ArrayNumericExpression – Array of positional products.

__ne__(other)

Returns True if the two expressions are not equal.

Examples

>>> x = hl.literal(5)
>>> y = hl.literal(5)
>>> z = hl.literal(1)

>>> hl.eval(x != y)
False

>>> hl.eval(x != z)
True

Notes

This method will fail with an error if the two expressions are not of comparable types.

Parameters:: other (Expression) – Expression for inequality comparison.
Returns:: BooleanExpression – True if the two expressions are not equal.

__neg__()[source]

Negate elements of the array.

Examples

>>> hl.eval(-a1)
[0, -1, -2, -3, -4, -5]

Returns:: ArrayNumericExpression – Array expression of the same type.

__pow__(other)[source]

Positionally raise to the power of an array or a scalar.

Examples

>>> hl.eval(a1 ** 2)
[0.0, 1.0, 4.0, 9.0, 16.0, 25.0]

>>> hl.eval(a1 ** a2)
[0.0, 1.0, 2.0, 0.3333333333333333, 4.0, 0.2]

Parameters:: other (NumericExpression or ArrayNumericExpression)
Returns:: ArrayNumericExpression

__sub__(other)[source]

Positionally subtract an array or a scalar.

Examples

>>> hl.eval(a2 - 1)
[0, -2, 0, -2, 0, -2]

>>> hl.eval(a1 - a2)
[-1, 2, 1, 4, 3, 6]

Parameters:: other (NumericExpression or ArrayNumericExpression) – Value or array to subtract.
Returns:: ArrayNumericExpression – Array of positional differences.

__truediv__(other)[source]

Positionally divide by an array or a scalar.

Examples

>>> hl.eval(a1 / 10)  
[0.0, 0.1, 0.2, 0.3, 0.4, 0.5]

>>> hl.eval(a2 / a1)  
[inf, -1.0, 0.5, -0.3333333333333333, 0.25, -0.2]

Parameters:: other (NumericExpression or ArrayNumericExpression) – Value or array to divide by.
Returns:: ArrayNumericExpression – Array of positional quotients.

aggregate(f)

Uses the aggregator library to compute a summary from an array.

This method is useful for accessing functionality that exists in the aggregator library but not the basic expression library, for instance, call_stats().

Parameters:: f – Aggregation function
Returns:: Expression

all(f)

Returns True if f returns True for every element.

Examples

>>> hl.eval(a.all(lambda x: x < 10))
True

Notes

This method returns True if the collection is empty.

Parameters:: f (function ( (arg) -> BooleanExpression)) – Function to evaluate for each element of the collection. Must return a BooleanExpression.
Returns:: BooleanExpression. – True if f returns True for every element, False otherwise.

any(f)

Returns True if f returns True for any element.

Examples

>>> hl.eval(a.any(lambda x: x % 2 == 0))
True

>>> hl.eval(s3.any(lambda x: x[0] == 'D'))
False

Notes

This method always returns False for empty collections.

Parameters:: f (function ( (arg) -> BooleanExpression)) – Function to evaluate for each element of the collection. Must return a BooleanExpression.
Returns:: BooleanExpression. – True if f returns True for any element, False otherwise.

append(item)

Append an element to the array and return the result.

Examples

>>> hl.eval(names.append('Dan'))
['Alice', 'Bob', 'Charlie', 'Dan']

Note

This method does not mutate the caller, but instead returns a new array by copying the caller and adding item.

Parameters:: item (Expression) – Element to append, same type as the array element type.
Returns:: ArrayExpression

collect(_localize=True)

Collect all records of an expression into a local list.

Examples

Collect all the values from C1:

>>> table1.C1.collect()
[2, 2, 10, 11]

Warning

Extremely experimental.

Warning

The list of records may be very large.

Returns:: list

contains(item)

Returns a boolean indicating whether item is found in the array.

Examples

>>> hl.eval(names.contains('Charlie'))
True

>>> hl.eval(names.contains('Helen'))
False

Parameters:: item (Expression) – Item for inclusion test.

Warning

This method takes time proportional to the length of the array. If a pipeline uses this method on the same array several times, it may be more efficient to convert the array to a set first early in the script (set()).

Returns:: BooleanExpression – True if the element is found in the array, False otherwise.

describe(handler=<built-in function print>): Print information about type, index, and dependencies.

property dtype

The data type of the expression.

Returns:: HailType

export(path, delimiter='\t', missing='NA', header=True)

Export a field to a text file.

Examples

>>> small_mt.GT.export('output/gt.tsv')
>>> with open('output/gt.tsv', 'r') as f:
...     for line in f:
...         print(line, end='')
locus   alleles 0       1       2       3
1:1     ["A","C"]       0/1     0/0     0/1     0/0
1:2     ["A","C"]       1/1     0/1     0/1     0/1
1:3     ["A","C"]       0/0     0/1     0/0     0/0
1:4     ["A","C"]       0/1     1/1     0/1     0/1

>>> small_mt.GT.export('output/gt-no-header.tsv', header=False)
>>> with open('output/gt-no-header.tsv', 'r') as f:
...     for line in f:
...         print(line, end='')
1:1     ["A","C"]       0/1     0/0     0/1     0/0
1:2     ["A","C"]       1/1     0/1     0/1     0/1
1:3     ["A","C"]       0/0     0/1     0/0     0/0
1:4     ["A","C"]       0/1     1/1     0/1     0/1

>>> small_mt.pop.export('output/pops.tsv')
>>> with open('output/pops.tsv', 'r') as f:
...     for line in f:
...         print(line, end='')
sample_idx      pop
0       1
1       2
2       2
3       2

>>> small_mt.ancestral_af.export('output/ancestral_af.tsv')
>>> with open('output/ancestral_af.tsv', 'r') as f:
...     for line in f:
...         print(line, end='')
locus   alleles ancestral_af
1:1     ["A","C"]       3.8152e-01
1:2     ["A","C"]       7.0588e-01
1:3     ["A","C"]       4.9991e-01
1:4     ["A","C"]       3.9616e-01

>>> small_mt.bn.export('output/bn.tsv')
>>> with open('output/bn.tsv', 'r') as f:
...     for line in f:
...         print(line, end='')
bn
{"n_populations":3,"n_samples":4,"n_variants":4,"n_partitions":4,"pop_dist":[1,1,1],"fst":[0.1,0.1,0.1],"mixture":false}

Notes

For entry-indexed expressions, if there is one column key field, the result of calling str() on that field is used as the column header. Otherwise, each compound column key is converted to JSON and used as a column header. For example:

>>> small_mt = small_mt.key_cols_by(s=small_mt.sample_idx, family='fam1')
>>> small_mt.GT.export('output/gt-no-header.tsv')
>>> with open('output/gt-no-header.tsv', 'r') as f:
...     for line in f:
...         print(line, end='')
locus   alleles {"s":0,"family":"fam1"} {"s":1,"family":"fam1"} {"s":2,"family":"fam1"} {"s":3,"family":"fam1"}
1:1     ["A","C"]       0/1     0/0     0/1     0/0
1:2     ["A","C"]       1/1     0/1     0/1     0/1
1:3     ["A","C"]       0/0     0/1     0/0     0/0
1:4     ["A","C"]       0/1     1/1     0/1     0/1

Parameters:

path (str) – The path to which to export.
delimiter (str) – The string for delimiting columns.
missing (str) – The string to output for missing values.
header (bool) – When True include a header line.

extend(a)

Concatenate two arrays and return the result.

Examples

>>> hl.eval(names.extend(['Dan', 'Edith']))
['Alice', 'Bob', 'Charlie', 'Dan', 'Edith']

Parameters:: a (ArrayExpression) – Array to concatenate, same type as the callee.
Returns:: ArrayExpression

filter(f)

Returns a new collection containing elements where f returns True.

Examples

>>> hl.eval(a.filter(lambda x: x % 2 == 0))
[2, 4]

>>> hl.eval(s3.filter(lambda x: ~(x[-1] == 'e')))  
{'Bob'}

Notes

Returns a same-type expression; evaluated on a SetExpression, returns a SetExpression. Evaluated on an ArrayExpression, returns an ArrayExpression.

Parameters:: f (function ( (arg) -> BooleanExpression)) – Function to evaluate for each element of the collection. Must return a BooleanExpression.
Returns:: CollectionExpression – Expression of the same type as the callee.

find(f)

Returns the first element where f returns True.

Examples

>>> hl.eval(a.find(lambda x: x ** 2 > 20))
5

>>> hl.eval(s3.find(lambda x: x[0] == 'D'))
None

Notes

If f returns False for every element, then the result is missing.

Parameters:: f (function ( (arg) -> BooleanExpression)) – Function to evaluate for each element of the collection. Must return a BooleanExpression.
Returns:: Expression – Expression whose type is the element type of the collection.

first()

Returns the first element of the array, or missing if empty.

Returns:: Expression – Element.

Examples

>>> hl.eval(names.first())
'Alice'

If the array has no elements, then the result is missing: >>> hl.eval(names.filter(lambda x: x.startswith(‘D’)).first()) None

flatmap(f)

Map each element of the collection to a new collection, and flatten the results.

Examples

>>> hl.eval(a.flatmap(lambda x: hl.range(0, x)))
[0, 0, 1, 0, 1, 2, 0, 1, 2, 3, 0, 1, 2, 3, 4]

>>> hl.eval(s3.flatmap(lambda x: hl.set(hl.range(0, x.length()).map(lambda i: x[i]))))  
{'A', 'B', 'C', 'a', 'b', 'c', 'e', 'h', 'i', 'l', 'o', 'r'}

Parameters:: f (function ( (arg) -> CollectionExpression)) – Function from the element type of the collection to the type of the collection. For instance, flatmap on a set<str> should take a str and return a set.
Returns:: CollectionExpression

fold(f, zero)

Reduces the collection with the given function f, provided the initial value zero.

Examples

>>> a = [0, 1, 2]

>>> hl.eval(hl.fold(lambda i, j: i + j, 0, a))
3

Parameters:

f (function ( (Expression, Expression) -> Expression)) – Function which takes the cumulative value and the next element, and returns a new value.
zero (Expression) – Initial value to pass in as left argument of f.

Returns:

Expression.

group_by(f)

Group elements into a dict according to a lambda function.

Examples

>>> hl.eval(a.group_by(lambda x: x % 2 == 0))  
{False: [1, 3, 5], True: [2, 4]}

>>> hl.eval(s3.group_by(lambda x: x.length()))  
{3: {'Bob'}, 5: {'Alice'}, 7: {'Charlie'}}

Parameters:: f (function ( (arg) -> Expression)) – Function to evaluate for each element of the collection to produce a key for the resulting dictionary.
Returns:: DictExpression. – Dictionary keyed by results of f.

grouped(group_size)

Partition an array into fixed size subarrays.

Examples

>>> a = hl.array([0, 1, 2, 3, 4])

>>> hl.eval(a.grouped(2))
[[0, 1], [2, 3], [4]]

Parameters:: group_size (Int32Expression) – The number of elements per group.
Returns:: ArrayExpression.

head()

Deprecated in favor of first().

Returns the first element of the array, or missing if empty.

Returns:: Expression – Element.

Examples

>>> hl.eval(names.head())
'Alice'

If the array has no elements, then the result is missing:

>>> hl.eval(names.filter(lambda x: x.startswith('D')).head())
None

index(x)

Returns the first index of x, or missing.

Parameters:: x (Expression or typing.Callable) – Value to find, or function from element to Boolean expression.
Returns:: Int32Expression

Examples

>>> hl.eval(names.index('Bob'))
1

>>> hl.eval(names.index('Beth'))
None

>>> hl.eval(names.index(lambda x: x.endswith('e')))
0

>>> hl.eval(names.index(lambda x: x.endswith('h')))
None

last()

Returns the last element of the array, or missing if empty.

Returns:: Expression – Element.

Examples

>>> hl.eval(names.last())
'Charlie'

If the array has no elements, then the result is missing: >>> hl.eval(names.filter(lambda x: x.startswith(‘D’)).last()) None

length()

Returns the size of a collection.

Examples

>>> hl.eval(a.length())
5

>>> hl.eval(s3.length())
3

Returns:: Expression of type tint32 – The number of elements in the collection.

map(f)

Transform each element of a collection.

Examples

>>> hl.eval(a.map(lambda x: x ** 3))
[1.0, 8.0, 27.0, 64.0, 125.0]

>>> hl.eval(s3.map(lambda x: x.length()))
{3, 5, 7}

Parameters:: f (function ( (arg) -> Expression)) – Function to transform each element of the collection.
Returns:: CollectionExpression. – Collection where each element has been transformed according to f.

scan(f, zero)

Map each element of the array to cumulative value of function f, with initial value zero.

Examples

>>> a = [0, 1, 2]

>>> hl.eval(hl.array_scan(lambda i, j: i + j, 0, a))
[0, 0, 1, 3]

Parameters:

f (function ( (Expression, Expression) -> Expression)) – Function which takes the cumulative value and the next element, and returns a new value.
zero (Expression) – Initial value to pass in as left argument of f.

Returns:

ArrayExpression.

show(n=None, width=None, truncate=None, types=True, handler=None, n_rows=None, n_cols=None)

Print the first few records of the expression to the console.

If the expression refers to a value on a keyed axis of a table or matrix table, then the accompanying keys will be shown along with the records.

Examples

>>> table1.SEX.show()
+-------+-----+
|    ID | SEX |
+-------+-----+
| int32 | str |
+-------+-----+
|     1 | "M" |
|     2 | "M" |
|     3 | "F" |
|     4 | "F" |
+-------+-----+

>>> hl.literal(123).show()
+--------+
| <expr> |
+--------+
|  int32 |
+--------+
|    123 |
+--------+

Notes

The output can be passed piped to another output source using the handler argument:

>>> ht.foo.show(handler=lambda x: logging.info(x))  

Parameters:

n (int) – Maximum number of rows to show.
width (int) – Horizontal width at which to break columns.
truncate (int, optional) – Truncate each field to the given number of characters. If None, truncate fields to the given width.
types (bool) – Print an extra header line with the type of each field.

size()

Returns the size of a collection.

Examples

>>> hl.eval(a.size())
5

>>> hl.eval(s3.size())
3

Returns:: Expression of type tint32 – The number of elements in the collection.

starmap(f)

Transform each element of a collection of tuples.

Examples

>>> hl.eval(hl.array([(1, 2), (2, 3)]).starmap(lambda x, y: x+y))
[3, 5]

Parameters:: f (function ( (*args) -> Expression)) – Function to transform each element of the collection.
Returns:: CollectionExpression. – Collection where each element has been transformed according to f.

summarize(handler=None): Compute and print summary information about the expression.

Danger

This functionality is experimental. It may not be tested as well as other parts of Hail and the interface is subject to change.

take(n, _localize=True)

Collect the first n records of an expression.

Examples

Take the first three rows:

>>> table1.X.take(3)
[5, 6, 7]

Warning

Extremely experimental.

Parameters:: n (int) – Number of records to take.
Returns:: list