--------------------
Expressions Overview
--------------------
What is an Expression?
======================
Hail's expressions are lazy representations of data.
Each data type in Hail has its own :class:`.Expression` class. For example, an
:class:`.Int32Expression` represents a 32-bit integer, and a
:class:`.BooleanExpression` represents a boolean value of True or False.
>>> hl.int32(5)
>>> hl.bool(True)
Expressions can be combined with operations to form new expressions. Much like
you would add two integers in Python, you can also add two
:class:`.Int32Expression` objects in Hail.
>>> hl.int32(5) + hl.int32(6)
The result of adding two :class:`.Int32Expression` objects is another
:class:`.Int32Expression` object.
We say Hail's expressions are **lazy**, because they are not evaluated until the
result of the expression is needed. Let's explore what this means by comparing a
Python expression to a Hail expression.
In Python, an expression such as ``5+6`` will be immediately evaluated. If you
enter this expression into Python, you'll see the result, ``11``, right away.
>>> x = 5
>>> y = 6
>>> z = x + y
>>> z
11
The equivalent code written with Hail's expressions would look like:
>>> x = hl.int32(5)
>>> y = hl.int32(6)
>>> z = x + y
>>> z
Notice that when we enter ``z``, we don't see the result, ``11``, like we
did with Python. Hail is not running Python code on your data. Instead, Hail is
keeping track of the computations applied to your data, compiling these
computations into native code, and running them in parallel.
The result of the expression is computed only when it is needed. So ``z`` is
an expression representing the computation of ``x + y``, but not the actual
value.
To peek at the value of this computation, there are two options:
:func:`~hail.expr.eval`, which returns a Python value, and :meth:`.Expression.show`,
which prints a human-readable representation of an expression.
>>> hl.eval(z)
11
>>> z.show()
+--------+
| |
+--------+
| int32 |
+--------+
| 11 |
+--------+
Hail's expressions are especially important for interacting with fields in
tables and matrix tables. Throughout Hail documentation and tutorials, you will
see code like this:
>>> ht2 = ht.annotate(C4 = ht.C3 + 3 * ht.C2 ** 2)
This snippet of code is adding a field, ``C4``, to a table, ``ht``, and
returning the result as a new table, ``ht2``. The code passed to the
:meth:`.Table.annotate` method is an expression that references the fields
``C3`` and ``C2`` in ``ht``.
Notice that ``3`` and ``2`` are not wrapped in constructor functions like
``hl.int32(3)``. In the same way that Hail expressions can be combined together
via operations like addition and multiplication, they can also be combined with
Python objects.
For example, we can add a Python :obj:`int` to an :class:`.Int32Expression`.
>>> x + 3
Addition is commutative, so we can also add an :class:`.Int32Expression` to an
:obj:`int`.
>>> 3 + x
Note that Hail expressions cannot be used in other modules, like :mod:`numpy`
or :mod:`scipy`.
Hail has many subclasses of :class:`.Expression` -- one for each Hail type. Each
subclass has its own constructor method. For example, if we have a list of Python
integers, we can convert this to a Hail :class:`.ArrayNumericExpression` with
:func:`~hail.expr.functions.array`:
>>> a = hl.array([1, 2, -3, 0, 5])
>>> a
>
:class:`.Expression` objects keep track of their data type, which is
why we can see that ``a`` is of type ``array`` in the output above. An
expression's type can also be accessed with :meth:`.Expression.dtype`.
>>> a.dtype
dtype('array')
Hail arrays can be indexed and sliced like Python lists or :mod:`numpy` arrays:
>>> a[1]
>>> a[1:-1]
>
In addition to constructor methods like :func:`~hail.expr.functions.array` and :func:`.bool`,
Hail expressions can also be constructed with the :func:`.literal` method,
which will impute the type of of the expression.
>>> hl.literal([0,1,2])
>
Boolean Logic
=============
Unlike Python, a Hail :class:`.BooleanExpression` cannot be used with the Python
keywords ``and``, ``or``, and ``not``. The Hail substitutes are ``&``, ``|``,
and ``~``.
>>> s1 = hl.int32(3) == 4
>>> s2 = hl.int32(3) != 4
>>> s1 & s2
>>> s1 | s2
>>> ~s1
Remember that you can use :func:`~hail.expr.eval`: to evaluate the expression.
>>> hl.eval(~s1)
True
.. caution::
The operator precedence of ``&`` and ``|`` is different from ``and`` and
``or``. You will need parentheses around expressions like this:
>>> (x == 3) & (x != 4)
Conditional Expressions
=======================
If/Else Statements
~~~~~~~~~~~~~~~~~~
Python ``if`` / ``else`` statements do not work with Hail expressions. Instead,
you must use the :func:`.if_else`, :func:`.case`, and :func:`.switch` functions.
A conditional expression has three components: the condition to evaluate, the
consequent value to return if the condition is ``True``, and the alternate to
return if the condition is ``False``. For example:
.. code-block:: python
if (x > 0):
return 1
else:
return 0
In the above conditional, the condition is ``x > 0``, the consequent is ``1``,
and the alternate is ``0``.
Here is the Hail expression equivalent with :func:`.if_else`:
>>> hl.if_else(x > 0, 1, 0)
This example returns an :class:`.Int32Expression` which can be used in more
computations. We can add the conditional expression to our array ``a`` from
earlier:
>>> a + hl.if_else(x > 0, 1, 0)
>
Case Statements
~~~~~~~~~~~~~~~
More complicated conditional statements can be constructed with :func:`.case`.
For example, we might want to return ``1`` if ``x < -1``, ``2`` if
``-1 <= x <= 2`` and ``3`` if ``x > 2``.
>>> (hl.case()
... .when(x < -1, 1)
... .when((x >= -1) & (x <= 2), 2)
... .when(x > 2, 3)
... .or_missing())
Notice that this expression ends with a call to :meth:`~hail.expr.builders.CaseBuilder.or_missing`,
which means that if none of the conditions are met, a missing value is returned.
Cases started with :func:`.case` can end with a call to
:meth:`~hail.expr.builders.CaseBuilder.or_missing`, :meth:`~hail.expr.builders.CaseBuilder.default`, or
:meth:`~hail.expr.builders.CaseBuilder.or_error`, depending on what you want to happen if none
of the *when* clauses are met.
It's important to note that missingness propagates up in Hail, so if the value
of the discriminant in a case statement is missing, then the result will be
missing as well.
>>> y = hl.missing(hl.tint32)
>>> result = hl.case().when(y > 0, 1).default(-1)
>>> hl.eval(result)
The value of ``result`` will be missing, not ``1`` or ``-1``, because the
discriminant, ``y``, is missing.
Switch Statements
~~~~~~~~~~~~~~~~~
Finally, Hail has the :func:`.switch` function to build a conditional tree based
on the value of an expression. In the example below, ``csq`` is a
:class:`.StringExpression` representing the functional consequence of a
mutation. If ``csq`` does not match one of the cases specified by
:meth:`~hail.expr.builders.SwitchBuilder.when`, it is set to missing with
:meth:`~hail.expr.builders.SwitchBuilder.or_missing`. Other switch statements are documented in the
:class:`.SwitchBuilder` class.
>>> csq = hl.str('nonsense')
>>> (hl.switch(csq)
... .when("synonymous", False)
... .when("intron", False)
... .when("nonsense", True)
... .when("indel", True)
... .or_missing())
As with case statements, missingness will propagate up through a switch
statement. If we changed the value of ``csq`` to the missing value
``hl.missing(hl.tstr)``, then the result of the switch statement above would also
be missing.
Missingness
===========
In Hail, all expressions can be missing. An expression representing a missing
value of a given type can be generated with the :func:`.null` function, which
takes the type as its single argument.
An example of generating a :class:`.Float64Expression` that is missing is:
>>> hl.missing('float64')
These can be used with conditional statements to set values to missing if they
don't satisfy a condition:
>>> hl.if_else(x > 2.0, x, hl.missing(hl.tfloat))
The Python representation of a missing value is ``None``. For example, if
we define ``cnull`` to be a missing value with type :obj:`.tcall`, calling
the method `is_het` will return ``None`` and not ``False``.
>>> cnull = hl.missing('call')
>>> hl.eval(cnull.is_het())
None
Functions
=========
In addition to the methods exposed on each :class:`.Expression`, Hail also has
numerous functions that can be applied to expressions, which also return an
expression.
Take a look at the :ref:`sec-functions` page for full documentation.