Python API

This is the API documentation for Batch, and provides detailed information on the Python programming interface.

Use import hailtop.batch to access this functionality.

Batches

A Batch is an object that represents the set of jobs to run and the order or dependencies between the jobs. Each Job has an image in which to execute commands and settings for storage, memory, and CPU. A BashJob is a subclass of Job that runs bash commands while a PythonJob executes Python functions.

batch.Batch

Object representing the distributed acyclic graph (DAG) of jobs to run.

job.Job

Object representing a single job to execute.

job.BashJob

Object representing a single bash job to execute.

job.PythonJob

Object representing a single Python job to execute.

Resources

A Resource is an abstract class that represents files in a Batch and has two subtypes: ResourceFile and ResourceGroup.

A single file is represented by a ResourceFile which has two subtypes: InputResourceFile and JobResourceFile. An InputResourceFile is used to specify files that are inputs to a Batch. These files are not generated as outputs from a Job. Likewise, a JobResourceFile is a file that is produced by a job. JobResourceFiles generated by one job can be used in subsequent job, creating a dependency between the jobs.

A ResourceGroup represents a collection of files that should be treated as one unit. All files share a common root, but each file has its own extension.

A PythonResult stores the output from running a PythonJob.

resource.Resource

Abstract class for resources.

resource.ResourceFile

Class representing a single file resource.

resource.InputResourceFile

Class representing a resource from an input file.

resource.JobResourceFile

Class representing an intermediate file from a job.

resource.ResourceGroup

Class representing a mapping of identifiers to a resource file.

resource.PythonResult

Class representing a result from a Python job.

Batch Pool Executor

A BatchPoolExecutor provides roughly the same interface as the Python standard library’s concurrent.futures.Executor. It facilitates executing arbitrary Python functions in the cloud.

batch_pool_executor.BatchPoolExecutor

An executor which executes Python functions in the cloud.

batch_pool_executor.BatchPoolFuture

Backends

A Backend is an abstract class that can execute a Batch. Currently, there are two types of backends: LocalBackend and ServiceBackend. The local backend executes a batch on your local computer by running a shell script. The service backend executes a batch on Google Compute Engine VMs operated by the Hail team (Batch Service). You can access the UI for the Batch Service at https://batch.hail.is.

backend.RunningBatchType

The type of value returned by Backend._run().

backend.Backend

Abstract class for backends.

backend.LocalBackend

Backend that executes batches on a local computer.

backend.ServiceBackend

Backend that executes batches on Hail’s Batch Service on Google Cloud.

Utilities

docker.build_python_image

Build a new Python image with dill and the specified pip packages installed.

utils.concatenate

Concatenate files using tree aggregation.

utils.plink_merge

Merge binary PLINK files using tree aggregation.