.. _sec-api: ========== Python API ========== This is the API documentation for Batch, and provides detailed information on the Python programming interface. Use ``import hailtop.batch`` to access this functionality. .. currentmodule:: hailtop.batch Batches ~~~~~~~ A :class:`.Batch` is an object that represents the set of jobs to run and the order or dependencies between the jobs. Each :class:`.Job` has an image in which to execute commands and settings for storage, memory, and CPU. A :class:`.BashJob` is a subclass of :class:`.Job` that runs bash commands while a :class:`.PythonJob` executes Python functions. .. autosummary:: :toctree: api/batch/ :nosignatures: :template: class.rst batch.Batch job.Job job.BashJob job.PythonJob Resources ~~~~~~~~~ A :class:`.Resource` is an abstract class that represents files in a :class:`.Batch` and has two subtypes: :class:`.ResourceFile` and :class:`.ResourceGroup`. A single file is represented by a :class:`.ResourceFile` which has two subtypes: :class:`.InputResourceFile` and :class:`.JobResourceFile`. An InputResourceFile is used to specify files that are inputs to a :class:`.Batch`. These files are not generated as outputs from a :class:`.Job`. Likewise, a JobResourceFile is a file that is produced by a job. JobResourceFiles generated by one job can be used in subsequent job, creating a dependency between the jobs. A :class:`.ResourceGroup` represents a collection of files that should be treated as one unit. All files share a common root, but each file has its own extension. A :class:`.PythonResult` stores the output from running a :class:`.PythonJob`. .. autosummary:: :toctree: api/resource/ :nosignatures: :template: class.rst resource.Resource resource.ResourceFile resource.InputResourceFile resource.JobResourceFile resource.ResourceGroup resource.PythonResult Batch Pool Executor ~~~~~~~~~~~~~~~~~~~ A :class:`.BatchPoolExecutor` provides roughly the same interface as the Python standard library's :class:`.concurrent.futures.Executor`. It facilitates executing arbitrary Python functions in the cloud. .. autosummary:: :toctree: api/batch_pool_executor/ :nosignatures: :template: class.rst batch_pool_executor.BatchPoolExecutor batch_pool_executor.BatchPoolFuture Backends ~~~~~~~~ A :class:`.Backend` is an abstract class that can execute a :class:`.Batch`. Currently, there are two types of backends: :class:`.LocalBackend` and :class:`.ServiceBackend`. The local backend executes a batch on your local computer by running a shell script. The service backend executes a batch on Google Compute Engine VMs operated by the Hail team (:ref:`Batch Service `). You can access the UI for the Batch Service at ``__. .. autosummary:: :toctree: api/backend/ :nosignatures: :template: class.rst backend.RunningBatchType backend.Backend backend.LocalBackend backend.ServiceBackend Utilities ~~~~~~~~~ .. autosummary:: :toctree: api/utils/ :nosignatures: docker.build_python_image utils.concatenate utils.plink_merge