ServiceBackend

class hailtop.batch.backend.ServiceBackend(*args, billing_project=None, bucket=None, remote_tmpdir=None, google_project=None, token=None)

Bases: hailtop.batch.backend.Backend[hailtop.batch_client.client.Batch]

Backend that executes batches on Hail’s Batch Service on Google Cloud.

Examples

>>> service_backend = ServiceBackend(billing_project='my-billing-account', remote_tmpdir='gs://my-bucket/temporary-files/') 
>>> b = Batch(backend=service_backend) 
>>> b.run() 
>>> service_backend.close()

If the Hail configuration parameters batch/billing_project and batch/remote_tmpdir were previously set with hailctl config set, then one may elide the billing_project and remote_tmpdir parameters.

>>> service_backend = ServiceBackend()
>>> b = Batch(backend=service_backend)
>>> b.run() 
>>> service_backend.close()
Parameters

  • billing_project (Optional[str]) – Name of billing project to use.

  • bucket (Optional[str]) – Name of bucket to use. Should not include the gs:// prefix. Cannot be used with remote_tmpdir. Temporary data will be stored in the “/batch” folder of this bucket. This argument is deprecated. Use remote_tmpdir instead.

  • remote_tmpdir (Optional[str]) – Temporary data will be stored in this cloud storage folder. Cannot be used with deprecated argument bucket. Paths should start with one of gs://, hail-az://, or s3://.

  • google_project (Optional[str]) – If specified, the project to use when authenticating with Google Storage. Google Storage is used to transfer serialized values between this computer and the cloud machines that execute Python jobs.

  • token (Optional[str]) – The authorization token to pass to the batch client. Should only be set for user delegation purposes.

Methods

_run

Execute a batch.

validate_file_scheme

rtype

None
_run(batch, dry_run, verbose, delete_scratch_on_exit, wait=True, open=False, disable_progress_bar=False, callback=None, token=None, **backend_kwargs)

Execute a batch.

Warning

This method should not be called directly. Instead, use batch.Batch.run() and pass ServiceBackend specific arguments as key-word arguments.

Parameters

  • batch (Batch) – Batch to execute.

  • dry_run (bool) – If True, don’t execute code.

  • verbose (bool) – If True, print debugging output.

  • delete_scratch_on_exit (bool) – If True, delete temporary directories with intermediate files.

  • wait (bool) – If True, wait for the batch to finish executing before returning.

  • open (bool) – If True, open the UI page for the batch.

  • disable_progress_bar (bool) – If True, disable the progress bar.

  • callback (Optional[str]) – If not None, a URL that will receive at most one POST request after the entire batch completes.

  • token (Optional[str]) – If not None, a string used for idempotency of batch submission.

Return type

Batch

validate_file_scheme(uri)
Return type

None