ServiceBackend

class hailtop.batch.backend.ServiceBackend(*args, billing_project=None, bucket=None, remote_tmpdir=None, google_project=None, token=None, regions=None, gcs_requester_pays_configuration=None, gcs_bucket_allow_list=None)

Bases: Backend[Batch]

Backend that executes batches on Hail’s Batch Service on Google Cloud.

Examples

Create and use a backend that bills to the Hail Batch billing project named “my-billing-account” and stores temporary intermediate files in “gs://my-bucket/temporary-files”.

>>> import hailtop.batch as hb
>>> service_backend = hb.ServiceBackend(
...     billing_project='my-billing-account',
...     remote_tmpdir='gs://my-bucket/temporary-files/'
... )  
>>> b = hb.Batch(backend=service_backend)  
>>> j = b.new_job()  
>>> j.command('echo hello world!')  
>>> b.run() 

Same as above, but set the billing project and temporary intermediate folders via a configuration file:

cat >my-batch-script.py >>EOF
import hailtop.batch as hb
b = hb.Batch(backend=ServiceBackend())
j = b.new_job()
j.command('echo hello world!')
b.run()
EOF
hailctl config set batch/billing_project my-billing-account
hailctl config set batch/remote_tmpdir gs://my-bucket/temporary-files/
python3 my-batch-script.py

Same as above, but also specify the use of the ServiceBackend via configuration file:

cat >my-batch-script.py >>EOF
import hailtop.batch as hb
b = hb.Batch()
j = b.new_job()
j.command('echo hello world!')
b.run()
EOF
hailctl config set batch/billing_project my-billing-account
hailctl config set batch/remote_tmpdir gs://my-bucket/temporary-files/
hailctl config set batch/backend service
python3 my-batch-script.py

Create a backend which stores temporary intermediate files in “https://my-account.blob.core.windows.net/my-container/tempdir”.

>>> service_backend = hb.ServiceBackend(
...     billing_project='my-billing-account',
...     remote_tmpdir='https://my-account.blob.core.windows.net/my-container/tempdir'
... )  

Require all jobs in all batches in this backend to execute in us-central1:

>>> b = hb.Batch(backend=hb.ServiceBackend(regions=['us-central1']))

Same as above, but using a configuration file:

hailctl config set batch/regions us-central1
python3 my-batch-script.py

Same as above, but using the HAIL_BATCH_REGIONS environment variable:

export HAIL_BATCH_REGIONS=us-central1
python3 my-batch-script.py

Permit jobs to execute in either us-central1 or us-east1:

>>> b = hb.Batch(backend=hb.ServiceBackend(regions=['us-central1', 'us-east1']))

Same as above, but using a configuration file:

hailctl config set batch/regions us-central1,us-east1

Allow reading or writing to buckets even though they are “cold” storage:

>>> b = hb.Batch(
...     backend=hb.ServiceBackend(
...         gcs_bucket_allow_list=['cold-bucket', 'cold-bucket2'],
...     ),
... )

Parameters:

billing_project (Optional[str]) – Name of billing project to use.
bucket (Optional[str]) – This argument is deprecated. Use remote_tmpdir instead.
remote_tmpdir (Optional[str]) – Temporary data will be stored in this cloud storage folder.
google_project (Optional[str]) – This argument is deprecated. Use gcs_requester_pays_configuration instead.
gcs_requester_pays_configuration (either str or tuple of str and list of str, optional) – If a string is provided, configure the Google Cloud Storage file system to bill usage to the project identified by that string. If a tuple is provided, configure the Google Cloud Storage file system to bill usage to the specified project for buckets specified in the list.
token (Optional[str]) – The authorization token to pass to the batch client. Should only be set for user delegation purposes.
regions (Optional[List[str]]) – Cloud regions in which jobs may run. ServiceBackend.ANY_REGION indicates jobs may run in any region. If unspecified or None, the batch/regions Hail configuration variable is consulted. See examples above. If none of these variables are set, then jobs run in the region the Batch instance resides in (“us-central1” for the Hail team-maintained instance). ServiceBackend.default_region() returns the default region. ServiceBackend.supported_regions() lists the available regions.
gcs_bucket_allow_list (Optional[List[str]]) – A list of buckets that the ServiceBackend should be permitted to read from or write to, even if their default policy is to use “cold” storage.

Attributes

ANY_REGION

A special value that indicates a job may run in any region.

Methods

`_async_run`	Execute a batch.
`default_region`	Get the default cloud region
`supported_regions`	Get the supported cloud regions

ANY_REGION: ClassVar[List[str]] = ['any_region']: A special value that indicates a job may run in any region.

async _async_run(batch, dry_run, verbose, delete_scratch_on_exit, wait=True, open=False, disable_progress_bar=False, callback=None, token=None, **backend_kwargs)

Execute a batch.

Warning

This method should not be called directly. Instead, use batch.Batch.run() and pass ServiceBackend specific arguments as key-word arguments.

Parameters:

batch (Batch) – Batch to execute.
dry_run (bool) – If True, don’t execute code.
verbose (bool) – If True, print debugging output.
delete_scratch_on_exit (bool) – If True, delete temporary directories with intermediate files.
wait (bool) – If True, wait for the batch to finish executing before returning.
open (bool) – If True, open the UI page for the batch.
disable_progress_bar (bool) – If True, disable the progress bar.
callback (Optional[str]) – If not None, a URL that will receive at most one POST request after the entire batch completes.
token (Optional[str]) – If not None, a string used for idempotency of batch submission.

Return type:

Optional[Batch]

static default_region()

Get the default cloud region

This value is “us-central1” for the Hail Team maintained Batch instance.

Examples

>>> region = ServiceBackend.default_region()

Returns:: The default region jobs run in when no regions are specified

static supported_regions()

Get the supported cloud regions

Examples

>>> regions = ServiceBackend.supported_regions()

Returns:: A list of the supported cloud regions