Python Version Compatibility Policy

Hail complies with NumPy’s compatibility policy on Python versions. In particular, Hail officially supports:

  • All minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions.

  • All minor versions of numpy released in the 24 months prior to the project, and at minimum the last three minor versions.

Change Log

Version 0.2.140

  • (#15663) hailtop now supports Python 3.12, 3.13 and 3.14. Support for Python 3.10 and 3.11 is removed. The default Python version for jobs is now 3.12.

  • (#15824) hailtop no longer uses nest_asyncio. Hail’s async work now runs on its own event loop thread. This fixes hailctl batch submit and other clients failing with RuntimeError: Timeout context manager should be used inside a task on Python 3.14, and hailctl batch submit --wait hanging after it finished. hailtop.hail_event_loop.hail_event_loop() is removed.

  • (#15883) User jobs no longer have the CAP_NET_RAW capability, so tools like ping and traceroute no longer work inside jobs.

  • (#15692) hailctl dataproc now creates clusters with Dataproc 3.0 images, which ship Spark 4. Notebooks are served by JupyterLab through the Dataproc component gateway, and hailctl dataproc connect opens the gateway URL in your browser. The --port flag of hailctl dataproc connect is removed. The default machine type is now n2 instead of n1.

  • (#15683) Add N4 machine types for job-private jobs. N4 machines use Hyperdisk volumes, capped at the free IOPS and throughput rates.

  • (#15675) JVM jobs can now declare the Spark version their jar was built against with the optional spark_version field of the job spec. It defaults to 3.5.3, so existing clients are unaffected.

  • (#15768) Add /timing and /job_graph API endpoints that return per-attempt timing data and the job dependency graph of a batch. The job details response now includes job_group_id.

  • (#15888) Job resource charts and the job timing chart now label the time axis with clock time. A toggle switches back to time relative to the job start.

  • (#15730) The job page now shows the requested and actual region of each attempt.

  • (#15738) Job logs in the new UI layout are now colored according to their ANSI color codes.

  • (#15782, #15708) Add new-layout versions of the batch details page and the batch driver dashboard.

  • (#15690) Creating job groups is now idempotent and no longer needs to happen in id order, so clients can safely retry after a lost response.

  • (#15890) Query-on-Batch JVMs now use the serial garbage collector and size themselves for their job’s cores. This fixes drivers being killed for running out of memory without an OutOfMemoryError.

  • (#15773) Fix a bug where users were billed for about a minute of startup time when a job-private VM could not be created because no capacity was available.

  • (#15723) Fix a deadlock between creating and canceling job groups that could make requests fail with a 400 error.

  • (#15620) Fix a bug where inactive users did not see the “account is inactive” error message, e.g. in Jupyter notebooks or the web UI.

  • (#15701) Fix a race condition that crashed the batch driver when it deleted an instance.

Version 0.2.139

  • (#15200) Services now use role-based access control for system-level actions. Users can be granted specific system roles (e.g. sysadmin, billing manager) instead of the previous all-or-nothing developer flag.

  • (#15498) Add a hailctl batch jobs command to query the jobs within a batch, e.g. hailctl batch jobs BATCH_ID --state bad to list failed or errored jobs.

  • (#15409) Billing data can now be exported to CSV from the billing page and a new API endpoint.

  • (#15555) The batch details page now shows billing project information and warns when the billing project is low on or over its limit.

  • (#15588) Add an API endpoint that returns a signed URL for downloading job logs directly from cloud storage, for logs too large to fetch via the standard endpoint.

  • (#15523, #15572, #15574) All queries previously only available through the web UI now have REST /api/ equivalents, and the batch driver API is now documented with OpenAPI.

  • (#15452) Fix timeouts when localizing large input files to a job, significantly speeding up large file downloads.

  • (#15494) Fix fetching logs of a specific container with hailctl, support fetching logs of previous attempts, and add a command to list a job’s attempts.

  • (#15298) Show a clearer error message, with instructions, when submitting a batch while unauthenticated or with an inactive account.

  • (#15362) Improve the error message reported when a job’s docker image cannot be inspected.

  • (#15552) Improve behavior in hosted Jupyter environments: subprocesses now use the running Python interpreter rather than the system one.

Version 0.2.138

  • (#15266) Add the ability to filter jobs by exit code in the Batch UI and API.

  • (#15283) Batch and job status pages now automatically refresh while the batch is still running.

  • (#15286) Show a more helpful error message when a user’s account is inactive.

  • (#15287) Fix a bug causing duplicate job rows to appear in the Batch UI.

  • (#15308) Fix an incorrect --files switch in the hailctl batch submit documentation.

  • (#15343) Fix a bug in hailctl curl where the configured namespace was not respected.

Version 0.2.137

  • (#15199) Upgrade dataproc to image version 2.3.17

  • (#15195) Updates hailgenetics images base to ubuntu:noble-20251013

  • (#15152) For security purposes, hailgenetics docker images no longer contain gcc and g++. Please run hail-apt-get-install gcc g++ within containers if you depend on these compilers.

  • (#15128) New interface for hailctl batch submit.

  • (#15102) Adds slim flag to hailtop.batch.docker.build_python_image to build from non-slim images.

Version 0.2.136

  • (#14980) Update dataproc image version to 2.2.60

  • (#14970) Fix dry-run and docs for dataproc submit

  • (#14918) Upgrade default Python version to 3.11

  • (#14937) Fix a bug that caused randomisation of user-defined tmpdirs

Version 0.2.135

  • (#14927) Worker image and derived docker images upgraded to ubuntu 24.04.

  • (#14925) UI filtering improvements to user table

  • (#14916) Improved logging around JVM initialization failures

  • (#14880) Truncate job names when used as a directory name under /io

  • (#14919) Fixes a bug that made workers seem unresponsive and prevented jobs from being marked complete.

  • (#14912) hailctl batch init now creates buckets in the default region with a 7 day lifecycle policy instead of 30 day.

  • (#14895) Add support for multiple configuration profiles

  • (#14848) Use the default region when a user doesn’t specify a region

  • (#14908) Adds default_region API endpoint

  • (#14901) Fixes bug where remove_tmpdir jobs were not run in the regions the user specified.

  • (#14810) Adds SESSION_MAX_AGE_SECS to configure session timeout duration

  • (#14849) Temporary buckets are initialized with soft delete disabled.

  • (#14876) Batch progress bar now displays accrued cost

  • (#14878)’ Login UI requires ToS acceptance.

  • (#14844) Adds support for n1 machines with T4 GPUs in GCP

  • (#14833) Various openapi documentation additions and fixes.

  • (#14841) Fixed a regression preventing hyphens in usernames.

Version 0.2.132

  • (#14576) Fixed bug where submitting many Python jobs would fail with RecursionError.

Version 0.2.131

  • (#14544) batch.read_input and batch.read_input_group now accept os.PathLike objects as well as strings.

  • (#14328) Job resource usage data can now be retrieved from the Batch API.

Version 0.2.130

  • (#14425) A job’s ‘always run’ state is rendered in the Job and Batch pages. This makes it easier to understand why a job is queued to run when others have failed or been cancelled.

  • (#14437) The billing page now reports users’ spend on the batch service.

Version 0.2.128

  • (#14224) hb.Batch now accepts a default_regions argument which is the default for all jobs in the Batch.

Version 0.2.124

  • (#13681) Fix hailctl batch init and hailctl auth login for new users who have never set up a configuration before.

Version 0.2.123

  • (#13643) Python jobs in Hail Batch that use the default image now support all supported python versions and include the hail python package.

  • (#13614) Fixed a bug that broke the LocalBackend when run inside a Jupyter notebook.

  • (#13200) hailtop.batch will now raise an error by default if a pipeline attempts to read or write files from or two cold storage buckets in GCP.

Version 0.2.122

  • (#13565) Users can now use VEP images from the hailgenetics DockerHub in Hail Batch.

Version 0.2.121

Version 0.2.117

  • (#13007) Memory and storage request strings may now be optionally terminated with a B for bytes.

  • (#13051) Azure Blob Storage https URLs are now supported.

Version 0.2.115

  • (#12731) Introduced hailtop.fs that makes public a filesystem module that works for local fs, gs, s3 and abs. This can be used by import hailtop.fs as hfs.

  • (#12918) Fixed a combinatorial explosion in cancellation calculation in the LocalBackend

  • (#12917) ABS blob URIs in the form of https://<ACCOUNT_NAME>.blob.core.windows.net/<CONTAINER_NAME>/<PATH> are now supported when running in Azure. The hail-az scheme for referencing ABS blobs is now deprecated and will be removed in a future release.

Version 0.2.114

  • (#12780) PythonJobs now handle arguments with resources nested inside dicts and lists.

  • (#12900) Reading data from public blobs is now supported in Azure.

Version 0.2.113

  • (#12780) The LocalBackend now supports always_run jobs. The LocalBackend will no longer immediately error when a job fails, rather now aligns with the ServiceBackend in running all jobs whose parents have succeeded.

  • (#12845) The LocalBackend now sets the working directory for dockerized jobs to the root directory instead of the temp directory. This behavior now matches ServiceBackend jobs.

Version 0.2.111

  • (#12530) Added the ability to update an existing batch with additional jobs by calling Batch.run() more than once. The method Batch.from_batch_id() can be used to construct a Batch from a previously submitted batch.

Version 0.2.110

  • (#12734) PythonJob.call() now immediately errors when supplied arguments are incompatible with the called function instead of erroring only when the job is run.

  • (#12726) PythonJob now supports intermediate file resources the same as BashJob.

  • (#12684) PythonJob now correctly uses the default region when a specific region for the job is not given.

Version 0.2.103

  • Added a new method Job.regions() as well as a configurable parameter to the ServiceBackend to specify which cloud regions a job can run in. The default value is a job can run in any available region.

Version 0.2.89

  • Support passing an authorization token to the ServiceBackend.

Version 0.2.79

  • The bucket parameter in the ServiceBackend has been deprecated. Use remote_tmpdir instead.

Version 0.2.75

  • Fixed a bug introduced in 0.2.74 where large commands were not interpolated correctly

  • Made resource files be represented as an explicit path in the command rather than using environment variables

  • Fixed Backend.close to be idempotent

  • Fixed BatchPoolExecutor to always cancel all batches on errors

Version 0.2.74

  • Large job commands are now written to GCS to avoid Linux argument length and number limitations.

Version 0.2.72

  • Made failed Python Jobs have non-zero exit codes.

Version 0.2.71

  • Added the ability to set values for Job.cpu, Job.memory, Job.storage, and Job.timeout to None

Version 0.2.70

  • Made submitting PythonJob faster when using the ServiceBackend

Version 0.2.69

  • Added the option to specify either remote_tmpdir or bucket when using the ServiceBackend

Version 0.2.68

  • Fixed copying a directory from GCS when using the LocalBackend

  • Fixed writing files to GCS when the bucket name starts with a “g” or an “s”

  • Fixed the error “Argument list too long” when using the LocalBackend

  • Fixed an error where memory is set to None when using the LocalBackend

Version 0.2.66

  • Removed the need for the project argument in Batch() unless you are creating a PythonJob

  • Set the default for Job.memory to be ‘standard’

  • Added the cancel_after_n_failures option to Batch()

  • Fixed executing a job with Job.memory set to ‘lowmem’, ‘standard’, and ‘highmem’ when using the LocalBackend

  • Fixed executing a PythonJob when using the LocalBackend

Version 0.2.65

  • Added PythonJob

  • Added new Job.memory inputs lowmem, standard, and highmem corresponding to ~1Gi/core, ~4Gi/core, and ~7Gi/core respectively.

  • Job.storage is now interpreted as the desired extra storage mounted at /io in addition to the default root filesystem / when using the ServiceBackend. The root filesystem is allocated 5Gi for all jobs except 1.25Gi for 0.25 core jobs and 2.5Gi for 0.5 core jobs.

  • Changed how we bill for storage when using the ServiceBackend by decoupling storage requests from CPU and memory requests.

  • Added new worker types when using the ServiceBackend and automatically select the cheapest worker type based on a job’s CPU and memory requests.

Version 0.2.58

  • Added concatenate and plink_merge functions that use tree aggregation when merging.

  • BatchPoolExecutor now raises an informative error message for a variety of “system” errors, such as missing container images.

Version 0.2.56

  • Fix LocalBackend.run() succeeding when intermediate command fails

Version 0.2.55

  • Attempts are now sorted by attempt time in the Batch Service UI.

Version 0.2.53

  • Implement and document BatchPoolExecutor.

Version 0.2.50

  • Add requester_pays_project as a new parameter on batches.

Version 0.2.43

  • Add support for a user-specified, at-most-once HTTP POST callback when a Batch completes.

Version 0.2.42

  • Fixed the documentation for job memory and storage requests to have default units in bytes.