Changelog

Note: in some releases, there are no changes, because we always guarantee releasing in step with fsspec.

Unreleased

  • sign() now works with credentials that have no private key, such as Compute Engine and GKE Workload Identity credentials. When the credentials cannot sign, the URL is signed through the IAM Credentials signBlob API using the service account’s email and access token; the service account needs the roles/iam.serviceAccountTokenCreator role on itself. service_account_email and access_token can also be passed explicitly, and explicitly passed credentials are used as given (#653).

2026.10.0

New Features

Mutual TLS (mTLS) for HTTP requests when a client certificate is configured. When google-auth is configured to use a client certificate (GOOGLE_API_USE_CLIENT_CERTIFICATE=true, or a certificate config with a workload section) and a default client certificate source exists, GCSFS now presents that certificate on its HTTP session and, for the default googleapis.com universe, defaults to https://storage.mtls.googleapis.com, as google-cloud-storage does. This is needed for access tokens bound to a client certificate (e.g. Agent Identity), which GCS accepts only over mTLS.

  • endpoint_url and STORAGE_EMULATOR_HOST still take precedence over the mTLS endpoint. If session_kwargs provides a connector, it is used as-is and the certificate is not presented.

  • Set GOOGLE_API_USE_MTLS_ENDPOINT=never to keep the regular endpoint (the certificate is still presented).

  • No behavior change when no client certificate is configured.

  • Known limitations: gRPC connections (used for Rapid bucket reads/writes, bucket-type detection, and hierarchical namespace folder operations) do not present the client certificate and stay on the regular endpoint. The certificate is loaded when the HTTP session is created and is not reloaded if it rotates.

(#1080)

Migrate adaptive prefetcher to native fsspec AdaptiveReadaheadCache: The internal gcsfs.prefetcher module has been removed from GCSFS and upstreamed into fsspec>=2026.9.0, and GCSFS now uses cache_type="adaptive" as its default streaming read caching policy. The prefetcher is still enabled by default when cache_type is not set, and max_prefetch_size and concurrency passed to open() still apply. To use a different cache or disable prefetching, pass cache_type explicitly (e.g. cache_type="none"). For configuration options, architecture, and benchmarks, see https://github.com/fsspec/gcsfs/blob/main/docs/source/prefetcher.rst

  • Removed experimental configuration flags USE_EXPERIMENTAL_ADAPTIVE_PREFETCHING and use_experimental_adaptive_prefetching. (Warning) They are now silently ignored: if you used them to turn prefetching off, set cache_type explicitly instead.

  • Bumped minimum fsspec dependency to >=2026.9.0.

(#993)

Bug Fixes & Improvements

  • 401 Invalid Credentials responses are now retried in single-request get_file() downloads. The error body was previously not read before validation, so the retry check never saw the “Invalid Credentials” message. Additionally, 401 responses whose message contains lowercase “invalid” (e.g. invalid_token) now raise HttpError instead of ValueError, so requests that GCSFS retries are retried on them when the message contains “Invalid Credentials”. (#1084)

  • Fixed errors and deadlocks when files are closed from the event loop thread or by the garbage collector. Previously this could raise NotImplementedError or deadlock the fsspec I/O event loop. Files closed in those situations are now torn down on a dedicated background worker thread; errors during these deferred closes are logged rather than raised. ZonalFile read-pool and write-stream teardown now run independently, each bounded by a timeout (default 60 s), so a read-side failure no longer skips write finalization. Files should still be closed explicitly (e.g. with fs.open(...)): a file still open at interpreter shutdown is marked closed without being flushed. (#1009, #1025, #1031, #1040)

  • Fixed paths for objects whose names start with /. Listing and info() results previously dropped the bucket name for such objects (returning /name); they now return bucket//name. (#1044)

  • rm() now raises the original error when a whole delete batch fails, instead of an unrelated TypeError. Aggregation of batch results is also linear instead of quadratic. (#1060)

  • Lower CPU overhead in recursive find() (also used by glob(), du(), and recursive rm()/get()/copy() path expansion): directory entries are now built once per unique directory. In an in-memory benchmark that excludes network time, this step was up to 14x faster on deep hierarchies. (#1046)

  • Opening an existing object for reading no longer calls mimetypes.guess_type() when GCS metadata already provides contentType. (#1059)

  • Rapid (zonal) bucket gRPC reads now report the cache type in request metadata (x-goog-api-client), as HTTP reads already do in the User-Agent. (#1019)

  • Bumped minimum google-cloud-storage dependency to >=3.14.1. (#1039, #1050)

Contributor Tooling: Agent Skills

The repository now includes AI coding-agent skills under _agents/skills/ (https://github.com/fsspec/gcsfs/tree/main/_agents/skills) for performance work on GCSFS:

  • create-microbenchmark: instructions for scaffolding a new microbenchmark for a given method under gcsfs/tests/perf/microbenchmarks. The agent first checks that no benchmark for that method exists yet, and asks for confirmation if a method it calls is already benchmarked. (#1012)

  • autoresearch-profiling: instructions for profile-guided performance optimization. The agent picks a profiler (py-spy, cProfile, or memory_profiler) to find bottlenecks, then runs modify → verify → keep/discard iterations against microbenchmark or subsystem-benchmark metrics. The iteration loop requires the separate autoresearch skill, which is not included in this repository. (#1071)

2026.8.1

  • Default cat_file concurrency to 1 (#1051, fixes #1048). Restores single-request sequential reads without range headers or extra round-trips for small objects (e.g. Zarr, Xarray, Parquet metadata). Callers can still explicitly pass concurrency=... to cat_file for concurrent fetches of large files.

2026.8.0

Adaptive Concurrent Prefetching is now the default read path

Enhanced read path via adaptive concurrent prefetching is now the default in GCSFS. Starting with this version, GCSFS predicts the next byte range an application will read, fetches it in the background across several concurrent HTTP requests, and keeps the bytes in memory before next read() is called. Network round-trips overlap with application compute instead of blocking calls where compute has to wait for data to be fetched. We are also enabling read concurrency, so a single reader is no longer limited by the bandwidth of a single HTTP connection.

GCSFS prefetcher adapts to workload read IO patterns. It tracks the rolling average of recent read sizes and scales the prefetch window linearly with the detected sequential streak, rather than using a fixed block size or exponential doubling. This is inline to what modern Linux kernels will do to balance prefetch and memory footprint. When the pattern turns random read i.e. we are not able to leverage the prefetched buffer to answer the next read() call, it drains the buffer to zero, so that random-access workloads pay no bandwidth or memory penalty.

Why this matters for AI/ML workloads

  • Model loading and checkpoint restore are typically large sequential reads - and prefetcher shines there. In our benchmarking, a single-stream sequential throughput improved from 23.69 MB/s to 658.71 MB/s for 1 MB I/O, and from 156 MB/s to 736 MB/s for 16 MB I/O.

  • Training data pipelines stay fed. Parquet and sharded dataset reads issue small-to-medium sequential ranges that previously suffered from low throughput, but now achieve significantly more; at 16 MB I/O throughput rises from 150 MB/s to 730 MB/s. Reducing the wait time for data loading improves accelerator goodput(amount of time accelerator is utilised for training than waiting).

  • Multi-worker dataloader scaling. The prefetcher manufactures its own parallelism per worker instead of relying on process count alone.

  • Accelerate the throughput even further with Rapid Buckets. With Rapid Buckets single node throughput reaches 21 GiB/s with 16-process sequentially reading at 16 MiB I/O compared to standard buckets with 48processes.

Adaptive prefetcher (cache_type=”adaptive”) is enabled by default when cache_type is not explicitly set, using DEFAULT_GCSFS_CONCURRENCY=4 for both Standard and Rapid buckets. Users can also explicitly specify cache_type="adaptive" or choose an alternative caching strategy (e.g., cache_type="readahead", cache_type="readahead_chunked", or cache_type="none").

(Warning) Impact on memory: Prefetching trades memory for throughput. Peak memory rises from ~170 MB to 600 MB on single-stream reads for 16 MB IO size and varies with requested IO sizes, and would be materially more under high process counts. Please ensure that application memory limits accordingly to use prefetcher without any Out of Memory(OOM) issues. To put a hard limit, you can configure cache_options={"max_prefetch_size": <bytes>}.

For details on architecture, tuning, full benchmark tables, along with known limitations please refer to : https://github.com/fsspec/gcsfs/blob/main/docs/source/prefetcher.rst

(#795, #805, #818, #877)

Bug Fixes & Improvements

  • Zero-cost local backward seeks in PrefetchConsumer - Parquet footer and ZIP directory reads are served from the existing buffer instead of re-issuing a network request. (#930)

  • Concurrent downloads cap task count against a minimum chunk size, removing per-task overhead on small ranges. (#926)

  • Generation consistency across parallel fetches, guaranteeing every chunk comes from the same object version. (#921)

  • Fixed silent truncation on short reads in zonal bucket downloads. (#920)

  • Zero-copy read and write paths via memoryview, cutting CPU and transient memory in the hot path. (#840, #907, #928)

  • Graceful fallback where ctypes.pythonapi is unavailable. (#938)

2026.7.0

  • fix(macrobenchmarks): fix cloud build machine type networkPerformanceConfig error (#951)

  • fix(macrobenchmarks): use random optimizer moments, not zeros, in CPU sim (#949)

  • test(macrobench): use default DDP timeout in llama CPU simulation (#947)

  • Update release process documentation (#945)

  • Add permissions and path filters to release-on-merge workflow (#944)

  • fix(cleanup): retry GKE cluster deletion on failure (#943)

  • fix(macrobenchmarks): remove HNS validation and external model bucket check (#942)

  • fix: seed timer at training start to avoid AttributeError (#941)

  • Automate monthly release process (#916)

  • refactor(macrobench): share helm args and update cloudbuild config (#940)

  • test(perf): add microbenchmarks for put operations (#939)

  • Update benchmarking tables schema and values under ReadAhead Cache (#934)

  • support environment where ctypes.pythonapi is not available (#938)

  • Enhance CPU simulator logging and throughput metrics (#937)

  • ci: isolate macrobenchmark GKE cluster in a dedicated VPC network (#933)

  • macrobenchmarks: parameterize macrobenchmarks (#936)

  • macrobenchmarks: add checkpoint seeding and reduce emulator ranks to 4 (#935)

  • Implement zero-cost local backward seeks in PrefetchConsumer (#930)

  • Add cache_type in user-agent fo http client (#908)

  • Fix root cache invalidation (#931)

  • Use memoryview to enable zero-copy in _pipe_file (#928)

  • Fix ValueError message for invalid access in GCSFileSystem (#932)

  • feat(macrobench): add BigQuery ingestion pipeline with dynamic schema evolution (#915)

  • Fix silent truncation on short reads in zonal bucket downloads (#920)

  • Fix generation threading in gcsfs url and concurrent fetches (#921)

  • macrobenchmarks: Parameterize training strategy and simulated step compute; generalize requirements (#927)

  • Implement makedirs for HNS buckets (#906)

  • docs: update benchmarking sections with single-threaded data (#925)

  • Optimize concurrent downloads by capping task count based on minimum chunk size (#926)

  • ci: add macrobenchmark infrastructure unit tests (#924)

  • prefetcher docs: update no cache benchmark tables for Standard buckets (#922)

  • perf(benchmarks): increase pipe benchmark size and exclude setup overhead from write benchmark (#917)

  • feat(macrobench): add metrics calculation and summary assembly engine (#913)

  • feat(macrobench): add metrics extraction and storage engine (#912)

  • feat(macrobench): add GKE orchestration scripts and Cloud Build pipeline (#914)

  • feat(macrobench): add Llama 3.1 8B CPU simulator workload (hf-pytorch-lightning-cpu) (#911)

  • Update prefetcher.rst with Rapid Buckets performance numbers (#909)

  • fix: force schema recreation for staging external table in ingestion pipeline (#910)

  • Implement zero-copy in write path of standard bucket (#907)

  • updated latest fsspec version in pyproject.toml (#903)

  • chore: update build configurations and database schema (#904)

  • Remove GCE_METADATA_MTLS_MODE export from e2e pipeline (#905)

  • chore(benchmarks): make create-vm wait for cleanup-leaked-resources (#902)

  • Remove 48 process rapid read benchmark (#901)

  • Fix order-independent assertion in test_read_block_zb (#897)

2026.6.0

  • check finalized state to optimize MRD pool initialization (#896)

  • fix streaming upload alignment for non-final chunks (#894)

  • optimize _rm batchsize and speed up tests (#893)

  • optimize E2E tests: parallelize resource creation and enable parallel test execution (#892)

  • fix CPU and Memory monitoring aggregation logic (#891)

  • update Codecov config (#890)

  • remove mrd_supports_multi_request and assume it’s always true (#884)

  • fix multi-process benchmark child process crash handling (#883)

  • fix ZonalFile generation parameter overwriting (#882)

  • fix ResourceMonitor thread shutdown latency affecting benchmark durations (#881)

  • fix generation parameter not forwarded to GCSFile in open() (#880)

  • route GCSMap through dynamically resolved GCSFileSystem (#879)

  • fix multiple _info() calls in _process_limits_to_offset_and_length (#878)

  • introduce async API in Prefetcher and integrate it with disk reads (#877)

  • clean up zonal/HNS test markers and GHA exports (#876)

  • cache metadata details from MRDPoolCache info lookup to avoid redundant network calls (#874)

  • add a comprehensive performance micro-benchmark suite targeting fs.glob() (#873)

  • refactor directory cache updates and optimize caching logic (#870)

  • create flat bucket dynamically (#869)

  • enable exp reads (#868)

  • clean up duplicate tests (#867)

  • block release pipeline on Google Cloud Build E2E integration tests (#866)

  • use conda-forge for docs to avoid Anaconda SSL error (#865)

  • add log for download range for Rapid Bucket (#863)

  • update read performance microbenchmark config to use 1 file (#861)

  • add open microbenchmark (#860)

  • fix multiprocessing emulator _get_bucket_type calls (#859)

  • merge HNS and flat bucket deletion routing in rm (#858)

  • fix teardown order in close_resources() (#857)

  • refcount in-flight MRDs in MRDPool (#856)

  • fix off-by-one error in upload_chunk shortfall recursion (#855)

  • fix concurrent MRD reuse by tracking in-flight MRDs on close (#854)

  • fix swapped arguments in _process_object inside inventory_report.py (#853)

  • optimize benchmarks pipeline, fix quota leaks and handle manual builds (#851)

  • update default value of mrd_pool_size in micro benchmark to None and add pool parameters to benchmark schema (#850)

  • fix BigQuery ingestion errors and improve Cloud Build robustness (#849)

  • use pytest markers to gate Rapid/HNS tests (#848)

  • fix Storage Control endpoint resolution for TPC (#847)

  • add mrd pool cache (#846)

  • run tests concurrently in CI (#845)

  • document intentional non-caching and fallback behavior for UNKNOWN bucket layouts (#844)

  • improve test speed and emulator compatibility (#843)

  • add Pipe function in microbenchmark (#842)

  • add support for testing against fsspec HEAD (#841)

  • implement zero-copy optimization for single-read operations (#840)

  • pass trailing / to createFolder as per documentation (#839)

  • add multi-threaded fixed-duration read microbenchmarks (#838)

  • remove fsspec exclusion following prefix fix (#837)

  • add mixed pattern in gcsfs microbenchmarks (#822)

  • refactor prefetcher code (#818)

2026.5.0

  • add pypi environment to release workflow (#836)

  • fix HttpError message formatting and handle None content in validate_response (#835)

  • adjust fsspec dependency version constraint (#834)

  • add support for partial prefixes in find method for HNS buckets (#831)

  • fix issue with special characters in rm method (#831)

  • update zonal doc (#828)

  • add workflow to automate PyPI package publishing on release (#824)

  • enable branch-wise tracking in benchmarks (#819)

  • update the benchmark config, and fix the block size propagation (#808)

  • integrate prefetcher engine with zonal buckets (#805)

2026.4.0

  • add fallback when missing __version__ (#826)

  • use project ID in gRPC, if set (#825)

  • fix location lookup in put_file (#821)

  • error message formatting in rm_files (#820)

  • requestor pays in ExtendedGcs (#817)

  • logging error->warning in storage layout API (#816)

  • fix prefetcher docs (#815)

  • cleanup on failure in cloud build benchmarks (#812)

  • benchmark updates (#812, 811, 810, 802)

  • migrate setuptools to hatch (#809)

  • retreive coroutine exceptions (#807)

  • honour NO_GCE_CHECK (#803)

  • update README (#801)

  • use moveTo in standard buckets (#800)

  • typos (#798, 796, 794)

  • prefetcher for standard buckets (#795)

2026.3.0

Support for Rapid and hierarchical buckets is moved from opt-in to default. The default implementation is now ExtendedGcsFileSystem. This still defers to GCSFileSystem for operations on “normal” buckets (i.e., not zonal, rapid, hierarchical). Set GCSFS_EXPERIMENTAL_ZB_HNS_SUPPORT to disable.

  • docs for zonal/rapid storage support (#792, 788, 781)

  • test/ci/cov fixes (#785, 772, 766, 765, 764)

  • fix zonalfile writes to track byte count (#777)

  • benchmarks (#775, 770, 767, 762)

  • fix folder rename race (HNS) (#771)

  • create zonal buckets (#769)

  • graceful close (#763)

  • list: pass only valid kwargs (#759)

  • make mv_file atomic (#758)

2026.2.0

  • New readahead-chunks cacher (#754, 750)

  • test coverage (#752)

  • info() integration tests for hierarchical (#749)

  • more benchmark work (#748, 745)

  • zonal file downloads (#744)

  • rm for hierarchical (#742)

2026.1.0

New opt-in support for hierarchical buckets.

  • Support alternative GCP Universes (#732)

  • overrides for HNS buckets (#731, 730, 727

  • CI for HNS and zonal buckets (#729, 721, 719)

  • Zonal bucket write mode (#726)

  • CI auth fixes (#720, 725

  • Microbenchmarks framework (#722)

  • capture generation value after upload (#718)

2025.12.0

  • Fix CI when run against rela SGC buckets (#71)

  • Run extended rests when env var is set (#712)

  • Support py3.14 and drop 3.9 (#709)

  • Introduce ExtendedGcsFileSystem for Zonal Bucket gRPC Read Path (#707)

2025.10.0

  • fix info() performance regression (#705)

  • add CoC (#703)

  • mkdir should not create bucket by default (#701)

  • add anonymous tracker to docs (#700)

2025.9.0

  • fix slow ls iterations (#697)

  • Ensure right error type for get() on nonexistent (#695)

2025.7.0

  • acknowledge Anaconda support (#691)

  • less refreshing for CI (#690)

2025.5.1

  • Fix token timezone comparison (#683, 688)

2025.5.0

  • Avoid deprecated utcnow (#680)

  • Add support for specifying Cloud KMS keys when creating files (#679)

  • Yet another fix for isdir (#676)

  • Create warning for appending mode ‘a’ operations (#675)

  • add userProject to batch deletion query (#673)

2025.3.2

no changes

2025.3.1

  • Fix find with path not ending with “/” (#668)

  • remove “beta” note from doc (#666)

  • don’t check expiry of creds that don’t expire (#665)

2025.3.0

  • Improvements for credentials refresh under high load (#658)

2025.2.0

  • guess upload file MIME types (#655)

  • better shutdown cleanup (#657)

2024.12.0

  • Exclusive write (#651)

  • Avoid IndexError on integer seconds (#649)

  • note on non-posixness (#648)

  • handle chache_timeout=0 (#646)

2024.10.0

  • Remove race condition in credentials (#643)

  • fix md5 hash order logic (#640)

2024.9.0

  • In case error in a pure string (#631)

2024.6.1

no changes

2024.6.0

  • Add seek(0) to request data to prevent issues on retries (#624)

2024.5.0

  • swap order of “gcs”, “gs” protocols (#620)

  • fix get_file for relative lpath (#618)

2024.3.1

  • fix expiration= for sign() (#613)

  • do populate dircache in ls() (#612)

  • allow passing extra options to mkdir (#610)

  • credentials docs (#609)

  • retry in bulk rm (#608)

  • clean up loop on close (#606)

2024.2.0

  • doc for passing tokens (#603)

2023.12.2

no changes

2023.12.1

no changes

2023.12.0

  • use same version when paginating list (#591)

  • fix double asterisk glob test (#589)

2023.10.0

  • Fix for transactions of small files (#586)

2023.9.2

  • CI updates (#582)

2023.9.1

  • small fixes following #573 (#578)

2023.9.0

  • bulk operations edge cases (#576, 572)

  • inventory report based file listing (#573)

  • pickle HttpError (#571)

  • avoid warnings (#569)

  • maxdepth in find() (#566)

  • invalidate dircache (#564)

  • standard metadata field names (#563)

  • performance of building cache in find() (#561)

2023.6.0

  • allow raw/session token for auth (#554)

  • fix listings_expiry_time kwargs (#551)

  • allow setting fixed metadata on put/pipe (#550)

2023.5.0

  • Allow emulator host without protocol (#548)

  • Prevent upload retry from closing the file being sent (#540)

2023.4.0

No changes

2023.3.0

  • Don’t let find() mess up dircache (#531)

  • Drop py3.7 (#529)

  • Update docs (#528)

  • Make times UTC (#527)

  • Use BytesIO for large bodies (#525)

  • Fix: Don’t append generation when it is absent (#523)

  • get/put/cp consistency tests (#521)

2023.1.0

  • Support create time (#516, 518)

  • defer async session creation (#513, 514)

  • support listing of file versions (#509)

  • fix sign following versioned split protocol (#513)

2022.11.0

  • implement object versioning (#504)

2022.10.0

  • bump fsspec to 2022.10.0 (#503)

2022.8.1

  • don’t install prerelease aiohttp (#490)

2022.7.1

  • Try cloud auth by default (#479)

2022.5.0

  • invalidate listings cache for simple put/pipe (#474)

  • conform _mkdir and _cat_file to upstream (#471)

2022.3.0

(note that this release happened in 2022.4, but we label as 2022.3 to match fsspec)

  • bucket exists workaround (#464)

  • dirmarkers (#459)

  • check connection (#457)

  • browser connection now uses local server (#456)

  • bucket location (#455)

  • ensure auth is closed (#452)

2022.02.0

  • fix list_buckets without cache (#449)

  • drop py36 (#445)

2022.01.0

  • update refname for versions (#442)

2021.11.1

  • don’t touch cache when doing find with a prefix (#437)

2021.11.0

  • move to fsspec org

  • add support for google fixed_key_metadata (#429)

  • deprecate content_encoding parameter of setxattrs method (#429)

  • use emulator for resting instead of vcrpy (#424)

2021.10.1

  • url signing (#411)

  • default callback (#422)

2021.10.0

  • min version for decorator

  • default callback in get (#422)

2021.09.0

  • correctly recognise 404 (#419)

  • fix for .details due to upstream (#417)

  • callbacks in get/put (#416)

  • “%” in paths (#415)

2021.08.1

  • don’t retry 404s (#406)

2021.07.0

  • fix find/glob with a prefix (#399)

2021.06.1

  • kwargs to aiohttpClient session

  • graceful timeout when disconnecting at finalise (#397)

2021.06.0

  • negative ranges in cat_file (#394)

2021.05.0

  • no credentials bug fix (#390)

  • use googleapis.com (#388)

  • more retries (#387, 385, 380)

  • Code cleanup (#381)

  • license to match stated one (#378)

  • deps updated (#376)

Version 2021.04.0

  • switch to calver and fsspec pin

Version 0.8.0

  • keep up with fsspec 0.9.0 async

  • one-shot find

  • consistency checkers

  • retries for intermittent issues

  • timeouts

  • partial cat

  • http error status

  • CI to GHA

Version 0.7.0

  • async operations via aiohttp

Version 0.6.0

  • API-breaking: Changed requester-pays handling for GCSFileSystem.

    The user_project keyword has been removed, and has been replaced with the requester_pays keyword. If you’re working with a requester_pays bucket you will need to explicitly pass requester_pays-True. This will include your project ID in requests made to GCS.

Version 0.5.3

  • GCSFileSystem now validates that the project provided, if any, matches the Google default project when using token-'google_default' to authenticate (PR #219).

  • Fixed bug in GCSFileSystem.cat on objects in requester-pays buckets (PR #217).

Version 0.5.2

  • Fixed bug in user_project fallback for default Google authentication (PR #213)

Version 0.5.1

  • user_project now falls back to the project if provided (PR #208)

Version 0.5.0

  • Added the ability to make requester-pays requests with the user_project parameter (PR #206)

Version 0.4.0

  • Improved performance when serializing filesystem objects (PR #182)

  • Fixed authorization errors when using gcsfs within multithreaded code (PR #183, PR #192)

  • Added contributing instructions (PR #185)

  • Improved performance for gcsfs.GCSFileSystem.info() (PR #187)

  • Fixed bug in gcsfs.GCSFileSystem.info() raising an error (PR #190)