For Developers

We welcome contributions to gcsfs!

Please file issues and requests on github and we welcome pull requests.

Testing

The testing framework supports using your own GCS-compliant endpoint, by setting the “STORAGE_EMULATOR_HOST” environment variable. If this is not set, then an emulator will be spun up using docker and fake-gcs-server. This emulator has almost all the functionality of real GCS. A small number of tests run differently or are skipped.

If you want to actually test against real GCS, then you should set STORAGE_EMULATOR_HOST to “https://storage.googleapis.com” and also provide appropriate GCSFS_TEST_BUCKET, GCSFS_TEST_VERSIONED_BUCKET (To use for tests that target GCS object versioning, this bucket must have versioning enabled), GCSFS_ZONAL_TEST_BUCKET(To use for testing Rapid storage features), GCSFS_HNS_TEST_BUCKET (To use for testing HNS features) and GCSFS_TEST_PROJECT, as well as setting your default google credentials (or providing them via the fsspec config).

When running tests against a real GCS endpoint, you have two options for test buckets:

  • Provide existing buckets: If you specify buckets that already exist, the test suite will manage objects within them (creating, modifying, and deleting objects as needed). The buckets themselves will not be deleted upon completion. Warning: The test suite will clear the contents of the bucket at the beginning and end of the test run, so be sure to use a bucket that does not contain important data.

  • Let the tests create buckets: If you specify bucket names that do not exist, the test suite will create them for the test run and automatically delete them during final cleanup.

End-to-end Testing CI Pipeline

We have a Cloud Build pipeline for end-to-end tests which includes tests on zonal and regional buckets. When a pull request is created for the main branch, there will be a end-to-end-tests-trigger check in the GitHub checks section.

The pipeline’s behavior depends on the author of the pull request:

  • If the PR is created by an owner or a collaborator, the pipeline will be triggered immediately.

  • If the PR is from an external contributor, an owner or collaborator must add the comment /gcbrun to the PR to trigger the pipeline, until then pipeline would be in failure state.

The pipeline will also be triggered when a new commit is added to the PR. For external contributors, a new /gcbrun comment is required from an owner or collaborator after the new commit. The pipeline can also be manually re-triggered by adding a /gcbrun comment or by using re-run option from Github UI.

The logs from the test run are available in the “details” section of the Checks tab in the pull request.

Release Process

This project uses CalVer for versioning. Releases are generally kept in sync with the fsspec release cycle.

Pre-release steps:

  1. Verify that the CI pipeline is passing on the main branch.

  2. Update docs/source/changelog.rst manually with the changes for the new version. Follow the existing format in that file.

  3. Commit the changelog updates and merge them to the main branch before creating the tag. Ensure your local main branch is up-to-date.

Execution:

Create and push a git tag matching the version pattern. This will trigger the “Release” GitHub Actions workflow, which will automatically create a GitHub release and publish the package to PyPI.

Verification:

  1. Monitor the “Release” workflow in the GitHub Actions tab.

  2. Verify the new version is available on PyPI.