> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tornadoapi.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Google Cloud Storage
> Configure a GCS bucket as your Tornado delivery destination: bucket, service account, bucket-scoped IAM role, and how to verify it for real
## Overview
Tornado can deliver completed downloads straight into a Google Cloud Storage bucket you own. Authentication is a service account key: you create the service account in your own project, grant it a role on the bucket, download its JSON key, and paste that key into the dashboard.
The dashboard asks for exactly three values: the bucket, the project ID, and the whole service account key file as one value.
This page covers the **dashboard** path (the onboarding wizard and **Settings › Default storage destination**), which writes an organization-level configuration through `POST /settings/storage`. Credentials are stored in Azure Key Vault and are returned masked when the configuration is read back.
The dashboard route is not the same contract as the per-API-key endpoint
[`POST /user/gcs`](/features/custom-storage#google-cloud-storage). That
endpoint names the credentials field `service_account_json` and additionally
accepts `folder_prefix` and `base_folder`. The dashboard route names the field
`credentials_json` and accepts **three fields only**. Do not copy a payload
from one to the other.
## Current limitations
Read these before you create anything in Google Cloud. Both are properties of the GCS route as it exists today, not Google restrictions.
**"Test connection" does not contact Google.** For GCS there is no probe
object, no API call, and no credential check of any kind. The test loads the
submitted configuration, confirms that `bucket`, `project_id` and
`credentials_json` are present and non-empty, and returns:
```json theme={null}
{
"ok": true,
"verified": false,
"latency_ms": 0,
"note": "credentials look structurally valid — live round-trip test is not yet supported for provider 'gcs'"
}
```
The wizard renders that as a **green** banner reading *"Credentials accepted
and saved, verified on your first delivery."* While the request is in flight
the button reads *"Writing test object…"*, which is the shared wizard label:
for GCS nothing is written anywhere.
A green result here proves that you filled in three boxes. It does not parse
the JSON, and it says nothing about the key, the role, the project or the
bucket. Every one of those failures surfaces later, on a real job. Verify with
[a real delivery](#verify-it-for-real) before you consider the setup done.
The S3 branch of the same endpoint was recently extended to write, read back,
open and abort a multipart upload, and delete. **None of that applies here.**
If you have read the [AWS S3](/storage/aws-s3) or
[Cloudflare R2](/storage/cloudflare-r2) page, discard what it says about the
probe: for GCS no packet leaves Tornado.
**The bucket must already exist.** Tornado never creates it, and because the
connection test makes no call, a bucket that does not exist is accepted at
configuration time and fails on the first job.
## Before you start
You need a Google Cloud project and, within it, permission to create a bucket, grant IAM on that bucket, and create a service account key.
**Service account key creation may be blocked by organization policy.** The
`constraints/iam.disableServiceAccountKeyCreation` constraint prevents key
creation in a project, and Google enforces it **by default on projects created
after 3 May 2024**. If key creation fails with a policy error, an organization
administrator must grant an exception for the project before you can continue.
Tornado authenticates with a service account key only: Workload Identity
Federation, attached service accounts and impersonation are not supported on
this route.
## 1. Create the bucket
In the [Google Cloud console](https://console.cloud.google.com/storage/browser),
go to **Cloud Storage › Buckets**, then select **Create**. Console layouts
change; if the entry is not where you expect it, search for "Cloud Storage"
from the top search bar.
Bucket names are globally unique across all of Google Cloud. Tornado
additionally validates the name before every upload: 3 to 63 characters,
lowercase letters, digits, hyphens and periods only, starting and ending
with a letter or digit, no consecutive periods, and not shaped like an IP
address. A name Google accepts but Tornado rejects will fail at job time,
not at save time.
Pick a location type and location. See
[Choosing a location](#choosing-a-location) below.
Tornado authorises entirely through IAM and never reads or writes object
ACLs, so uniform bucket-level access is the right setting and is what the
rest of this page assumes. Note that uniform bucket-level access cannot be
turned off again once it has been active on a bucket for 90 consecutive
days.
Or from the command line:
```bash theme={null}
gcloud storage buckets create gs://tornado-downloads \
--project=my-gcp-project \
--location=europe-west1 \
--uniform-bucket-level-access
```
### Choosing a location
Google offers regions, dual-regions and multi-regions. For delivery buckets a single **region** is almost always correct:
* A region has the lowest storage price and no replication charges. Dual-region and multi-region buckets add a replication charge **on every write**, and Tornado writes large media objects, so the difference is not marginal.
* Redundancy across regions buys you nothing here. Delivered files are reproducible: if a bucket becomes unavailable you re-run the job.
For which region, colocate with the traffic that matters most to you. Each file is written once but may be read many times, so the region closest to whoever consumes the files is usually the right choice. If upload throughput matters more than read latency, the `latency_ms` reported by the connection test is a fair proxy for how far your bucket is from our workers.
## 2. Create the service account
Go to **IAM & Admin › Service Accounts** in the
[console](https://console.cloud.google.com/iam-admin/serviceaccounts), select
your project, then select **Create service account**.
Something like `tornado-delivery`. Use one service account per delivery
destination so you can revoke it without affecting anything else.
The creation flow offers an optional step titled along the lines of "Grant
this service account access to project". **Skip it.** Granting a role there
applies it to every bucket in the project, present and future. You will
grant the role on one bucket in the next step.
It looks like
`tornado-delivery@my-gcp-project.iam.gserviceaccount.com`. You need it for
the IAM grant.
```bash theme={null}
gcloud iam service-accounts create tornado-delivery \
--project=my-gcp-project \
--display-name="Tornado delivery"
```
## 3. Grant the role on the bucket
### What Tornado actually does in your bucket
The role has to cover every operation the delivery path performs, not just the upload. The upload strategy is chosen by file size, and the large-file path is a composite upload, which is a different set of permissions from a plain write.
| Operation | When it happens | IAM permission |
| -------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------- |
| Single-request upload | Files under 5 MB | `storage.objects.create` |
| Resumable upload | Files of 5 MB or more | `storage.objects.create` |
| Parallel composite upload: part uploads | Files of 100 MB or more | `storage.objects.create` |
| Parallel composite upload: `objects.compose` | Same, once the parts are uploaded | `storage.objects.create` **and** `storage.objects.get` |
| Deleting the temporary parts after a compose | After every composite upload | `storage.objects.delete` |
| Overwriting an object that already exists | Re-running a job that writes the same key | `storage.objects.create` **and** `storage.objects.delete` |
| Deleting a delivered object | Cleanup operations | `storage.objects.delete` |
| Credential probe on `POST /user/gcs` | Only on the API route, at configuration time | `storage.objects.create`, then `storage.objects.delete` to clean up after itself |
| Serving a signed download URL | When someone opens a delivered link | `storage.objects.get` |
Two of these deserve spelling out.
**Composite uploads read what they wrote.** Files of 100 MB or more are uploaded as parts and then merged server-side with `objects.compose`, in batches of up to 32 sources. Google's reference for that method lists `storage.objects.create` and `storage.objects.get` as always required, and `storage.objects.delete` as required "if the new composite object overwrites an existing object". Tornado then deletes the temporary parts, which are named after the destination key with a `.tornado.tmp..` suffix.
Two bucket settings interact badly with those temporary parts. On a bucket with **object versioning** enabled, each temp part leaves a noncurrent version behind when it is deleted, billed until a lifecycle rule reaps it. On a bucket with a **retention policy or bucket lock**, the temp-part deletes fail outright, and no role fixes that: the bucket configuration is the blocker, not the credential.
**Signed URLs are signed locally, but read remotely.** Tornado builds V4 signed GET URLs offline using the private key in your JSON file, so signing itself never calls Google and never fails on permissions. The permission check happens when the recipient opens the link, and the request is authorised as the signing service account. Without `storage.objects.get`, every signed link returns 403 even though the file is sitting in the bucket.
### Minimum
Three permissions, and no more:
```
storage.objects.create # every upload: media, resumable, composite parts, the compose destination, the probe object
storage.objects.get # signed download URLs, and objects.compose reading its own source parts
storage.objects.delete # temp-part cleanup, probe cleanup, and overwrite-on-retry
```
Put them in a **custom role** and grant it **on the bucket**, never on the project. That is the tightest configuration that works.
Explicitly not needed: `storage.objects.list` (nothing enumerates the bucket), `storage.objects.update`, `storage.buckets.get`, `storage.buckets.list`, and all of `storage.multipartUploads.*` (that is the S3-compatible XML API, which this route does not use).
### Recommended
The same three permissions, reached through a predefined role rather than a custom one:
| Role | Why it is worth the extra |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`roles/storage.objectUser`** | Google describes it as access to create, read, update and delete objects. It is one grant to maintain instead of a custom role definition that has to be revisited whenever Google adds a permission, and it over-grants only `list` and `update`, neither of which reaches anything outside the bucket you grant it on |
`roles/storage.objectAdmin` also works and is the answer you will find in most older write-ups; it is a slightly wider version of the same idea. `roles/storage.objectViewer` and `roles/storage.objectCreator` together are still not enough, because neither carries `delete`.
Grant whichever you pick on the bucket, never on the project.
**`roles/storage.objectCreator` is not enough,** despite the hint shown under
the GCS tile in the onboarding wizard ("The service account only needs
`roles/storage.objectCreator`"). That role carries `storage.objects.create`
and nothing else: no `get`, no `delete`.
With `objectCreator` alone, small files written to object names that do not
yet exist will appear to work, and then:
* Every file of 100 MB or more fails, because `objects.compose` requires
`storage.objects.get`.
* Every signed download URL returns 403, because the reader's GET requires
`storage.objects.get`.
* Re-delivering to an object name that already exists fails, because
overwriting requires `storage.objects.delete`.
* Temporary composite parts and the `verify_credentials.txt` probe object are
left behind in the bucket, because the cleanup deletes are rejected and
cleanup failures are not surfaced.
The wizard's hint is wrong. Grant `roles/storage.objectUser` on the bucket, or
a custom role with the three permissions above.
### Granting it
In **Cloud Storage › Buckets**, select the bucket name, then select the
**Permissions** tab.
Select **Grant access**, paste the service account email into **New
principals**, and choose **Storage Object User** from **Select a role**, or
the custom role you created.
Select **Save**. The grant applies to this bucket only.
```bash theme={null}
gcloud storage buckets add-iam-policy-binding gs://tornado-downloads \
--member=serviceAccount:tornado-delivery@my-gcp-project.iam.gserviceaccount.com \
--role=roles/storage.objectUser
```
Check the grant landed on the bucket and not the project:
`gcloud storage buckets get-iam-policy gs://tornado-downloads` should list your
service account, and `gcloud projects get-iam-policy my-gcp-project` should
not.
## 4. Create and download the JSON key
In **IAM & Admin › Service Accounts**, select the service account email, then
select the **Keys** tab.
Select **Add key › Create new key**, choose **JSON**, and select **Create**.
The file downloads immediately.
The private key cannot be downloaded again. Treat the file as a credential:
keep it out of version control, and delete your local copy once it is in the
dashboard.
```bash theme={null}
gcloud iam service-accounts keys create tornado-delivery.json \
--iam-account=tornado-delivery@my-gcp-project.iam.gserviceaccount.com
```
The file is a JSON object beginning `{"type": "service_account", ...}`, roughly 2.3 KB. Tornado requires `type` to be exactly `service_account` and requires `project_id`, `private_key` and `client_email` to be present.
## 5. Enter the three fields
Open the dashboard, choose **Google Cloud Storage** as the destination, and fill in the fields. Labels differ between the onboarding wizard and the settings page; both write the same three API fields.
| Onboarding label | Settings label | API field | What to paste |
| -------------------------- | ---------------- | ------------------ | ------------------------------------------------------------------------------------------------------ |
| Bucket name | Bucket | `bucket` | The bucket name only, for example `tornado-downloads`. No `gs://` prefix, no URL, no path segments. |
| Project ID | Project ID | `project_id` | The project **ID**, for example `my-gcp-project`. Not the project name and not the project number. |
| Service account key (JSON) | Credentials JSON | `credentials_json` | The entire contents of the downloaded `.json` file, as a single value, opening brace to closing brace. |
### Pasting the key
Open the downloaded file in a text editor, select all, and paste. The field is a textarea and takes the raw file content, newlines and all. Do not reformat it.
**Do not base64-encode the key for this field, and do not escape it by hand.**
Tornado parses the stored value directly as JSON. A base64 blob fails with
*"Invalid JSON in service account credentials"*, and so does a value where the
`private_key` newlines have been mangled by a copy that went through an
intermediate tool. If you are constructing the API payload programmatically
instead of pasting, let your JSON serialiser escape the string; do not do it
yourself.
The equivalent payload:
```json theme={null}
{
"provider": "gcs",
"gcs": {
"bucket": "tornado-downloads",
"project_id": "my-gcp-project",
"credentials_json": "{\"type\":\"service_account\",\"project_id\":\"my-gcp-project\",\"private_key_id\":\"...\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\\n...\\n-----END PRIVATE KEY-----\\n\",\"client_email\":\"tornado-delivery@my-gcp-project.iam.gserviceaccount.com\",\"client_id\":\"...\",\"token_uri\":\"https://oauth2.googleapis.com/token\"}"
}
}
```
Setting the storage destination is an organization-level action restricted to
**Admin** and **Owner** members, and one provider is active at a time. Saving a
GCS configuration replaces whatever provider was configured before.
### Field constraints
| Field | Limit enforced on save | Also validated at job time |
| ------------------ | ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `bucket` | 255 characters | 3 to 63 characters, lowercase letters, digits, hyphens and periods, no consecutive periods, not an IP address |
| `project_id` | 128 characters | Non-empty; a non-standard shape is warned about but accepted |
| `credentials_json` | 10 000 characters | Must parse as JSON, `type` must be `service_account`, must contain `project_id`, `private_key` and `client_email` |
A normal service account key is well under 3 KB, so the 10 000 character limit only bites if you paste something that is not a single key file.
## Verify it for real
Because the connection test never contacts Google, the first honest signal is a delivered file. Run one small job and look in the bucket.
```bash theme={null}
curl -X POST "https://api.tornadoapi.io/jobs" \
-H "x-api-key: sk_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
"max_resolution": "360"
}'
```
```bash theme={null}
gcloud storage ls -r gs://tornado-downloads/videos/
```
Delivered files land under the base folder, which defaults to `videos/`.
A file under 5 MB exercises only `storage.objects.create`. Run a second job
that produces at least 100 MB (a longer video at a higher resolution) to
exercise the composite upload path, which is where an under-privileged role
actually breaks.
Fetch the delivered object with the service account's own credentials, which
is the same authorisation a signed download URL uses:
```bash theme={null}
gcloud auth activate-service-account --key-file=tornado-delivery.json
gcloud storage cp gs://tornado-downloads/videos/ /tmp/check.mp4
```
A 403 here means the role is missing `storage.objects.get`, and every signed
link you hand out will fail the same way.
**Judge the outcome by the bucket, not by the job response.** The `s3_url`
field returned by `GET /jobs/{id}` and included in completion webhook payloads
is produced by Tornado's S3 signing path. For a GCS destination it does not
currently resolve to your GCS object, so neither its presence nor a failure to
fetch it tells you anything about whether the delivery worked. Look in the
bucket.
**If nothing arrives at all, check which configuration the job used before you
suspect IAM.** The job pipeline resolves storage per API key, from the
configuration set through [`POST /user/gcs`](/features/custom-storage#google-cloud-storage)
with that key's `x-api-key`, falling back to the organization default and then
to Tornado's managed storage. A job that ran against managed storage still
reports `Completed`, with the file in Tornado's bucket rather than yours. See
[Custom Storage](/features/custom-storage#job-completed-but-the-file-isnt-in-my-storage).
## Troubleshooting
The dashboard's connection test cannot produce any of these errors, because it makes no request. They surface on the first job instead.
The worker treats `401`, `403`, `404`, `invalid_grant` and `Invalid JWT` as permanent and fails the job immediately without retrying, reporting: *"Invalid GCS storage credentials. Please check your service\_account\_json, project\_id, and bucket name."* That single message covers several distinct causes.
| Symptom | Cause | Fix |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Small files deliver, files of 100 MB or more fail | The role lacks `storage.objects.get`, so `objects.compose` is rejected. This is the exact failure mode of `roles/storage.objectCreator`. | Grant `roles/storage.objectUser` on the bucket. |
| Files deliver, but download links return 403 | The role lacks `storage.objects.get`. Signing succeeds offline; the reader's GET is what gets rejected. | Grant `roles/storage.objectUser` on the bucket. |
| Re-running a job that writes an object name that already exists fails | Overwriting an existing object requires `storage.objects.delete` in addition to `storage.objects.create`. | Grant `roles/storage.objectUser` on the bucket. |
| `verify_credentials.txt` or `*.tornado.tmp.*` objects accumulate in the bucket | The role lacks `storage.objects.delete`. Cleanup deletes are best-effort and their failure is not surfaced, so the objects simply pile up. | Grant `roles/storage.objectUser`, then delete the leftovers. Objects matching those names are safe to remove. |
| Every job fails immediately, nothing appears in the bucket | The service account was granted on the **project** but not on the **bucket**, or granted on a different bucket. A project-level grant on a different project does nothing here. | Check `gcloud storage buckets get-iam-policy gs://` lists the service account. |
| Every job fails immediately, and the bucket exists | `project_id` does not match the project that owns the bucket, or the bucket name is misspelled. Bucket names are global, so a typo can resolve to somebody else's bucket and return 403 rather than 404. | Confirm both against `gcloud storage buckets describe gs://`. |
| Configuration saved green, every job fails on credentials | The pasted JSON is not a complete, valid key file: a partial paste, a base64 blob, or `private_key` newlines mangled in transit. The dashboard test does not parse the JSON, so this is accepted at save time. | Re-copy the whole file, opening brace to closing brace, into the field. |
| Jobs that used to work start failing on credentials | The key was disabled or deleted in the console, or the service account itself was deleted or disabled. | Create a new key and update the configuration. Note that deleting a key does not revoke short-lived tokens already issued from it, so a revoked key can appear to keep working briefly. |
| `Service account JSON 'type' must be 'service_account'` | An OAuth client secret, an external account configuration, or an `authorized_user` credential was pasted instead of a service account key. | Download the key from the service account's **Keys** tab, not from **APIs & Services › Credentials**. |
| Key creation is refused in the console | `constraints/iam.disableServiceAccountKeyCreation` is enforced on the project, which is the default for projects created after 3 May 2024. | Ask an organization administrator to grant an exception for the project. |
## Rotating the key
A service account can hold more than one key at a time, so overlap them and you rotate without an outage.
Add a new JSON key on the same service account. Both keys are now valid.
Paste the new key into the dashboard and run a test job through to the
bucket.
Disable the old key first and watch for failures before deleting it.
Deletion is permanent.
Rotation also invalidates links you have already handed out. Google's
documentation is explicit that a signed URL grants access "until the
expiration time for the URL is reached or the key used to sign the URL is
rotated", so retiring a key cuts short every outstanding link signed with it,
ahead of its 24 hour expiry. Rotate when those links are expendable, or expect
recipients to need fresh ones.
## Not available on this route
| Feature | Status |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Live credential validation from the dashboard | Not implemented. `POST /settings/storage/test` returns `verified: false` for GCS and makes no network call. The per-API-key endpoint [`POST /user/gcs`](/features/custom-storage#google-cloud-storage) does validate for real, by uploading and deleting a `verify_credentials.txt` probe object. |
| Reading the configuration back | Only through `GET /settings/storage`, with `credentials_json` masked. There is no `GET /user/gcs`. |
| `folder_prefix` / `base_folder` | Not accepted. Available on [`POST /user/gcs`](/features/custom-storage#folder-prefix) only. Deliveries through the dashboard route use the default `videos/` base folder. |
| Base64-encoded credentials | Not accepted. The value is parsed directly as JSON. |
| Workload Identity Federation, attached service accounts, impersonation | Not supported. A downloaded service account key is the only credential type. |
| HMAC keys / the GCS XML API | Not supported. The GCS route speaks the JSON API and authenticates with a service account key; there is no field for an access key and secret. |
| Customer-supplied encryption keys (CSEK) | Not supported. Customer-managed keys (CMEK) configured as the bucket default work, because they are transparent to the client. |
| Automatic bucket creation | Not supported. Create the bucket first. |