Google Cloud Storage

Overview

Tornado can deliver completed downloads straight into a Google Cloud Storage bucket you own. Authentication is a service account key: you create the service account in your own project, grant it a role on the bucket, download its JSON key, and paste that key into the dashboard. The dashboard asks for exactly three values: the bucket, the project ID, and the whole service account key file as one value. This page covers the dashboard path (the onboarding wizard and Settings › Default storage destination), which writes an organization-level configuration through POST /settings/storage. Credentials are stored in Azure Key Vault and are returned masked when the configuration is read back.

The dashboard route is not the same contract as the per-API-key endpoint POST /user/gcs. That endpoint names the credentials field service_account_json and additionally accepts folder_prefix and base_folder. The dashboard route names the field credentials_json and accepts three fields only. Do not copy a payload from one to the other.

Current limitations

Read these before you create anything in Google Cloud. Both are properties of the GCS route as it exists today, not Google restrictions.

“Test connection” does not contact Google. For GCS there is no probe object, no API call, and no credential check of any kind. The test loads the submitted configuration, confirms that bucket, project_id and credentials_json are present and non-empty, and returns:

{
  "ok": true,
  "verified": false,
  "latency_ms": 0,
  "note": "credentials look structurally valid — live round-trip test is not yet supported for provider 'gcs'"
}

The wizard renders that as a green banner reading “Credentials accepted and saved, verified on your first delivery.” While the request is in flight the button reads “Writing test object…”, which is the shared wizard label: for GCS nothing is written anywhere.A green result here proves that you filled in three boxes. It does not parse the JSON, and it says nothing about the key, the role, the project or the bucket. Every one of those failures surfaces later, on a real job. Verify with a real delivery before you consider the setup done.The S3 branch of the same endpoint was recently extended to write, read back, open and abort a multipart upload, and delete. None of that applies here. If you have read the AWS S3 or Cloudflare R2 page, discard what it says about the probe: for GCS no packet leaves Tornado.

The bucket must already exist. Tornado never creates it, and because the connection test makes no call, a bucket that does not exist is accepted at configuration time and fails on the first job.

Before you start

You need a Google Cloud project and, within it, permission to create a bucket, grant IAM on that bucket, and create a service account key.

Service account key creation may be blocked by organization policy. The constraints/iam.disableServiceAccountKeyCreation constraint prevents key creation in a project, and Google enforces it by default on projects created after 3 May 2024. If key creation fails with a policy error, an organization administrator must grant an exception for the project before you can continue. Tornado authenticates with a service account key only: Workload Identity Federation, attached service accounts and impersonation are not supported on this route.

1. Create the bucket

Open the bucket creation flow

In the Google Cloud console, go to Cloud Storage › Buckets, then select Create. Console layouts change; if the entry is not where you expect it, search for “Cloud Storage” from the top search bar.

Name it

Bucket names are globally unique across all of Google Cloud. Tornado additionally validates the name before every upload: 3 to 63 characters, lowercase letters, digits, hyphens and periods only, starting and ending with a letter or digit, no consecutive periods, and not shaped like an IP address. A name Google accepts but Tornado rejects will fail at job time, not at save time.

Choose the location

Pick a location type and location. See Choosing a location below.

Leave access control as uniform

Tornado authorises entirely through IAM and never reads or writes object ACLs, so uniform bucket-level access is the right setting and is what the rest of this page assumes. Note that uniform bucket-level access cannot be turned off again once it has been active on a bucket for 90 consecutive days.

Or from the command line:

gcloud storage buckets create gs://tornado-downloads \
  --project=my-gcp-project \
  --location=europe-west1 \
  --uniform-bucket-level-access

Choosing a location

Google offers regions, dual-regions and multi-regions. For delivery buckets a single region is almost always correct:

A region has the lowest storage price and no replication charges. Dual-region and multi-region buckets add a replication charge on every write, and Tornado writes large media objects, so the difference is not marginal.
Redundancy across regions buys you nothing here. Delivered files are reproducible: if a bucket becomes unavailable you re-run the job.

For which region, colocate with the traffic that matters most to you. Each file is written once but may be read many times, so the region closest to whoever consumes the files is usually the right choice. If upload throughput matters more than read latency, the latency_ms reported by the connection test is a fair proxy for how far your bucket is from our workers.

2. Create the service account

Open Service Accounts

Go to IAM & Admin › Service Accounts in the console, select your project, then select Create service account.

Name it for this purpose only

Something like tornado-delivery. Use one service account per delivery destination so you can revoke it without affecting anything else.

Skip the project-level role

The creation flow offers an optional step titled along the lines of “Grant this service account access to project”. Skip it. Granting a role there applies it to every bucket in the project, present and future. You will grant the role on one bucket in the next step.

Copy the service account email

It looks like tornado-delivery@my-gcp-project.iam.gserviceaccount.com. You need it for the IAM grant.

gcloud iam service-accounts create tornado-delivery \
  --project=my-gcp-project \
  --display-name="Tornado delivery"

3. Grant the role on the bucket

What Tornado actually does in your bucket

The role has to cover every operation the delivery path performs, not just the upload. The upload strategy is chosen by file size, and the large-file path is a composite upload, which is a different set of permissions from a plain write.

Operation	When it happens	IAM permission
Single-request upload	Files under 5 MB	`storage.objects.create`
Resumable upload	Files of 5 MB or more	`storage.objects.create`
Parallel composite upload: part uploads	Files of 100 MB or more	`storage.objects.create`
Parallel composite upload: `objects.compose`	Same, once the parts are uploaded	`storage.objects.create` and `storage.objects.get`
Deleting the temporary parts after a compose	After every composite upload	`storage.objects.delete`
Overwriting an object that already exists	Re-running a job that writes the same key	`storage.objects.create` and `storage.objects.delete`
Deleting a delivered object	Cleanup operations	`storage.objects.delete`
Credential probe on `POST /user/gcs`	Only on the API route, at configuration time	`storage.objects.create`, then `storage.objects.delete` to clean up after itself
Serving a signed download URL	When someone opens a delivered link	`storage.objects.get`

Two of these deserve spelling out. Composite uploads read what they wrote. Files of 100 MB or more are uploaded as parts and then merged server-side with objects.compose, in batches of up to 32 sources. Google’s reference for that method lists storage.objects.create and storage.objects.get as always required, and storage.objects.delete as required “if the new composite object overwrites an existing object”. Tornado then deletes the temporary parts, which are named after the destination key with a .tornado.tmp.<uuid>.<nnnnn> suffix.

Two bucket settings interact badly with those temporary parts. On a bucket with object versioning enabled, each temp part leaves a noncurrent version behind when it is deleted, billed until a lifecycle rule reaps it. On a bucket with a retention policy or bucket lock, the temp-part deletes fail outright, and no role fixes that: the bucket configuration is the blocker, not the credential.

Signed URLs are signed locally, but read remotely. Tornado builds V4 signed GET URLs offline using the private key in your JSON file, so signing itself never calls Google and never fails on permissions. The permission check happens when the recipient opens the link, and the request is authorised as the signing service account. Without storage.objects.get, every signed link returns 403 even though the file is sitting in the bucket.

Minimum

Three permissions, and no more:

storage.objects.create   # every upload: media, resumable, composite parts, the compose destination, the probe object
storage.objects.get      # signed download URLs, and objects.compose reading its own source parts
storage.objects.delete   # temp-part cleanup, probe cleanup, and overwrite-on-retry

Put them in a custom role and grant it on the bucket, never on the project. That is the tightest configuration that works. Explicitly not needed: storage.objects.list (nothing enumerates the bucket), storage.objects.update, storage.buckets.get, storage.buckets.list, and all of storage.multipartUploads.* (that is the S3-compatible XML API, which this route does not use).

Role	Why it is worth the extra
`roles/storage.objectUser`	Google describes it as access to create, read, update and delete objects. It is one grant to maintain instead of a custom role definition that has to be revisited whenever Google adds a permission, and it over-grants only `list` and `update`, neither of which reaches anything outside the bucket you grant it on

roles/storage.objectAdmin also works and is the answer you will find in most older write-ups; it is a slightly wider version of the same idea. roles/storage.objectViewer and roles/storage.objectCreator together are still not enough, because neither carries delete. Grant whichever you pick on the bucket, never on the project.

roles/storage.objectCreator is not enough, despite the hint shown under the GCS tile in the onboarding wizard (“The service account only needs roles/storage.objectCreator”). That role carries storage.objects.create and nothing else: no get, no delete.With objectCreator alone, small files written to object names that do not yet exist will appear to work, and then:

Every file of 100 MB or more fails, because objects.compose requires storage.objects.get.
Every signed download URL returns 403, because the reader’s GET requires storage.objects.get.
Re-delivering to an object name that already exists fails, because overwriting requires storage.objects.delete.
Temporary composite parts and the verify_credentials.txt probe object are left behind in the bucket, because the cleanup deletes are rejected and cleanup failures are not surfaced.

The wizard’s hint is wrong. Grant roles/storage.objectUser on the bucket, or a custom role with the three permissions above.

Granting it

Open the bucket's Permissions tab

In Cloud Storage › Buckets, select the bucket name, then select the Permissions tab.

Grant access

Select Grant access, paste the service account email into New principals, and choose Storage Object User from Select a role, or the custom role you created.

Save

Select Save. The grant applies to this bucket only.

gcloud storage buckets add-iam-policy-binding gs://tornado-downloads \
  --member=serviceAccount:tornado-delivery@my-gcp-project.iam.gserviceaccount.com \
  --role=roles/storage.objectUser

Check the grant landed on the bucket and not the project: gcloud storage buckets get-iam-policy gs://tornado-downloads should list your service account, and gcloud projects get-iam-policy my-gcp-project should not.

4. Create and download the JSON key

Open the Keys tab

In IAM & Admin › Service Accounts, select the service account email, then select the Keys tab.

Create a JSON key

Select Add key › Create new key, choose JSON, and select Create. The file downloads immediately.

Store it somewhere sane

The private key cannot be downloaded again. Treat the file as a credential: keep it out of version control, and delete your local copy once it is in the dashboard.

gcloud iam service-accounts keys create tornado-delivery.json \
  --iam-account=tornado-delivery@my-gcp-project.iam.gserviceaccount.com

The file is a JSON object beginning {"type": "service_account", ...}, roughly 2.3 KB. Tornado requires type to be exactly service_account and requires project_id, private_key and client_email to be present.

5. Enter the three fields

Open the dashboard, choose Google Cloud Storage as the destination, and fill in the fields. Labels differ between the onboarding wizard and the settings page; both write the same three API fields.

Onboarding label	Settings label	API field	What to paste
Bucket name	Bucket	`bucket`	The bucket name only, for example `tornado-downloads`. No `gs://` prefix, no URL, no path segments.
Project ID	Project ID	`project_id`	The project ID, for example `my-gcp-project`. Not the project name and not the project number.
Service account key (JSON)	Credentials JSON	`credentials_json`	The entire contents of the downloaded `.json` file, as a single value, opening brace to closing brace.

Pasting the key

Open the downloaded file in a text editor, select all, and paste. The field is a textarea and takes the raw file content, newlines and all. Do not reformat it.

Do not base64-encode the key for this field, and do not escape it by hand. Tornado parses the stored value directly as JSON. A base64 blob fails with “Invalid JSON in service account credentials”, and so does a value where the private_key newlines have been mangled by a copy that went through an intermediate tool. If you are constructing the API payload programmatically instead of pasting, let your JSON serialiser escape the string; do not do it yourself.

The equivalent payload:

{
  "provider": "gcs",
  "gcs": {
    "bucket": "tornado-downloads",
    "project_id": "my-gcp-project",
    "credentials_json": "{\"type\":\"service_account\",\"project_id\":\"my-gcp-project\",\"private_key_id\":\"...\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\\n...\\n-----END PRIVATE KEY-----\\n\",\"client_email\":\"tornado-delivery@my-gcp-project.iam.gserviceaccount.com\",\"client_id\":\"...\",\"token_uri\":\"https://oauth2.googleapis.com/token\"}"
  }
}

Setting the storage destination is an organization-level action restricted to Admin and Owner members, and one provider is active at a time. Saving a GCS configuration replaces whatever provider was configured before.

Field constraints

Field	Limit enforced on save	Also validated at job time
`bucket`	255 characters	3 to 63 characters, lowercase letters, digits, hyphens and periods, no consecutive periods, not an IP address
`project_id`	128 characters	Non-empty; a non-standard shape is warned about but accepted
`credentials_json`	10 000 characters	Must parse as JSON, `type` must be `service_account`, must contain `project_id`, `private_key` and `client_email`

A normal service account key is well under 3 KB, so the 10 000 character limit only bites if you paste something that is not a single key file.

Verify it for real

Because the connection test never contacts Google, the first honest signal is a delivered file. Run one small job and look in the bucket.

Submit a short download

curl -X POST "https://api.tornadoapi.io/jobs" \
  -H "x-api-key: sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "max_resolution": "360"
  }'

Wait for the job to finish, then list the bucket

gcloud storage ls -r gs://tornado-downloads/videos/

Delivered files land under the base folder, which defaults to videos/.

Then test a large file

A file under 5 MB exercises only storage.objects.create. Run a second job that produces at least 100 MB (a longer video at a higher resolution) to exercise the composite upload path, which is where an under-privileged role actually breaks.

Then test a read

Fetch the delivered object with the service account’s own credentials, which is the same authorisation a signed download URL uses:

gcloud auth activate-service-account --key-file=tornado-delivery.json
gcloud storage cp gs://tornado-downloads/videos/<delivered-file> /tmp/check.mp4

A 403 here means the role is missing storage.objects.get, and every signed link you hand out will fail the same way.

Judge the outcome by the bucket, not by the job response. The s3_url field returned by GET /jobs/{id} and included in completion webhook payloads is produced by Tornado’s S3 signing path. For a GCS destination it does not currently resolve to your GCS object, so neither its presence nor a failure to fetch it tells you anything about whether the delivery worked. Look in the bucket.

If nothing arrives at all, check which configuration the job used before you suspect IAM. The job pipeline resolves storage per API key, from the configuration set through POST /user/gcs with that key’s x-api-key, falling back to the organization default and then to Tornado’s managed storage. A job that ran against managed storage still reports Completed, with the file in Tornado’s bucket rather than yours. See Custom Storage.

Troubleshooting

The dashboard’s connection test cannot produce any of these errors, because it makes no request. They surface on the first job instead. The worker treats 401, 403, 404, invalid_grant and Invalid JWT as permanent and fails the job immediately without retrying, reporting: “Invalid GCS storage credentials. Please check your service_account_json, project_id, and bucket name.” That single message covers several distinct causes.

Symptom	Cause	Fix
Small files deliver, files of 100 MB or more fail	The role lacks `storage.objects.get`, so `objects.compose` is rejected. This is the exact failure mode of `roles/storage.objectCreator`.	Grant `roles/storage.objectUser` on the bucket.
Files deliver, but download links return 403	The role lacks `storage.objects.get`. Signing succeeds offline; the reader’s GET is what gets rejected.	Grant `roles/storage.objectUser` on the bucket.
Re-running a job that writes an object name that already exists fails	Overwriting an existing object requires `storage.objects.delete` in addition to `storage.objects.create`.	Grant `roles/storage.objectUser` on the bucket.
`verify_credentials.txt` or `.tornado.tmp.` objects accumulate in the bucket	The role lacks `storage.objects.delete`. Cleanup deletes are best-effort and their failure is not surfaced, so the objects simply pile up.	Grant `roles/storage.objectUser`, then delete the leftovers. Objects matching those names are safe to remove.
Every job fails immediately, nothing appears in the bucket	The service account was granted on the project but not on the bucket, or granted on a different bucket. A project-level grant on a different project does nothing here.	Check `gcloud storage buckets get-iam-policy gs://<bucket>` lists the service account.
Every job fails immediately, and the bucket exists	`project_id` does not match the project that owns the bucket, or the bucket name is misspelled. Bucket names are global, so a typo can resolve to somebody else’s bucket and return 403 rather than 404.	Confirm both against `gcloud storage buckets describe gs://<bucket>`.
Configuration saved green, every job fails on credentials	The pasted JSON is not a complete, valid key file: a partial paste, a base64 blob, or `private_key` newlines mangled in transit. The dashboard test does not parse the JSON, so this is accepted at save time.	Re-copy the whole file, opening brace to closing brace, into the field.
Jobs that used to work start failing on credentials	The key was disabled or deleted in the console, or the service account itself was deleted or disabled.	Create a new key and update the configuration. Note that deleting a key does not revoke short-lived tokens already issued from it, so a revoked key can appear to keep working briefly.
`Service account JSON 'type' must be 'service_account'`	An OAuth client secret, an external account configuration, or an `authorized_user` credential was pasted instead of a service account key.	Download the key from the service account’s Keys tab, not from APIs & Services › Credentials.
Key creation is refused in the console	`constraints/iam.disableServiceAccountKeyCreation` is enforced on the project, which is the default for projects created after 3 May 2024.	Ask an organization administrator to grant an exception for the project.

Rotating the key

A service account can hold more than one key at a time, so overlap them and you rotate without an outage.

Create a second key

Add a new JSON key on the same service account. Both keys are now valid.

Update Tornado and confirm

Paste the new key into the dashboard and run a test job through to the bucket.

Disable, then delete, the old key

Disable the old key first and watch for failures before deleting it. Deletion is permanent.

Rotation also invalidates links you have already handed out. Google’s documentation is explicit that a signed URL grants access “until the expiration time for the URL is reached or the key used to sign the URL is rotated”, so retiring a key cuts short every outstanding link signed with it, ahead of its 24 hour expiry. Rotate when those links are expendable, or expect recipients to need fresh ones.

Not available on this route

Feature	Status
Live credential validation from the dashboard	Not implemented. `POST /settings/storage/test` returns `verified: false` for GCS and makes no network call. The per-API-key endpoint `POST /user/gcs` does validate for real, by uploading and deleting a `verify_credentials.txt` probe object.
Reading the configuration back	Only through `GET /settings/storage`, with `credentials_json` masked. There is no `GET /user/gcs`.
`folder_prefix` / `base_folder`	Not accepted. Available on `POST /user/gcs` only. Deliveries through the dashboard route use the default `videos/` base folder.
Base64-encoded credentials	Not accepted. The value is parsed directly as JSON.
Workload Identity Federation, attached service accounts, impersonation	Not supported. A downloaded service account key is the only credential type.
HMAC keys / the GCS XML API	Not supported. The GCS route speaks the JSON API and authenticates with a service account key; there is no field for an access key and secret.
Customer-supplied encryption keys (CSEK)	Not supported. Customer-managed keys (CMEK) configured as the bucket default work, because they are transparent to the client.
Automatic bucket creation	Not supported. Create the bucket first.

Get Started

Features

Storage guides

Integrations

Overview

Current limitations

Before you start

1. Create the bucket

Choosing a location

2. Create the service account

3. Grant the role on the bucket

What Tornado actually does in your bucket

Minimum

Recommended

Granting it

4. Create and download the JSON key

5. Enter the three fields

Pasting the key

Field constraints

Verify it for real

Troubleshooting

Rotating the key

Not available on this route

​Overview

​Current limitations

​Before you start

​1. Create the bucket

​Choosing a location

​2. Create the service account

​3. Grant the role on the bucket

​What Tornado actually does in your bucket

​Minimum

​Recommended

​Granting it

​4. Create and download the JSON key

​5. Enter the three fields

​Pasting the key

​Field constraints

​Verify it for real

​Troubleshooting

​Rotating the key

​Not available on this route

Overview

Current limitations

Before you start

1. Create the bucket

Choosing a location

2. Create the service account

3. Grant the role on the bucket

What Tornado actually does in your bucket

Minimum

Recommended

Granting it

4. Create and download the JSON key

5. Enter the three fields

Pasting the key

Field constraints

Verify it for real

Troubleshooting

Rotating the key

Not available on this route