Google Cloud Storage

Overview

Tornado can deliver completed downloads straight into a Google Cloud Storage bucket you own. Authentication is a service account key: you create the service account in your own project, grant it a role on the bucket, download its JSON key, and paste that key into the dashboard. The dashboard asks for exactly three values: the bucket, the project ID, and the whole service account key file as one value. This page covers the dashboard path (the onboarding wizard and Settings › Default storage destination), which writes an organization-level configuration through POST /settings/storage. Credentials are stored in Azure Key Vault and are returned masked when the configuration is read back.

The dashboard route is not the same contract as the per-API-key endpoint POST /user/gcs. That endpoint names the credentials field service_account_json and additionally accepts folder_prefix and base_folder. The dashboard route names the field credentials_json and accepts three fields only. Do not copy a payload from one to the other.

Current limitations

Read these before you create anything in Google Cloud. Both are properties of the GCS route as it exists today, not Google restrictions.

“Test connection” does not contact Google. For GCS there is no probe object, no API call, and no credential check of any kind. The test loads the submitted configuration, confirms that bucket, project_id and credentials_json are present and non-empty, and returns:

{
  "ok": true,
  "verified": false,
  "latency_ms": 0,
  "note": "credentials look structurally valid — live round-trip test is not yet supported for provider 'gcs'"
}

The wizard renders that as a green banner reading “Credentials accepted and saved, verified on your first delivery.” While the request is in flight the button reads “Writing test object…”, which is the shared wizard label: for GCS nothing is written anywhere.A green result here proves that you filled in three boxes. It does not parse the JSON, and it says nothing about the key, the role, the project or the bucket. Every one of those failures surfaces later, on a real job. Verify with a real delivery before you consider the setup done.The S3 branch of the same endpoint was recently extended to write, read back, open and abort a multipart upload, and delete. None of that applies here. If you have read the AWS S3 or Cloudflare R2 page, discard what it says about the probe: for GCS no packet leaves Tornado.

The bucket must already exist. Tornado never creates it, and because the connection test makes no call, a bucket that does not exist is accepted at configuration time and fails on the first job.

Before you start

You need a Google Cloud project and, within it, permission to create a bucket, grant IAM on that bucket, and create a service account key.

Service account key creation may be blocked by organization policy. The constraints/iam.disableServiceAccountKeyCreation constraint prevents key creation in a project, and may be enforced by your organization. If key creation fails with a policy error, an organization administrator must grant an exception for the project before you can continue. Tornado authenticates with a service account key only: Workload Identity Federation, attached service accounts and impersonation are not supported on this route.

1. Create the bucket

Open the bucket creation flow

In the Google Cloud console, go to Cloud Storage › Buckets, then select Create. Console layouts change; if the entry is not where you expect it, search for “Cloud Storage” from the top search bar.

Name it

Bucket names are globally unique across all of Google Cloud. Tornado additionally validates the name before every upload: 3 to 63 characters, lowercase letters, digits, hyphens and periods only, starting and ending with a letter or digit, no consecutive periods, and not shaped like an IP address. A name Google accepts but Tornado rejects will fail at job time, not at save time.

Choose the location

Pick a location type and location. See Choosing a location below.

Leave access control as uniform

Tornado authorises entirely through IAM and never reads or writes object ACLs, so uniform bucket-level access is the right setting and is what the rest of this page assumes. Note that uniform bucket-level access cannot be turned off again once it has been active on a bucket for 90 consecutive days.

Or from the command line:

gcloud storage buckets create gs://tornado-downloads \
  --project=my-gcp-project \
  --location=europe-west1 \
  --uniform-bucket-level-access

Choosing a location

Google offers regions, dual-regions and multi-regions. Choose according to your availability requirements, data-location constraints and the provider’s current storage, replication and transfer prices. Re-running a download is only an option while the original media remains available. Compare upload duration and recipient download latency using real deliveries to candidate locations. The GCS dashboard test returns latency_ms: 0 without contacting Google, so it cannot measure distance to Tornado’s workers.

2. Create the service account

Open Service Accounts

Go to IAM & Admin › Service Accounts in the console, select your project, then select Create service account.

Name it for this purpose only

Something like tornado-delivery. Use one service account per delivery destination so you can revoke it without affecting anything else.

Skip the project-level role

The creation flow offers an optional step titled along the lines of “Grant this service account access to project”. Skip it. Granting a role there applies it to every bucket in the project, present and future. You will grant the role on one bucket in the next step.

Copy the service account email

It looks like tornado-delivery@my-gcp-project.iam.gserviceaccount.com. You need it for the IAM grant.

gcloud iam service-accounts create tornado-delivery \
  --project=my-gcp-project \
  --display-name="Tornado delivery"

3. Grant the role on the bucket

What Tornado actually does in your bucket

The role has to cover every operation the delivery path performs, not just the upload. The default file-upload strategy depends on size and concurrency. Large files can use parallel composite upload, while an optional worker mode uses XML multipart. Streaming uploads have their own strategy; a small dashboard probe does not exercise these paths.

Operation	When it happens	IAM permission
Single-request upload	Files under 5 MiB in the default file path	`storage.objects.create`
Resumable upload	Files of 5 MiB or more when composite/XML is not selected	`storage.objects.create`
Parallel composite upload: part uploads	Files of at least 100 MiB when composite upload is enabled and concurrency exceeds one	`storage.objects.create`
Parallel composite upload: `objects.compose`	Same, once the parts are uploaded	`storage.objects.create` and `storage.objects.get`
Deleting the temporary parts after a compose	After every composite upload	`storage.objects.delete`
Overwriting an object that already exists	Re-running a job that writes the same key	`storage.objects.create` and `storage.objects.delete`
Deleting a delivered object	Cleanup operations	`storage.objects.delete`
Credential probe on `POST /user/gcs`	Only on the API route, at configuration time	`storage.objects.create`, then `storage.objects.delete` to clean up after itself
Serving a signed download URL	When someone opens a delivered link	`storage.objects.get`

Two of these deserve spelling out. Composite uploads read what they wrote. When selected, the composite path uploads parts and merges them server-side with objects.compose, in batches of up to 32 sources. Google’s reference for that method lists storage.objects.create and storage.objects.get as always required, and storage.objects.delete as required “if the new composite object overwrites an existing object”. Tornado then deletes the temporary parts, which are named after the destination key with a .tornado.tmp.<uuid>.<nnnnn> suffix.

Two bucket settings interact badly with those temporary parts. On a bucket with object versioning enabled, each temp part leaves a noncurrent version behind when it is deleted, billed until a lifecycle rule reaps it. On a bucket with a retention policy or bucket lock, the temp-part deletes fail outright, and no role fixes that: the bucket configuration is the blocker, not the credential.

Signed URLs are signed locally, but read remotely. Tornado builds V4 signed GET URLs offline using the private key in your JSON file, so signing itself does not call Google or validate object-read permissions. The permission check happens when the recipient opens the link, and the request is authorised as the signing service account. Without storage.objects.get, every signed link returns 403 even though the file is sitting in the bucket.

Minimum

For the default JSON/composite paths, allow these object operations:

storage.objects.create   # every upload: media, resumable, composite parts, the compose destination, the probe object
storage.objects.get      # signed download URLs, and objects.compose reading its own source parts
storage.objects.delete   # temp-part cleanup, probe cleanup, and overwrite-on-retry

A custom role granted on this bucket avoids giving the service account access to other buckets in the project. Inherited project permissions can also authorize access, but are broader. The default JSON/composite path does not enumerate the bucket. Its operation set does not require storage.objects.list, storage.objects.update, storage.buckets.get or storage.buckets.list.

The optional worker mode GCS_XML_MULTIPART=1 uses XML multipart with the same service-account credential. That mode additionally needs the multipart permissions for creating and aborting uploads. Confirm the deployed upload mode and its IAM requirements before adopting a three-permission custom role. The dashboard does not expose an XML toggle or accept HMAC keys.

Role	Why it is worth the extra
`roles/storage.objectUser`	Covers object creation, reading and deletion, with additional permissions beyond this default path. Scope the grant to the destination bucket and verify its multipart permissions if the XML mode is enabled

roles/storage.objectAdmin also works and is the answer you will find in most older write-ups; it is a slightly wider version of the same idea. roles/storage.objectViewer and roles/storage.objectCreator together are still not enough, because neither carries delete. Prefer granting the selected role on the destination bucket rather than the whole project.

roles/storage.objectCreator is not enough. It does not provide the object-read and deletion permissions this delivery flow needs. The current onboarding hint recommends roles/storage.objectAdmin; a narrower role must still cover every operation used by your upload mode.With objectCreator alone, small files written to object names that do not yet exist will appear to work, and then:

A composite upload fails when objects.compose needs storage.objects.get. The default file path selects composite upload from 100 MiB when enabled and parallelism is available.
Every signed download URL returns 403, because the reader’s GET requires storage.objects.get.
Re-delivering to an object name that already exists fails, because overwriting requires storage.objects.delete.
Temporary composite parts and the verify_credentials.txt probe object are left behind in the bucket, because the cleanup deletes are rejected and cleanup failures are not surfaced.

Grant a bucket-scoped role covering the operations of your deployed upload mode, and validate both a real delivery and a recipient download.

Granting it

Open the bucket's Permissions tab

In Cloud Storage › Buckets, select the bucket name, then select the Permissions tab.

Grant access

Select Grant access, paste the service account email into New principals, and choose Storage Object User from Select a role, or the custom role you created.

Save

Select Save. The grant applies to this bucket only.

gcloud storage buckets add-iam-policy-binding gs://tornado-downloads \
  --member=serviceAccount:tornado-delivery@my-gcp-project.iam.gserviceaccount.com \
  --role=roles/storage.objectUser

Check the grant landed on the bucket and not the project: gcloud storage buckets get-iam-policy gs://tornado-downloads should list your service account, and gcloud projects get-iam-policy my-gcp-project should not.

4. Create and download the JSON key

Open the Keys tab

In IAM & Admin › Service Accounts, select the service account email, then select the Keys tab.

Create a JSON key

Select Add key › Create new key, choose JSON, and select Create. The file downloads immediately.

Store it somewhere sane

The private key cannot be downloaded again. Treat the file as a credential: keep it out of version control, and delete your local copy once it is in the dashboard.

gcloud iam service-accounts keys create tornado-delivery.json \
  --iam-account=tornado-delivery@my-gcp-project.iam.gserviceaccount.com

The file is a JSON object beginning {"type": "service_account", ...}, roughly 2.3 KB. Tornado requires type to be exactly service_account and requires project_id, private_key and client_email to be present.

5. Enter the three fields

Open the dashboard, choose Google Cloud Storage as the destination, and fill in the fields. Labels differ between the onboarding wizard and the settings page; both write the same three API fields.

Onboarding label	Settings label	API field	What to paste
Bucket name	Bucket	`bucket`	The bucket name only, for example `tornado-downloads`. No `gs://` prefix, no URL, no path segments.
Project ID	Project ID	`project_id`	The project ID, for example `my-gcp-project`. Not the project name and not the project number.
Service account key (JSON)	Credentials JSON	`credentials_json`	The entire contents of the downloaded `.json` file, as a single value, opening brace to closing brace.

Pasting the key

Open the downloaded file in a text editor, select all, and paste. The field is a textarea and takes the raw file content, newlines and all. Do not reformat it.

Do not base64-encode the key for this field, and do not escape it by hand. Tornado parses the stored value directly as JSON. A base64 blob fails with “Invalid JSON in service account credentials”, and so does a value where the private_key newlines have been mangled by a copy that went through an intermediate tool. If you are constructing the API payload programmatically instead of pasting, let your JSON serialiser escape the string; do not do it yourself.

The equivalent payload:

{
  "provider": "gcs",
  "gcs": {
    "bucket": "tornado-downloads",
    "project_id": "my-gcp-project",
    "credentials_json": "{\"type\":\"service_account\",\"project_id\":\"my-gcp-project\",\"private_key_id\":\"...\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\\n...\\n-----END PRIVATE KEY-----\\n\",\"client_email\":\"tornado-delivery@my-gcp-project.iam.gserviceaccount.com\",\"client_id\":\"...\",\"token_uri\":\"https://oauth2.googleapis.com/token\"}"
  }
}

Setting the storage destination is an organization-level action restricted to Admin and Owner members, and one provider is active at a time. Saving a GCS configuration replaces whatever provider was configured before.

Field constraints

Field	Limit enforced on save	Also validated at job time
`bucket`	255 characters	3 to 63 characters, lowercase letters, digits, hyphens and periods, no consecutive periods, not an IP address
`project_id`	128 characters	Non-empty; a non-standard shape is warned about but accepted
`credentials_json`	10 000 characters	Must parse as JSON, `type` must be `service_account`, must contain `project_id`, `private_key` and `client_email`

A normal service account key is well under 3 KB, so the 10 000 character limit only bites if you paste something that is not a single key file.

Verify it for real

Because the connection test never contacts Google, the first honest signal is a delivered file. Run one small job and look in the bucket.

Submit a short download

curl -X POST "https://api.tornadoapi.io/jobs" \
  -H "x-api-key: sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "max_resolution": "360"
  }'

Wait for the job to finish, then list the bucket

gcloud storage ls -r gs://tornado-downloads/videos/

Delivered files land under the base folder, which defaults to videos/.

Then test a large file

A small file may use only the simple-upload path. Run a second job that produces at least 100 MiB (a longer video at a higher resolution) and verify the large-file mode used by your deployment. Successful simple upload does not prove composite or XML multipart permissions.

Then test a read

Fetch the delivered object with the service account’s own credentials, which is the same authorisation a signed download URL uses:

gcloud auth activate-service-account --key-file=tornado-delivery.json
gcloud storage cp gs://tornado-downloads/videos/<delivered-file> /tmp/check.mp4

A 403 here can indicate missing storage.objects.get or another access-policy restriction. Also open the actual delivery link from the recipient’s network; a successful command-line read alone does not prove recipient access.

Verify both the object and its delivered URL. The worker uses the actual GCS backend to create the signed URL, stores it with the completed job, and includes it as s3_url in the per-job completion webhook. Job polling can return that recorded URL while it is fresh. The legacy field name does not imply an S3 destination. A missing or expired link is not proof that upload failed: also check the object in your bucket.Organization-level custom webhooks sent by the dashboard are a separate path: that dispatcher currently presigns S3/Azure only, so its download_url is not available for GCS.

If nothing arrives at all, check which configuration the job used before you suspect IAM. The job pipeline resolves storage per API key, from the configuration set through POST /user/gcs with that key’s x-api-key, falling back to the organization default and then to Tornado’s managed storage. A job that ran against managed storage still reports Completed, with the file in Tornado’s bucket rather than yours. See Custom Storage.

Troubleshooting

The dashboard’s connection test cannot produce any of these errors, because it makes no request. They surface on the first job instead. The worker treats 401, 403, 404, invalid_grant and Invalid JWT as permanent and fails the job immediately without retrying. When Google’s response identifies the cause, the job’s error says exactly what to fix, and failure_kind classifies it (the job status is failed):

`failure_kind`	Cause	What the error message gives you
`destination_permission`	The service account lacks a permission on the bucket	The service account, the missing permission (for example `storage.objects.create`, or `storage.objects.delete` when an object name is overwritten) and the narrowest predefined role that grants it
`destination_not_found`	The bucket does not exist, or the service account cannot see it	The bucket name and service account to check
`destination_credentials`	Google rejected the key: deleted, rotated, disabled or malformed	Google’s reason, for example `Invalid JWT Signature.`
`destination_billing`	The project that owns the bucket cannot accept uploads, for example billing disabled	Google’s message

Other permanent refusals keep the generic message “Invalid GCS storage credentials. Please check your service_account_json, project_id, and bucket name.”, which covers several distinct causes.

Symptom	Cause	Fix
Simple uploads deliver, composite uploads fail	One possible cause is missing `storage.objects.get` for `objects.compose`. XML multipart has different operation requirements.	Inspect the failing operation and grant the permissions for the upload mode in use.
Files deliver, but download links return 403	The role lacks `storage.objects.get`. Signing succeeds offline; the reader’s GET is what gets rejected.	Grant `roles/storage.objectUser` on the bucket.
Re-running a job that writes an object name that already exists fails	Overwriting an existing object requires `storage.objects.delete` in addition to `storage.objects.create`.	Grant `roles/storage.objectUser` on the bucket.
`verify_credentials.txt` or `.tornado.tmp.` objects accumulate in the bucket	Cleanup can fail because of missing delete permission, retention rules or an interrupted upload.	Check the failure and retention policy. Remove only confirmed abandoned Tornado objects after their uploads have stopped; a matching name alone does not prove an object is unused.
Every job fails immediately, nothing appears in the bucket	The service account may lack effective access to this bucket, or the key may be invalid. A project grant can be inherited by its buckets; a grant on an unrelated project does not authorize this one.	Check the service-account identity, key and effective bucket IAM, including inherited policies.
Every job fails immediately, and the bucket exists	The bucket name or service-account permissions may be wrong. The service account and bucket can belong to different projects when IAM grants access.	Confirm the bucket name and the service account’s effective permissions; do not require matching project ownership as a substitute for IAM.
Configuration saved green, every job fails on credentials	The pasted JSON is not a complete, valid key file: a partial paste, a base64 blob, or `private_key` newlines mangled in transit. The dashboard test does not parse the JSON, so this is accepted at save time.	Re-copy the whole file, opening brace to closing brace, into the field.
Jobs that used to work start failing on credentials	The key was disabled or deleted in the console, or the service account itself was deleted or disabled.	Create a new key and update the configuration. Note that deleting a key does not revoke short-lived tokens already issued from it, so a revoked key can appear to keep working briefly.
`Service account JSON 'type' must be 'service_account'`	An OAuth client secret, an external account configuration, or an `authorized_user` credential was pasted instead of a service account key.	Download the key from the service account’s Keys tab, not from APIs & Services › Credentials.
Key creation is refused in the console	`constraints/iam.disableServiceAccountKeyCreation` is enforced on the project, or another organization policy blocks key creation.	Ask an organization administrator to grant an exception for the project.

Rotating the key

A service account can hold more than one key at a time, so overlap them and you rotate without an outage.

Create a second key

Add a new JSON key on the same service account. Both keys are now valid.

Update Tornado and confirm

Paste the new key into the dashboard and run a test job through to the bucket.

Disable, then delete, the old key

Disable the old key first and watch for failures before deleting it. Deletion is permanent.

Rotation also invalidates links you have already handed out. Google’s documentation is explicit that a signed URL grants access “until the expiration time for the URL is reached or the key used to sign the URL is rotated”, so retiring a key cuts short every outstanding link signed with it, ahead of its 24 hour expiry. Rotate when those links are expendable, or expect recipients to need fresh ones.

Not available on this route

Feature	Status
Live credential validation from the dashboard	Not implemented. `POST /settings/storage/test` returns `verified: false` for GCS and makes no network call. The per-API-key endpoint `POST /user/gcs` does validate for real, by uploading and deleting a `verify_credentials.txt` probe object.
Reading the configuration back	Only through `GET /settings/storage`, with `credentials_json` masked. There is no `GET /user/gcs`.
`folder_prefix` / `base_folder`	Not accepted. Available on `POST /user/gcs` only. Deliveries through the dashboard route use the default `videos/` base folder.
Base64-encoded credentials	Not accepted. The value is parsed directly as JSON.
Workload Identity Federation, attached service accounts, impersonation	Not supported. A downloaded service account key is the only credential type.
HMAC keys / selecting XML from the dashboard	No HMAC key fields or customer-facing XML toggle. Workers can optionally use XML multipart with the same service-account credential; the default file path uses the JSON API.
Customer-supplied encryption keys (CSEK)	Not supported. Customer-managed keys (CMEK) configured as the bucket default also require the appropriate Google-side encryption-key permissions; verify with a real delivery.
Automatic bucket creation	Not supported. Create the bucket first.

API Reference

Jobs

Bulk

Metadata

Batch Operations

User

Overview

Current limitations

Before you start

1. Create the bucket

Choosing a location

2. Create the service account

3. Grant the role on the bucket

What Tornado actually does in your bucket

Minimum

Recommended

Granting it

4. Create and download the JSON key

5. Enter the three fields

Pasting the key

Field constraints

Verify it for real

Troubleshooting

Rotating the key

Not available on this route

​Overview

​Current limitations

​Before you start

​1. Create the bucket

​Choosing a location

​2. Create the service account

​3. Grant the role on the bucket

​What Tornado actually does in your bucket

​Minimum

​Recommended

​Granting it

​4. Create and download the JSON key

​5. Enter the three fields

​Pasting the key

​Field constraints

​Verify it for real

​Troubleshooting

​Rotating the key

​Not available on this route

Overview

Current limitations

Before you start

1. Create the bucket

Choosing a location

2. Create the service account

3. Grant the role on the bucket

What Tornado actually does in your bucket

Minimum

Recommended

Granting it

4. Create and download the JSON key

5. Enter the three fields

Pasting the key

Field constraints

Verify it for real

Troubleshooting

Rotating the key

Not available on this route