GoogleHelpit_
$ gcloud troubleshoot

# GCP Troubleshooting

The errors people actually hit on Google Cloud: symptom → cause → fix → official docs. Search or filter by area, then click a card to expand.

Catch these before they bite: gcphelpit is our free open-source CLI that scans a snapshot of your Google Cloud project for security, IAM, cost, and reliability issues, each with a plain-English fix. See how it compares to other GCP scanners.
all billing iam / permissions quotas apis auth / gcloud compute networking storage serverless gke bigquery databases logging / monitoring terraform / iac
// no matching fixes — try a different term or category.

💸 Billing & cost surprises

▸"This API method requires billing to be enabled" / project has no billing account
Cause
The project isn't linked to an active Cloud Billing account, so paid services (and even some API calls) are blocked.
Fix
Console → Billing → Link a billing account, or run:
gcloud billing projects link PROJECT_ID --billing-account=XXXXXX-XXXXXX-XXXXXX
Verify with gcloud billing projects describe PROJECT_ID.
Docs
cloud.google.com/billing/docs/how-to/modify-project
▸Unexpected charges / bill much higher than expected
Cause
Always-on resources — idle VMs, static IPs on stopped VMs, load balancers, provisioned disks, NAT gateways, logging — keep billing even when "not used."
Fix
Billing → Reports to find the top service/SKU; delete or stop idle resources; release unused static IPs (charged when unattached); act on cost recommendations in the Recommender.
Docs
cloud.google.com/billing/docs/how-to/reports
▸A budget was set but spending wasn't capped
Cause
Budgets are alerting only — they never stop usage or billing automatically.
Fix
Create budgets with threshold alerts (Billing → Budgets & alerts). For a true hard stop, wire the budget's Pub/Sub notification to a Cloud Function that disables billing.
Docs
budgets · disable billing with notifications
▸Free-tier / $300 trial confusion (charged after trial or over limits)
Cause
The $300 credit expires after 90 days; "Always Free" tiers have monthly caps that don't roll over and exclude many regions/sizes.
Fix
Check per-product free limits before deploying; keep VMs to the eligible e2-micro region/size; monitor the credit balance in Billing.
Docs
cloud.google.com/free/docs/free-cloud-features
▸Budget alert emails not arriving
Cause
Alerts default to Billing Account Admins/Users; custom recipients or Pub/Sub/Monitoring channels weren't configured.
Fix
Edit the budget → Manage notifications → add email channels via Cloud Monitoring, or add members with a Billing role.
Docs
cloud.google.com/billing/docs/how-to/budgets

🔐 IAM & permissions

▸"PERMISSION_DENIED: <email> does not have <service>.<resource>.<verb> access"
Cause
The principal lacks a role containing that specific permission (or a deny policy blocks it).
Fix
Read the exact missing permission from the error, then grant a role that includes it:
gcloud projects add-iam-policy-binding PROJECT_ID \ --member=user:EMAIL --role=roles/ROLE
Use the suggested-roles list Google prints in the error.
Docs
iam/docs/permission-error-messages
▸"The caller does not have permission"
Cause
Same as above but often on the service agent or a parent resource (folder/org), or you're authenticated as the wrong identity.
Fix
Confirm the active identity with gcloud auth list, then use the Policy Troubleshooter to see which policy grants/denies access.
Docs
iam/docs/troubleshooting-access
▸Confused about roles vs. permissions
Cause
You can't grant a single permission directly — permissions are bundled into roles (basic, predefined, custom).
Fix
Prefer predefined roles scoped to the service; create a custom role only if none fits. Grant at the lowest resource level needed.
Docs
iam/docs/understanding-roles
▸Service account "does not have permission" when running code
Cause
The runtime service account (default Compute/Cloud Run SA) lacks roles for the resources the code touches.
Fix
Identify the SA the workload runs as and grant it the needed role(s). Prefer attaching a dedicated least-privilege SA over the default one.
Docs
best-practices-service-accounts
▸Action blocked by an Organization Policy (can't create SA keys / allow public access / share externally)
Cause
An org policy constraint (e.g. iam.disableServiceAccountKeyCreation, iam.allowedPolicyMemberDomains, storage.publicAccessPrevention) overrides IAM.
Fix
An Org Policy Admin must adjust/override the constraint at project/folder level, or use the allowed alternative (e.g. Workload Identity instead of SA keys).
Docs
organization-policy/using-constraints
▸IAM change "didn't work" immediately
Cause
IAM policy changes are usually fast but can take up to ~7 minutes to propagate.
Fix
Wait a few minutes and retry before deeper debugging.
Docs
iam/docs/access-change-propagation

📊 Quotas & limits

▸"Quota exceeded" / RESOURCE_EXHAUSTED (CPUs, IN_USE_ADDRESSES, …)
Cause
You hit a per-project, per-region quota — many default low, especially for new/trial accounts.
Fix
IAM & Admin → Quotas & System Limits → filter by the service/metric → select → Edit quotas to request an increase. Deleting/stopping resources frees region-scoped quota immediately.
Docs
quotas/view-manage · troubleshoot
▸Quota increase denied / trial account can't raise quota
Cause
Free-trial billing accounts have restricted quotas (e.g. 8 vCPUs) and limited increase eligibility.
Fix
Upgrade to a paid/self-serve billing account, then re-request the increase.
Docs
cloud.google.com/docs/quotas/troubleshoot
▸Surprise quota hit at scale
Cause
No proactive monitoring of usage vs. limit.
Fix
Set up Cloud Quotas alerts to notify at a % of the limit.
Docs
quotas/set-up-quota-alerts

🔌 APIs not enabled

▸"API [name] has not been used in project N before or it is disabled"
Cause
The required API isn't enabled in the project (each API must be enabled once).
Fix
Click the URL in the error, or run:
gcloud services enable SERVICE.googleapis.com
Wait a few minutes for propagation, then retry.
Docs
service-usage/docs/enable-disable
▸Can't enable an API ("permission denied to enable service")
Cause
You lack serviceusage.services.enable (Service Usage Admin).
Fix
Get granted roles/serviceusage.serviceUsageAdmin, or ask an admin to enable it.
Docs
service-usage/docs/access-control

🔑 Authentication & gcloud CLI

▸Commands run against the wrong project
Cause
The gcloud core/project property points to a different project than intended.
Fix
Check and set it:
gcloud config get-value project gcloud config set project PROJECT_ID
Or add --project= per command. Use named configurations for multiple projects.
Docs
sdk/docs/configurations
▸"Could not automatically determine credentials" (ADC missing)
Cause
Client libraries use Application Default Credentials, which aren't set up locally.
Fix
Run:
gcloud auth application-default login
This is separate from gcloud auth login (which only authenticates the CLI). Set a quota project if prompted.
Docs
authentication/provide-credentials-adc
▸Did "gcloud auth login" but libraries still fail
Cause
gcloud auth login authenticates the CLI only — not the ADC used by SDKs.
Fix
Also run gcloud auth application-default login. Remember the two are different.
Docs
authentication/troubleshoot-adc
▸Reauthentication / expired credentials on the CLI
Cause
Tokens expired, or the org requires periodic reauth.
Fix
Run gcloud auth login again. For automation, use gcloud auth activate-service-account or attached service accounts.
Docs
sdk/gcloud/reference/auth/login
▸Service-account-key security warnings / leaked keys
Cause
Long-lived JSON SA keys are risky and often blocked by org policy.
Fix
Prefer Workload Identity Federation, attached service accounts, or impersonation instead of downloaded keys.
Docs
best-practices-for-managing-service-account-keys

🖥️ Compute Engine

▸Can't SSH into a Linux VM (timeout / connection refused / permission denied)
Cause
Missing firewall rule for TCP 22, SSH keys not propagated, guest environment not running, or OS Login misconfig.
Fix
Try the console SSH-in-browser first; ensure an ingress rule allows tcp:22 from your IP (or 35.235.240.0/20 for IAP); run the built-in SSH troubleshooting tool; check the serial console.
Docs
troubleshooting-ssh-errors
▸"Permission denied (publickey)" via terminal SSH
Cause
Key mismatch, or OS Login is enabled but your account lacks the OS Login role.
Fix
Grant roles/compute.osLogin; or add your SSH key to instance/project metadata if not using OS Login.
Docs
troubleshoot-os-login
▸VM won't start / stuck / "resources not available in zone"
Cause
Zonal stockout for the machine type, or quota exhausted.
Fix
Try another zone/region or a different machine type; check quota; review the VM's serial console for boot errors.
Docs
troubleshooting-using-serial-console
▸Can't reach the VM's web app from the internet
Cause
No external IP, or no firewall rule opening the app port (80/443).
Fix
Assign an external IP (or use a load balancer); add an ingress allow rule for the port and target tags/service account.
Docs
firewall/docs/using-firewalls
▸Lost external IP after stopping/starting the VM
Cause
Ephemeral external IPs are released on stop.
Fix
Promote/reserve a static external IP and attach it. Note: reserved-but-unattached IPs are billed.
Docs
reserve-static-external-ip-address
▸"No space left on device" / disk full on a running VM
Cause
The boot/data disk filled up; unlike some clouds, resizing the disk in the console does not auto-grow the filesystem.
Fix
Resize the disk (Compute → Disks → Edit → increase size — this is online), then grow the partition/filesystem inside the guest (growpart + resize2fs/xfs_growfs). Clean logs/caches to buy time.
Docs
disks/resize-persistent-disk
▸Metadata / startup script changes didn't take effect
Cause
Startup scripts run at boot; editing metadata on a running VM doesn't re-run them.
Fix
Reset/reboot the VM to re-run the startup script, or run it manually. Check execution in the serial console output.
Docs
instances/startup-scripts/linux

🌐 Networking / VPC

▸"Connection timed out" / "connection refused" between resources
Cause
Firewall rule missing/misordered, wrong target tags, service not listening, or wrong port.
Fix
Confirm an ingress allow rule matches source range, protocol, and port and applies to the target; verify the service listens on that port/interface. Use Connectivity Tests to pinpoint.
Docs
vpc/docs/firewalls · connectivity tests
▸Private VM (no external IP) can't reach the internet (apt/pip timeouts)
Cause
VMs without external IPs need Cloud NAT for outbound internet; egress may also be firewalled.
Fix
Create a Cloud Router + Cloud NAT gateway for the subnet's region; ensure no egress deny rule blocks traffic (egress rules apply before NAT).
Docs
nat/docs/gce-example · troubleshooting
▸VPC Peering set up but resources still can't talk
Cause
Peering isn't transitive, subnet CIDRs overlap, or firewall rules don't allow the peer range; both sides must create the peering.
Fix
Create peering on both VPCs, ensure non-overlapping subnets, and add firewall rules allowing the peer's ranges.
Docs
vpc/docs/vpc-peering
▸Default firewall behavior confusion
Cause
VPC default is deny ingress / allow egress; the "default" network's implied allow rules may not exist on custom VPCs.
Fix
Explicitly create needed allow rules (allow-ssh, allow-internal, allow-http). Use firewall rule logging to debug.
Docs
firewall/docs/firewalls

🪣 Cloud Storage

▸Can't make a bucket/object public ("Public access prevention is enforced")
Cause
Public Access Prevention is on (bucket, or inherited from org policy), blocking allUsers/allAuthenticatedUsers grants.
Fix
If policy allows, disable PAP on the bucket, then grant roles/storage.objectViewer to allUsers. If enforced by org policy, an admin must change it.
Docs
public-access-prevention · making-data-public
▸"AccessDenied" / 403 on uploads or reads
Cause
Principal lacks the right Storage role, or the bucket uses uniform bucket-level access (ACLs disabled).
Fix
Grant roles/storage.objectAdmin/objectViewer at bucket level; with uniform access, use IAM (not object ACLs).
Docs
access-control/iam · uniform-bucket-level-access
▸Uploads failing / general 4xx–5xx from Storage
Cause
Wrong bucket name, missing permission, or CORS/precondition issues for browser uploads.
Fix
Follow the Storage troubleshooting matrix; for browser uploads, configure CORS on the bucket.
Docs
storage/docs/troubleshooting
▸Accidental public / exposed bucket
Cause
Legacy ACLs or allUsers grants left data open.
Fix
Enable uniform bucket-level access + public access prevention; audit with the IAM policy analyzer.
Docs
using-public-access-prevention

⚡ Serverless — Cloud Run, Functions, App Engine

▸Cloud Run: "container failed to start and listen on the port defined by the PORT environment variable"
Cause
The app doesn't listen on 0.0.0.0:$PORT (hardcoded port, bound to 127.0.0.1, or startup crash/timeout).
Fix
Read PORT from env and bind to 0.0.0.0; test the image locally; check Cloud Logging for the startup stack trace; raise the startup timeout if init is slow.
Docs
run/docs/troubleshooting · container-contract
▸Cloud Run / Functions: cold starts / slow first request
Cause
New instances spin up on demand; heavy init or large images increase latency.
Fix
Set --min-instances to keep instances warm; slim the image; move heavy init to global scope. (Min instances incur cost.)
Docs
configuring/min-instances
▸Cloud Run: 403 "Forbidden" when calling the service
Cause
Service requires authentication (default) and the caller isn't an invoker.
Fix
For public access, --allow-unauthenticated (grants roles/run.invoker to allUsers); for service-to-service, grant the calling SA roles/run.invoker and send an ID token.
Docs
authenticating/public · service-to-service
▸Cloud Functions: deploy fails at build (dependency or entry-point error)
Cause
Missing/incorrect entry point name, dependency install failure, or wrong runtime.
Fix
Match --entry-point to the exported function name; verify the dependency manifest; read the Cloud Build logs linked in the error.
Docs
functions/docs/troubleshooting
▸Deploy: "Build failed / permission denied" on Cloud Build or Artifact Registry
Cause
The Cloud Build or default compute service agent lacks required roles, or the Cloud Build/Artifact Registry APIs aren't enabled.
Fix
Enable Cloud Build + Artifact Registry APIs; ensure the build SA has needed roles (Google usually auto-grants; re-grant if removed).
Docs
build/docs/troubleshooting
▸App Engine: `gcloud app deploy` fails (region locked / app not created)
Cause
An App Engine app must be created once per project and its region can't be changed later.
Fix
Run gcloud app create --region=REGION before the first deploy; pick the region carefully (permanent).
Docs
appengine/docs/…/creating-project
▸App Engine: 502 / scaling / startup errors after deploy
Cause
App not listening on $PORT/8080, failing health checks, or missing runtime config.
Fix
Bind to the provided port, verify app.yaml, and check Logs Explorer for the failing request.
Docs
appengine/docs/standard/troubleshooting

☸️ Kubernetes / GKE

▸Pod stuck in ImagePullBackOff / ErrImagePull
Cause
Wrong image name/tag, or the node/cluster SA can't authenticate to the registry (Artifact Registry).
Fix
Verify the image path and tag; grant the node service account roles/artifactregistry.reader; confirm the image exists in the region.
Docs
troubleshooting/image-pulls
▸Pod in CrashLoopBackOff
Cause
The container starts then exits repeatedly (app error, bad command, missing config/secret, failing liveness probe).
Fix
Inspect the crash:
kubectl logs POD --previous
Fix the command/env/probe; check the exit code.
Docs
troubleshooting/crashloopbackoff-events
▸Pod Pending — "Insufficient cpu/memory" / won't schedule
Cause
No node has enough allocatable resources for the pod's requests, or the node pool is too small.
Fix
Lower resource requests, scale/enable the node pool, or turn on cluster autoscaling.
Docs
troubleshooting/deployed-workloads
▸kubectl can't connect / "You must be logged in" after cluster create
Cause
kubeconfig credentials not fetched, or missing the gke-gcloud-auth-plugin.
Fix
Fetch credentials and install the plugin:
gcloud container clusters get-credentials CLUSTER --region REGION
Docs
how-to/cluster-access-for-kubectl
▸Service LoadBalancer stuck in <pending> / no external IP
Cause
LB provisioning in progress, external-IP quota, or annotation/subnet misconfig.
Fix
Wait a few minutes; check IN_USE_ADDRESSES quota and events with kubectl describe svc.
Docs
concepts/service-load-balancer
▸403 / permission denied creating or managing GKE
Cause
Missing roles/container.admin/developer, or the GKE API/service agent isn't set up.
Fix
Enable the Kubernetes Engine API; grant appropriate container roles.
Docs
kubernetes-engine/docs/how-to/iam

🗄️ BigQuery

▸"Not found: Table/Dataset" even though it exists
Cause
Wrong project/dataset qualification, or a region mismatch — you can't query a dataset in US from a job running in EU.
Fix
Fully-qualify as `project.dataset.table`; confirm the dataset's location and run the job in the same region; check for typos/case (dataset IDs are case-sensitive).
Docs
bigquery/docs/error-messages
▸Query scanned way more data than expected / surprise cost
Cause
On-demand pricing bills by bytes scanned. SELECT *, no partition filter, or querying a huge table scans everything.
Fix
Select only needed columns; filter on the partition column; check the estimate in the UI (dry-run bytes) before running; set a Maximum bytes billed limit on the query to hard-cap it.
Docs
bigquery/docs/best-practices-costs
▸"Access Denied: BigQuery ... permission bigquery.jobs.create"
Cause
To run any query you need job-creation permission in the billing project, plus data access on the dataset — they're separate grants.
Fix
Grant roles/bigquery.jobUser on the project (to run jobs) and roles/bigquery.dataViewer on the dataset (to read data). bigquery.user covers both for own datasets.
Docs
bigquery/docs/access-control

🛢️ Cloud SQL & databases

▸App / Cloud Run can't connect to Cloud SQL
Cause
Trying to reach a private IP with no connectivity, or a public IP with no authorized network — plus the connector service often isn't wired up.
Fix
For Cloud Run, add the Cloud SQL connection (--add-cloudsql-instances) and connect via the Unix socket, or use the Cloud SQL Auth Proxy / connector. For private IP, use a Serverless VPC connector. Grant the runtime SA roles/cloudsql.client.
Docs
sql/docs/connect-run · auth proxy
▸Cloud SQL: "too many connections" / connection pool exhausted
Cause
Serverless scale-out opens many connections; each instance's pool multiplies past the DB's max_connections.
Fix
Use a small per-instance pool, cap Cloud Run --max-instances, raise max_connections via a database flag, or front the DB with a connection pooler (e.g. PgBouncer).
Docs
sql/docs/manage-connections

📈 Logging & Monitoring

▸Logs aren't showing up in Logs Explorer
Cause
Wrong time range/resource filter, a log exclusion/sink dropping them, or the workload's SA lacks roles/logging.logWriter.
Fix
Widen the time range and clear filters; check Logging → Log Router for exclusions; ensure the writing SA has Logs Writer. For VMs, confirm the Ops Agent is installed and running.
Docs
logging/docs/view/logs-explorer · log router
▸Alerting policy / uptime check never fires (or fires constantly)
Cause
No notification channel attached, threshold/duration misconfigured, or the uptime check can't reach a private endpoint through the firewall.
Fix
Attach a notification channel to the policy; tune the condition threshold + duration window; for uptime checks, allow Google's uptime-checker source ranges to the target.
Docs
monitoring/alerts · uptime-checks
▸Logging bill unexpectedly high
Cause
High-volume logs (data-access audit logs, verbose app logs) ingested beyond the free allotment.
Fix
Add exclusion filters in the Log Router for noisy, low-value logs; keep only what you need in _Default; route long-term logs to a cheaper bucket/BigQuery sink.
Docs
logging/docs/routing/overview

🧱 Terraform & IaC deploys

▸Terraform apply: "Error 403" or "API not enabled"
Cause
The identity Terraform runs as lacks the role for that resource, or the service's API isn't enabled in the project.
Fix
Read the exact permission/API in the error; enable the API (gcloud services enable … or a google_project_service resource) and grant the running SA the needed role. Re-run after propagation.
Docs
docs/terraform/best-practices
▸"Error acquiring the state lock" (stale lock)
Cause
A previous run crashed and left a lock in the GCS backend, or two runs are racing.
Fix
Make sure no other apply is running, then force-unlock with the lock ID from the error: terraform force-unlock LOCK_ID. Use a GCS backend with per-workspace state to avoid collisions.
Docs
terraform/store-state (GCS backend)
▸Terraform wants to destroy/recreate a resource unexpectedly
Cause
A changed immutable field (name, location) forces replacement, or drift from a manual console change.
Fix
Read the plan's # forces replacement lines; if the resource already exists, terraform import it instead of recreating; use lifecycle { prevent_destroy = true } on critical resources.
Docs
terraform/import
First-response checklist: 1) confirm identity & project (gcloud auth list, gcloud config get-value project) · 2) confirm the API is enabled (gcloud services list --enabled) · 3) read the exact error string — GCP names the missing permission/API/quota · 4) check Logs Explorer · 5) wait for propagation before deeper debugging.