Google Cloud Pub/Sub Core Knowledge
Full Reference: See advanced.md for Java/Python/Go producer patterns, Spring Cloud GCP consumer, push subscription handlers, and Terraform for DLT, IAM, VPC Service Controls, and monitoring alerts.
Deep Knowledge: Use mcp__documentation__fetch_docs with technology: google-pubsub for comprehensive documentation.
Quick Start (Emulator)
# docker-compose.yml
services:
pubsub:
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators
command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8085
ports:
- "8085:8085"
docker-compose up -d
# Set emulator env
export PUBSUB_EMULATOR_HOST=localhost:8085
# Create topic and subscription
gcloud pubsub topics create orders
gcloud pubsub subscriptions create order-processor --topic=orders
# Test
gcloud pubsub topics publish orders --message='{"id":"123"}'
gcloud pubsub subscriptions pull order-processor --auto-ack
Core Concepts
| Concept | Description |
|---|
| Topic | Named resource for publishing messages |
| Subscription | Named resource for receiving messages |
| Publisher | Sends messages to topic |
| Subscriber | Receives messages from subscription |
| Ack Deadline | Time to acknowledge before redelivery |
| Message Retention | How long unacked messages are kept |
Delivery Types
| Type | Description | Use Case |
|---|
| Pull | Subscriber requests messages | Batch processing, variable load |
| Push | Pub/Sub sends to endpoint | Serverless, webhooks |
| BigQuery | Direct export to BigQuery | Analytics pipelines |
| Cloud Storage | Direct export to GCS | Data archival |
Architecture
Publisher ──▶ Topic ──▶ Subscription 1 (Pull) ──▶ Subscriber
──▶ Subscription 2 (Push) ──▶ Cloud Run
──▶ Subscription 3 ──▶ BigQuery
Node.js Producer (@google-cloud/pubsub)
import { PubSub } from '@google-cloud/pubsub';
const pubsub = new PubSub({
projectId: 'my-project',
});
const topic = pubsub.topic('orders');
// Publish single message
const messageId = await topic.publishMessage({
data: Buffer.from(JSON.stringify(order)),
attributes: {
'correlation-id': correlationId,
'order-type': order.type,
},
orderingKey: order.customerId, // For ordered delivery
});
// Batch publishing (automatic batching)
const publishOptions = {
batching: {
maxMessages: 100,
maxMilliseconds: 10,
},
};
const batchTopic = pubsub.topic('orders', publishOptions);
Node.js Consumer (Pull Subscription)
const subscription = pubsub.subscription('order-processor', {
flowControl: {
maxMessages: 100,
maxExtensionMinutes: 10,
},
ackDeadline: 30,
});
const messageHandler = async (message) => {
try {
const order = JSON.parse(message.data.toString());
const correlationId = message.attributes['correlation-id'];
await processOrder(order);
message.ack();
} catch (error) {
console.error('Processing failed:', error);
message.nack(); // Will be redelivered
}
};
subscription.on('message', messageHandler);
subscription.on('error', (error) => console.error('Subscription error:', error));
// Graceful shutdown
process.on('SIGTERM', async () => {
await subscription.close();
});
Dead Letter Topics
# Create DLT
gcloud pubsub topics create orders-dlq
gcloud pubsub subscriptions create orders-dlq-sub --topic=orders-dlq
# Create subscription with DLT
gcloud pubsub subscriptions create order-processor \
--topic=orders \
--dead-letter-topic=orders-dlq \
--max-delivery-attempts=5
When NOT to Use This Skill
Use alternative messaging solutions when:
- AWS-native architecture - SQS has better AWS integration
- Azure-native architecture - Use Azure Service Bus
- Event streaming with replay - Use Dataflow or Kafka
- On-premise deployment - Use RabbitMQ or ActiveMQ
- Multi-cloud portability - Use Kafka or RabbitMQ
- Complex routing patterns - RabbitMQ provides more flexibility
- JMS compliance required - Use ActiveMQ
Anti-Patterns
| Anti-Pattern | Why It's Bad | Solution |
|---|
| No dead letter topic | Failed messages lost | Configure DLT for all subscriptions |
| Short ack deadline | Duplicate processing | Set deadline > max processing time |
| No retry policy | Immediate redelivery on failure | Configure exponential backoff |
| Synchronous publish | Poor throughput | Use batching and async publish |
| Pull without flow control | Consumer overwhelmed | Set max_messages limit |
| No message ordering when needed | Out of order processing | Use ordering keys |
| Large message payloads | Higher costs, poor performance | Use Cloud Storage with reference |
| No IAM least privilege | Security risk | Use service accounts with minimal roles |
Quick Troubleshooting
| Issue | Likely Cause | Fix |
|---|
| Messages not received | No subscription or wrong topic | Create subscription, verify topic |
| Duplicate messages | Ack deadline expired | Increase ack deadline or process faster |
| Messages in DLT | Max delivery attempts exceeded | Check processing logic, review DLT |
| Permission denied | Missing IAM roles | Grant Publisher/Subscriber roles |
| Ordering not working | No ordering key or wrong subscription | Set ordering key, enable message ordering |
| High latency | Batching delay or network | Reduce batch delay, check network |
| Push subscription failing | Endpoint down or auth failure | Check endpoint health, verify auth |
| Backlog growing | Slow consumers | Add consumers or optimize processing |
Production Readiness
Monitoring Metrics
| Metric | Alert Threshold |
|---|
| num_undelivered_messages | > 10000 |
| oldest_unacked_message_age | > 3600s |
| num_outstanding_messages | > 10000 |
| dead_letter_message_count | > 0 |
| publish_latencies | p99 > 1s |
Checklist
- IAM roles with least privilege
- Service account per component
- Dead letter topic configured
- Retry policy configured
- Ack deadline appropriate
- Message retention set
- Monitoring alerts configured
- Schema validation (if needed)
- Message ordering (if needed)
- VPC Service Controls (if needed)
Reference Documentation
Deep Knowledge: Use mcp__documentation__fetch_docs with technology: google-pubsub for comprehensive documentation.
Available topics: basics, producers, consumers, production
