Containers and orchestration platforms promise reproducible deployments, but when things go wrong in cloud environments, debugging can feel like staring into a black box.
Why is a simple microservice container ballooning to 1.5GB? Why does a newly deployed Kubernetes pod get stuck in an endless CrashLoopBackOff cycle when it started fine locally?
Here are four real-world infrastructure issues you will encounter running containers in production, along with how to troubleshoot and fix them.
1. Slashing Docker Image Bloat: Multi-Stage Builds
Bundling compilers, development packages, and source files into production containers bloats image sizes, slows down CI/CD rollouts, and widens your security vulnerability attack surface.
# Stage 1: Install dependencies and compile
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
# Stage 2: Minimal production runtime
FROM node:20-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/package*.json ./
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
USER node
EXPOSE 3000
CMD ["node", "dist/index.js"]Separating build tools from the runtime environment regularly reduces Node image sizes from 1.2GB down to 150MB.
2. Kubernetes Pod Failures: OOMKilled and CrashLoopBackOff
When your Kubernetes dashboard shows red pods, run kubectl describe pod <pod-name> first. You’ll usually find one of these two culprits:
💡 1. Exit Code 137: OOMKilled
The pod suddenly dies under traffic spikes, and restart counts increment.
- Cause: The container attempted to allocate more memory than defined in its
resources.limits.memory. The Linux kernel Out-Of-Memory killer dispatched aSIGKILL (9)to the process (128 + 9 = 137). - Remedy: Adjust container memory limits to realistic peak workloads, and profile application heap allocation to catch uncollected buffers or memory leaks.
💡 2. CrashLoopBackOff Due to Premature Health Checks
The container starts up, runs briefly, and gets killed in an infinite loop.
- Common Gotcha: The application takes 15 seconds to connect to the database and warm up caches, but the
livenessProbestarts checking at 5 seconds. Kubernetes assumes the container is frozen and kills it. - Remedy: Use
startupProbeor adjustinitialDelaySecondsto give the application enough runway to complete initialization before readiness probes begin.
3. Mitigating Cloud Run Cold Starts
GCP Cloud Run scales down to zero instances when idle to save infrastructure costs, but the first incoming request can stall for 2 to 5 seconds during container provisioning.
- Use Lightweight Base Images: Alpine and Distroless images pull significantly faster over internal VPC networks.
- Set Minimum Instances: If budget permits, configure
min-instances: 1on production services so at least one warm instance always handles incoming traffic without cold-start latency. - Defer Heavy Bootstrapping: Don’t synchronously initialize large third-party connections at the global module scope if they can be lazily established on the first request.
4. Securing Supabase with Row Level Security (RLS)
Pairing modern frontend frameworks with Supabase allows teams to ship fast. However, database security hinges entirely on Row Level Security (RLS):
- The Risk: If you create a table and forget to enable RLS, anyone with your public
anon keycan query your entire database table directly from their browser console. - The Rule: Always enable RLS on every table and define explicit policies:
ALTER TABLE user_profiles ENABLE ROW LEVEL SECURITY;
CREATE POLICY "Users can only read own profile"
ON user_profiles
FOR SELECT
USING (auth.uid() = user_id);Start Here
Continue with the core guides that pull steady search traffic.
- Operating Redis, RabbitMQ, and Kafka in Production: Avoiding Common Bottlenecks When middleware stalls, the entire backend grinds to a halt. Practical advice for detecting Redis big keys, managing RabbitMQ DLX queues, and surviving Kafka rebalance storms.
- When Your Vercel Build Breaks: Fixing OOM Crashes, 504 Timeouts, and Missing Envs Why projects build fine on localhost but fail on Vercel. Practical fixes for out-of-memory errors, 504 gateway timeouts, client vs server env vars, and fast rollbacks.
- Building Production AI Agents & RAG: Architecture Lessons & Practical Tips Moving beyond simple chatbots to autonomous AI agents. Key lessons on temperature and sampling, prompt caching, RAG vs fine-tuning tradeoffs, and state management with LangGraph.
Related guides are shown to help you explore more.