A high-performance, modular concurrency engine implemented in Go.
This system is designed around the Communicating Sequential Processes (CSP) model to execute resource-intensive workloads while enforcing strict CPU and memory bounds.
The primary goal is predictable concurrency: no goroutine explosions, no unbounded queues, and no shutdown races.
Limits the maximum number of active goroutines to prevent runaway scheduling, excessive context switching, and out-of-memory (OOM) failures.
Uses sync.WaitGroup and coordination channels to ensure:
- No new tasks are accepted after shutdown begins
- All in-flight tasks complete before process termination
A buffered task queue naturally applies backpressure:
- Producers block when the system is at capacity
- No custom rate-limiting logic required
The WorkerPool interface decouples consumers from implementation details:
- Easy mocking for tests
- Drop-in replacement or extension of pool strategies
A multi-stage Docker build produces a minimal, statically linked binary suitable for production and orchestration environments.
.
├── cmd/workerpool/
│ └── main.go # Application entry point
├── internal/pool/
│ ├── pool.go # Core worker pool logic + interface
│ ├── task.go # Task type definitions
│ └── pool_test.go # Concurrency-focused unit tests
├── build/
│ └── Dockerfile # Multi-stage production build
├── go.mod # Go module definition
└── Makefile # Automation shortcuts
- Go 1.22+
- Docker (optional, for container builds)
Clean dependencies and run the simulation:
go mod tidy
go run cmd/workerpool/main.goThe test suite is executed with the Go race detector enabled to validate memory safety under concurrent load:
go test -v -race ./internal/pool/...Go manages concurrency using a three-layer scheduling system:
-
G (Goroutine)
Lightweight execution unit (2KB initial stack, grows dynamically) -
M (Machine)
An OS thread responsible for executing goroutines -
P (Processor)
Scheduler context holding a local run queue and execution resources
A bounded worker pool stabilizes this system by:
- Limiting runnable goroutines per
P - Reducing scheduler churn
- Keeping local run queues hot and predictable
This improves cache locality and reduces unnecessary context switching.
The taskQueue is implemented as a buffered channel:
- Sending to a full channel blocks the sender (natural backpressure)
- Workers receive tasks sequentially, ensuring bounded parallelism
selectstatements introduce pseudo-random fairness when listening for:- New tasks
- Shutdown signals
This prevents starvation and avoids priority inversion without explicit locks.
The Docker build uses:
CGO_ENABLED=0
This produces a fully statically linked binary, embedding:
- Go runtime
- Garbage collector
- Scheduler
Benefits:
- No dependency on host libc
- Smaller attack surface
- Fully portable binary
In a container, the application runs as PID 1, meaning:
- It receives termination signals directly
- Improper shutdown handling can leak goroutines or lose work
This is why explicit shutdown coordination is mandatory, not optional.
Build and run the minimal production image (≈15MB):
# Build the image
docker build -t worker-pool-app -f build/Dockerfile .
# Run the container
docker run --rm worker-pool-appThe resulting container contains:
- A single statically linked binary
- No shell
- No package manager
- No unnecessary runtime dependencies
This project prioritizes:
- Deterministic concurrency over raw throughput
- Backpressure instead of buffering everything
- Explicit lifecycle management
- Simple primitives (channels, goroutines, interfaces) over frameworks
It is intended as a foundation for:
- Job schedulers
- Background task engines
- Rate-limited service executors
- Infrastructure-level concurrency control
MIT License