Introduction
Managing CI/CD pipelines for a monorepo can quickly become challenging as your project grows. What starts as a simple, single pipeline can evolve into a resource-intensive process that slows down development and increases costs. Here's how I implemented path-based pipeline triggering to solve these challenges.
The Challenge: Monorepo Growing Pains
Our project started with a single pipeline managing all components (frontend, backend, database, infrastructure). As we grew, several issues emerged:
- Inefficient Resource Usage: Every code change triggered a full pipeline run
- Slow Feedback: Developers waited too long for build results
- Agent Pool Congestion: Our Azure DevOps agents were constantly busy
- Increased Costs: Unnecessary pipeline runs consumed build minutes
- Poor Developer Experience: Waiting for pipelines slowed development
We needed to run pipelines only for components that changed, reduce build times, and optimize resources.
The Solution: Path-Based Triggering
I implemented a path-based pipeline triggering strategy with these key components:
1. Component-Specific Pipelines
I created separate pipelines for each major component (frontend, backend, database, infrastructure).
2. Strategic Path-Based Triggers
For each pipeline, I configured path-based triggers:
trigger:
batch: true
branches:
include:
- main
paths:
include:
- component-directory/**
- .azure-pipelines/component-pipeline.yaml
Key elements:
-
Path Patterns: Using
/**to match directories recursively - Self-Referential Inclusion: Including the pipeline file itself
-
Batch Processing: Combining rapid commits with
batch: true
3. Consistent PR Triggers
I applied identical path filtering to PR validations:
pr:
branches:
include:
- main
paths:
include:
- component-directory/**
- .azure-pipelines/component-pipeline.yaml
4. Precise Path Matching
Success required attention to:
- Directory structure alignment
- File extension accuracy (.yaml vs .yml)
- Comprehensive path coverage
Implementation Challenges
My initial attempts had issues:
- I forgot to configure PR triggers, causing PR validations to run all pipelines
- Azure DevOps requires exact path matching, so precision matters
- Finding the right balance of path patterns took iteration
The Results
After implementation, we achieved:
- 70% reduction in pipeline runs
- 3-4x faster feedback cycles for developers
- Improved agent availability for other projects
- Significant cost savings in build minutes
- Better developer experience with faster turnaround
Key Lessons Learned
- Path-Based Triggers Need Precision: Patterns must exactly match your repository structure
- Configure Both CI and PR Triggers: Missing either causes unnecessary pipeline runs
- Include Pipeline Files in Triggers: Ensures pipeline changes trigger validation
- Use Batch Processing: Optimizes performance for busy repositories
Best Practices
- Map Component Boundaries: Understand which directories belong to which components
- Start Broad, Then Refine: Monitor and adjust path patterns based on observations
- Test on a Development Branch First: Avoid disrupting your main branch
- Document Your Approach: Help team members understand the pipeline organization
Conclusion
Path-based pipeline triggering transformed our monorepo CI/CD process from a bottleneck to an efficient system. The approach saved time and resources while enhancing developer productivity.
If you're managing a growing monorepo in Azure DevOps, I recommend exploring path-based triggers. The initial setup requires attention to detail, but the benefits in efficiency, cost savings, and developer satisfaction make it worthwhile.
# Sample Configuration Template
trigger:
batch: true
branches:
include:
- main
paths:
include:
- your-component-directory/**
- .azure-pipelines/your-component-pipeline.yaml
pr:
branches:
include:
- main
paths:
include:
- your-component-directory/**
- .azure-pipelines/your-component-pipeline.yaml
Have you implemented path-based triggering in your DevOps setup? I'd love to hear your experiences!
Top comments (0)