Enterprise Infrastructure Automation Platform

Overview

At Ribbon Communications, provisioning a new Active Directory lab environment took about four weeks: manual configuration, cross-team coordination, and hand-off delays between engineers who each owned a different part of the infrastructure.

I replaced it with a fully automated multi-stack Infrastructure as Code framework on AWS CloudFormation and CodePipeline: 20+ resources across five logical stacks (network, security, compute, data, observability) with cross-stack dependency resolution, JSON schema validation on all inputs, idempotent deployments, automated multi-stack rollback, and scheduled drift detection. Provisioning dropped to under thirty minutes: one config file, one pipeline trigger, zero manual steps.

---

The Problem

Every engineer needing a new environment went through the same ordeal: submit a request, wait for the infrastructure team, then manually configure the VPC, AD domain controllers, security groups and IAM policies, EC2 instances and profiles, RDS parameter groups, and CloudWatch logging, each step sequential, each owned by a different person.

The actual config work was maybe two to three days of expert effort. The other three and a half weeks was coordination overhead: scheduling, approvals, waiting on other teams, re-work when a misconfigured step had to be manually rolled back.

The bottleneck throttled velocity: teams building reliability tooling (like the autonomous SRE agent) shared environments, causing cross-workload interference and unreliable test results. I inherited it as a side constraint of that SRE agent work (I needed reproducible test environments) and chose to solve it properly.

---

Goals

• Cut AD lab provisioning from four weeks of manual work to a single pipeline run under one hour.
• Architect a multi-stack CloudFormation framework with independently deployable stacks: network, security, compute, data, observability.
• Resolve cross-stack dependencies so stacks deploy in order and share outputs without hardcoded IDs.
• Validate all inputs against a JSON schema before any resource is created.
• Make deployments idempotent: update only what changed.
• Roll back automatically on stack failure; detect drift on a schedule and alert on divergence.
• Design for longevity: maintainable by engineers who didn't build it, no tribal knowledge.

---

Technical Architecture

The Multi-Stack Design Decision

The most consequential decision was monolithic template vs. coordinated set of stacks. A monolith is fine for small environments, but for 20+ resources across networking, IAM, compute, database, and observability it creates real operational problems: a single security-group change forces a redeploy of the whole template (including compute/database resources CloudFormation may need to replace); any one resource failure blocks the entire deployment; and layers that change at very different rates (stable network topology vs. frequent security and monitoring edits) get coupled.

The multi-stack architecture splits along natural domain boundaries, each stack owning a coherent resource set and exposing outputs downstream stacks consume.

Stack 1: Network Foundation. VPC, public/private subnets across AZs, internet/NAT gateways, route tables, VPC endpoints. The dependency root; every other stack imports its VPC ID, subnet IDs, and route table ARNs. Changes rarely.

Stack 2: Security and IAM. Per-tier security groups (web, app, database, management), IAM roles/policies, EC2 instance profiles. Depends on Stack 1 via Fn::ImportValue for the VPC ID. IAM changes more often than topology, so isolating it keeps security edits off network and compute.

Stack 3: Compute. EC2 instances for AD domain controllers, app servers, and the jump host: launch templates with AMIs, instance types, AD user-data scripts, and the Stack 2 instance profiles. Depends on Stacks 1 and 2. The most expensive layer to replace (EC2 replacement causes downtime), so it's touched only when compute config changes.

Stack 4: Data. RDS instances with parameter/subnet groups, S3 buckets for artifacts and logs, DynamoDB tables for AD management tooling. Depends on Stacks 1 and 2. Most sensitive to unintended replacement (an RDS replacement deletes the instance), so it carries the most conservative update policy.

Stack 5: Observability. CloudWatch log groups, metric filters, alarms, dashboards, and SNS alert topics. Depends on Stacks 3 and 4 for log-forwarding and alarm targets. Updates independently of the infrastructure it monitors.

Cross-Stack Dependency Resolution

A stack declares an Output with an Export key; a downstream stack pulls it via Fn::ImportValue. This is a hard dependency: CloudFormation blocks deletion of a stack whose exports are still imported, and guarantees exported values exist before the importer deploys.

CodePipeline deploys in dependency order: Stack 1, then Stacks 2 and 3 in parallel (both depend only on Stack 1), then Stack 4, then Stack 5, the parallel step shaving each run.

Export names follow a strict convention, {environment}-{stack-name}-{resource-type}-{qualifier} (e.g. prod-network-vpc-id, dev-security-web-sg-id), making imports self-documenting and preventing collisions when dev, staging, and prod share one account.

JSON Schema Validation

A single JSON config file drives each deployment, specifying everything that varies between environments: name, region, VPC and subnet CIDRs, EC2 instance types, RDS instance class, AD domain name, alert emails. Before the pipeline runs, a validation stage checks it against a JSON Schema enforcing:

• Type correctness. CIDRs match the CIDR pattern; instance types come from an approved enum; environment names match the naming convention.
• Range constraints. Numeric values like retention periods have min/max bounds.
• Cross-field consistency. Subnet CIDRs must fall within the VPC CIDR (a custom check atop JSON Schema).
• Required fields. Every mandatory parameter must be present; optional ones with no default must be provided or explicitly null.

Failures abort before any AWS API call, naming the field, value, and violated constraint, so a bad VPC CIDR fails immediately, not 15 minutes and a dozen resources into deployment. This eliminates a whole class of mid-deployment failures that previously required manual rollback.

Idempotent Deployments

A CloudFormation stack update applies only the change-set delta. But true idempotency needs the pipeline to detect whether a stack should be created, updated, or left alone. The CodeBuild stage checks this per stack:

1. Describe the stack to get its current status
2. Doesn't exist → create-stack
3. Stable state (CREATE_COMPLETE, UPDATE_COMPLETE) → compute a change set; empty, skip; non-empty, execute update-stack
4. Failed state (ROLLBACK_COMPLETE, UPDATE_ROLLBACK_COMPLETE) → delete the failed stack and recreate
5. Transition state (CREATE_IN_PROGRESS, UPDATE_IN_PROGRESS) → wait for completion before proceeding

So the same pipeline behaves correctly everywhere: a no-op against an unchanged environment, a delta-only update after a config change, a full build against a fresh account.

Automated Rollback

CloudFormation's --on-failure ROLLBACK handles single-stack rollback, and update failures auto-revert. But across five stacks, a failure in Stack 3 leaves Stacks 1 and 2 in their new state: a version mismatch.

The CodeBuild script tracks which stacks were updated this run and, on failure, rolls them back in reverse dependency order (Stack 3, then 2, then 1), returning the environment to its pre-run state.

Scheduled Drift Detection

Drift, divergence between declared IaC and actual resource state, happens through manual console/CLI changes, AWS automatic changes (cert renewals, security patches), or other automation.

A CloudWatch Events rule runs detect-stack-drift against all five stacks daily, comparing each resource against its last-known template. Detected drift raises a CloudWatch alarm and an SNS notification with a structured report: which stacks drifted, which resources, what properties changed.

Drift is not auto-remediated: the manual change may have been intentional. The alert surfaces it so a human decides whether to accept it (update the template) or revert it (re-run the pipeline).

---

Key Technical Challenges

Challenge 1: The Monolith vs. Multi-Stack Decision Under Time Pressure

The monolith would have been faster to write and enough to hit the time target. The five-stack split was about post-deployment operational quality: independent update cycles, isolated failure blast radius, and changing security policy without touching compute.

As a co-op on a four-month rotation, I had to be confident the system stayed maintainable after I left: a hard-to-maintain monolith or an over-engineered split would both cost whoever inherited it. The split was defensible because its boundaries map directly to the concerns that change independently.

Challenge 2: Cross-Stack Reference Naming at Multiple Environment Scales

The first implementation used simple export names like vpc-id. That broke the moment the team needed dev and staging in the same account: export names are region-scoped and globally unique, so two environments exporting vpc-id collide.

The fix was the {environment}-{stack-name}-{resource-type}-{qualifier} convention via parameter substitution: every export uses !Sub "${EnvironmentName}-network-vpc-id". That meant updating every Fn::ImportValue in every downstream stack: a mechanical refactor where any miss would silently fail at deploy time.

Challenge 3: Validation Errors That Actually Help

The first validation stage produced errors like "instance" does not match "^(10|172|192)\." : accurate, but useless unless you can read the regex, defeating the purpose for the engineers operating it.

The fix was a custom wrapper mapping each constraint to a human-readable message: "VPC CIDR block '192.168.0.0/16' is not in the 10.0.0.0/8 range required for internal lab environments. Use a 10.x.x.x/16 block." Modest authoring overhead, large drop in time-to-fix.

Challenge 4: Teardown Was Left Manual

One deliberate limitation: teardown isn't automated to the same standard as provisioning. Deleting stacks with data (RDS, populated S3 buckets) requires environment-specific retention decisions a pipeline shouldn't make unattended.

So teardown is a documented manual process: engineers trigger deletions in reverse dependency order. Automating it properly would need a separate pipeline with approval gates, backup confirmation, and per-environment retention policies, out of scope on a rotation already carrying two other deliverables. Better a clear manual process than a partial automation that fails silently on data-sensitive cases.

---

Outcome and What It Demonstrates

Provisioning dropped from four weeks to under thirty minutes. The pipeline runs end-to-end with no human intervention: validate the config, deploy all five stacks in dependency order, run health checks, and emit a structured summary of resource identifiers and endpoints. It provisioned environments for the SRE agent work and became the standard for new Ribbon labs.

• IaC architecture at real scale. Cross-stack dependencies, parameterized naming across environments, and coordinated multi-stack rollback are materially harder than a single template, and are the decisions that matter in production.
• Pre-flight validation as a first-class concern. Failures caught before AWS API calls are categorically cheaper than those caught mid-deployment.
• Idempotency by design. Explicit state detection (create / update / skip / recreate) makes the pipeline safe to re-run.
• Maintainability over cleverness. The multi-stack split, naming conventions, readable validation errors, and manual teardown all favor operational quality over build-time convenience: the automation had to work for engineers who didn't build it.

---

Tech Stack Summary