GitOps and IaC at Scale – ArgoCD and Open Tofu – Part 3 – Hardening and Manage users

#devops #aws #terraform #gitops

Level 400

Earlier blogs covered scalable delivery, architecture patterns, and code samples. This section provides additional production-ready examples and best practices using the Well Architected Framework and GitOps with ArgoCD.

Architecture Overview

Now, suppose that you must apply some custom addons for the environment, also enable on-boarding application developer’s teams and platform teams to manage their projects and applications with the least manual effort, integration with AWS ecosystem like IAM identity Center and apply best practices for ArgoCD setup. For accomplishing this the following image depicts the high-level reference architecture with the main services:

Here the new resources are ALB as an ingress gateway for Argo server, Route 53 to enable a Hosted Zone for public Domain, AWS Certificate management for SSL trough the ALB, and IAM Identity Center for SSO capabilities.

For production environments there could be a private hosted Zone and use PKI architecture internal to enable SSL access.

For other hand, *how can you be modeling the groups and access to the Argo projects and applications? *
There is not correct answer and depending on your operational model (Conway’s Law) for example, the following image depicts a simple group for each team around a workload.

Assigning permissions should follow the principle of the least privilege, separating application and project actions. For example, the application engineering team can run all actions on Applications but only watch and update the platform and addon applications.

Hands On

It’s time to create code.

First, we must modify the hub blueprint to enable argo cd best practices
❌ Don’t use Default admin for management- Disable it.
❌ Don’t use the default project
❌ Don’t use local users just if it is completely necessary for CLI or external automatons.
✅ Create Roles and assign to groups instead of individual assignments.

Now, according to Argo cd documentation the SSO is configured with IAM Identity Center.

A working Single Sign-On configuration using Identity Center (AWS SSO) has been achieved using SAML (with Dex)

To manage IAM Identity Center in AWS, applications must be created because the CreateApplicationAPI only supports custom OAuth 2.0 apps. Third-party SAML or OAuth 2.0 apps must be set up via their respective services or the AWS console. Application creation cannot be automated with Terraform, but user assignments can still be handled through IaC.

Initially, a custom Terraform module was developed to establish the ArgoCD hardening baseline. This module leverages the gitops bridge module as its foundation and incorporates SSO configuration, along with group and default role setup and baseline specifications.

The module name is terraform-helm-hardening-gitops-bridge

You can find the source code module here 👉

thothforge / terraform-helm-hardening-gitops-bridge

GitOps bridge extended module with hardennig practices and examples

Terraform Hardening GitOps Bridge Module

A comprehensive Terraform module that provides a hardened GitOps bridge for Amazon EKS clusters, implementing security best practices and enterprise-grade configurations for GitOps workflows using ArgoCD.

📖 Table of Contents

Features

Prerequisites

Architecture

Quick Start

Examples

Configuration Options

Security Best Practices

Troubleshooting

Deployment Patterns

Monitoring and Observability

Performance Optimization

Advanced Security Configuration

DNS and Ingress Configuration

Customization Options

Cost Optimization

Testing and Validation

Debugging and Diagnostics

FAQ

Migration Guide

Best Practices

Operational Procedures

Advanced Troubleshooting

Additional Resources

Versioning and Changelog

Contributing

License

Support

🚀 Features

Core Capabilities

GitOps Integration: Seamless integration with ArgoCD for GitOps workflows

Security Hardening: Enterprise-grade security configurations and best practices

Multi-Repository Support: Support for addons, platform, and workloads repositories

Flexible Deployment: Single cluster or hub-spoke architecture support

Comprehensive Addons: Pre-configured essential Kubernetes addons

Security Features

RBAC Integration: Role-based access control with customizable permissions

SSO Support…

View on GitHub

or 👉

registry.terraform.io

The main files are: bootstrap/argocd-values.yaml where the default values are created and parser some parameters from terraform parameters like default project name, dns or argo host, kind of ingress, tags, subnets and more.

# bootstrap/argocd-values.yaml
global:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "topology.kubernetes.io/hostname"
      whenUnsatisfiable: "ScheduleAnyway"
    # Default logging options used by all components
  logging:
    # -- Set the global logging format. Either: `text` or `json`
    format: text
    # -- Set the global logging level. One of: `debug`, `info`, `warn` or `error`
    level: debug



configs:
  params:
    server.insecure: true
  # SSO configuration with Entra ID
  cm:

    url: https://${argo_host}
    dex.config: |
      logger:
        level: debug
        format: json
      connectors:
      - type: saml
        id: aws
        name: "AWS IAM Identity Center"
        config:
          # You need value of Identity Center APP SAML (IAM Identity Center sign-in URL)
          ssoURL: ${sso_assertion_url} #https://portal.sso.yourregion.amazonaws.com/saml/assertion/id
          # You need `caData` _OR_ `ca`, but not both.
          #<CA cert (IAM Identity Center Certificate of Identity Center APP SAML) passed through base64 encoding>
          caData: ${ca_data_iam_app} 
          # Path to mount the secret to the dex container
          entityIssuer: https://${argo_host}/api/dex/callback
          redirectURI: https://${argo_host}/api/dex/callback
          usernameAttr: email
          emailAttr: email
          groupsAttr: groups

  rbac:
    policy.csv: |
      p, role:platform-team, applications, *, */*, allow
      p, role:platform-team, projects, *, */*, allow
      p, role:platform-team, clusters, *, *, allow
      p, role:platform-team, repositories, *, *, allow
      p, role:platform-team, certificates, *, *, allow

      p, role:platform-team, logs, get, */*, allow
      p, role:platform-team, exec, create, */*, allow
      p, role:platform-team, applicationsets, *, */*, allow
      p, role:platform-team, accounts, get, */*, allow
      p, role:platform-team, sessions, create, */*, allow
      p, role:platform-team, sessions, delete, */*, allow
      p, role:platform-team, projects, get, *, allow
      p, role:platform-team, projects, list, *, allow
      p, role:platform-team, projects, update, *, allow
      p, role:platform-team, projects, create, *, allow

      p, role:platform-team, actions, *, */*, allow
      g, ${admin_idp_group_id}, role:platform-team

    policy.default: role:deny
    policy.matchMode: glob
    scopes: '[groups, email]'
server:
  autoscaling:
    minReplicas: 2
    maxReplicas: 20
    targetCPUUtilizationPercentage: 80
    targetMemoryUtilizationPercentage: 80
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
  service:
    type:  "NodePort"
  ingress:
    enabled: "${enable_argo_ingress}"
    hostname: "${argo_host}"
    controller: "aws"
    ingressClassName: "alb"
    tls: true
    aws:
      backendProtocolVersion: GRPC
      serviceType: "NodePort" #"ClusterIP"
    annotations:
      alb.ingress.kubernetes.io/group.name: argocd
      alb.ingress.kubernetes.io/backend-protocol: HTTP
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: "${aws_load_balancer_type}"
      alb.ingress.kubernetes.io/security-groups: "${argo_ingress_sg}"
      alb.ingress.kubernetes.io/certificate-arn: "${acm_certificate_arn}"
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/healthcheck-path: /healthz
      alb.ingress.kubernetes.io/subnets: "${ingress_subnets}"
      alb.ingress.kubernetes.io/tags: "${required_tags}"
    path: /

    # -- Ingress path type. One of `Exact`, `Prefix` or `ImplementationSpecific`
    pathType: Prefix
    #extraRules:
    #- http:
    #    paths:
    #    - path: /argocd
    #      pathType: Prefix
    #      backend:
    #        service:
    #          name: 'argo-cd-argocd-server-grpc'
    #          port:
    #            name: 'https'
  ingressGrpc:
    # -- Enable an ingress resource for the Argo CD server for dedicated [gRPC-ingress]
    enabled: false #"${enable_argo_ingress}"
    hostname: "${argo_host}"
    ingressClassName: "alb"
    annotations:
      alb.ingress.kubernetes.io/group.name: argocd
      alb.ingress.kubernetes.io/backend-protocol: HTTPS
      alb.ingress.kubernetes.io/backend-protocol-version: GRPC
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/target-type: "ip"
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/healthcheck-path: /healthz
      alb.ingress.kubernetes.io/success-codes: "0-99"
      alb.ingress.kubernetes.io/security-groups: "${argo_ingress_sg}"
      alb.ingress.kubernetes.io/certificate-arn: "${acm_certificate_arn}"
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
      alb.ingress.kubernetes.io/subnets: "${ingress_subnets}"
      alb.ingress.kubernetes.io/tags: "${required_tags}"
controller:
  replicas: 3
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
  metrics:
    enable: true
  priorityClassName: "system-node-critical"
  podAnnotations:
    prometheus.io/scrape: true
    prometheus.io/port: 8082
    prometheus.io/path: "/metrics"
  service:
    annotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8082
      prometheus.io/path: "/metrics"


repoServer:
  replicas: 2
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
    metrics:
      enable: true
    priorityClassName: "system-node-critical"
    podAnnotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8084
      prometheus.io/path: "/metrics"
    service:
      annotations:
        prometheus.io/scrape: true
        prometheus.io/port: 8084
        prometheus.io/path: "/metrics"


notificationsController:
  enable: true

redis-ha:
  enabled: true
  waitForVotes: false
  quorum: 2

applicationSet:
  metrics:
    enable: true
  replicas: 2
  podAnnotations:
    prometheus.io/scrape: true
    prometheus.io/port: 8085
    prometheus.io/path: "/metrics"
  service:
    annotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8085
      prometheus.io/path: "/metrics"

For other hand, the argo project default template is: bootstrap/default_argproj.yaml

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: ${default_argoproj_name}
  namespace: argocd
  labels:
    argocd.argoproj.io/instance: argoprojects-platform-team
spec:
  description: Team for TI to manage Argo projects and applications
  sourceRepos: ${jsonencode(repositories)}
  destinations:
    - namespace: '*'
      server: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'

You can modify according to the baseline’s requirements; this is the platform project like default.

The platform addons and platform appsets just receive the parameter for default project in argoCD but keep as the beginning of this series.

Other kinds of resources for the module are security groups and default secrets, if the SSO is not enabled for platform, for admin, developers and view only users.

For other hand, the module can create a certificate and CNAME in a public hosted zone. Is usual that these resources are managed from other team, but in this setup in the same account exists a delegate hosted zone for internal use. This allows use external DNS addon to enable easy integration and register additional services in your control plane.

Sensitive data like ca data and group id could be loaded from parameter store provided for security team. Or save in secrets manager to mount the secret onto the dex container in the argocd-dex-server Deployment.

The module can be utilized with Terragrunt as follows:

#eks_control_plane-terragrunt.hcl
include "root" {
  path   = find_in_parent_folders("root.hcl")
  expose = true
}

#include "kubectl_provider" {
#  path = find_in_parent_folders("/common/additional_providers/provider_kubectl.hcl")
#}

include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_helm.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name                       = "dummy-cluster-name"
    cluster_endpoint                   = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version                    = "1.31"
    cluster_platform_version           = "1.31"
    oidc_provider_arn                  = "dummy_arn"
    node_security_group_id = "sg-xasr18923d"
    cluster_primary_security_group_id = "sg-xasr18923d"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
dependency "eks_role" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/iam/eks_role"
  mock_outputs = {
    iam_role_arn = "arn::..."
  }
  mock_outputs_merge_strategy_with_state = "shallow"

}
dependency "vpc" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/network/vpc"
  mock_outputs = {
    vpc_id = "vpc-04e3e1e302f8c8f06"
    public_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
    ]
    private_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
      "subnet-0d5061f70b69eda15",
    ]
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}


locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment = "control-plane"
      default_project = include.root.locals.environment.locals.default_project
      enable_argo_ingress = true
      sso_assertion_url = "https://portal.sso.<SSO_REGION>.amazonaws.com/saml/assertion/<id>"
      ca_data= "BASE 64 CA Data"
      argo_host_dns = {
        domain_name            = "argocd.devsecops.labvel.io"
        zone_id                = "Z101221820UZWIPQ27722"
        aws_load_balancer_type = "internet-facing"
        validation = "public"
      }
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        include.root.locals.environment.locals.addons_repos_properties
      )


        tags = {
          Environment = "control-plane"
          Layer       = "Containers"
        }
      }

    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace        = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source ="tfr:///thothforge/hardening-gitops-bridge/helm?version=1.0.1"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  node_security_group = dependency.eks.outputs.cluster_primary_security_group_id
  vpc_id         = dependency.vpc.outputs.vpc_id
  private_subnet_ids = dependency.vpc.outputs.private_subnets
  public_subnet_ids = dependency.vpc.outputs.public_subnets

  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data
  enable_argo_ingress = local.workspace["enable_argo_ingress"]
  argo_host_dns = local.workspace["argo_host_dns"]
  #external_dns_domain_filters = [local.workspace["argo_host_dns"]]
  sso_assertion_url = local.workspace["sso_assertion_url"]
  ca_data= local.workspace["ca_data"]

  admin_idp_group_id ="318xx789-x071-70f5-63x6-xx21233xxx33"

  tags = local.workspace["tags"]

}