Kubernetes Cost Monitoring: Deploying OpenCost and Kubecost for Pod-Level Cost Allocation

by

Monitoring infrastructure expenses in Kubernetes requires translating node-level compute, memory, storage, and network costs down to granular logical entities: Namespaces, Deployments, StatefulSets, and individual Pods.

While OpenCost serves as the core, CNCF-hosted open-source engine calculating real-time resource allocation and cost metrics, Kubecost extends this engine with a rich UI, historical enterprise reporting, multi-cluster visibility, dynamic budget alerts, and automated rightsizing recommendations.

1. Architectural Overview: OpenCost Engine vs. Kubecost Ecosystem

To understand how cost allocation works under the hood, it helps to see how the underlying telemetry components interact within your cluster.

┌────────────────────────────────────────────────────────────────────────┐
│                        KUBERNETES CLUSTER                              │
│                                                                        │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐  │
│  │   cAdvisor /     │    │  Kube State      │    │ Cloud Provider   │  │
│  │   Metrics Server │    │  Metrics (KSM)   │    │ Billing APIs     │  │
│  └────────┬─────────┘    └────────┬─────────┘    └────────┬─────────┘  │
└───────────┼───────────────────────┼───────────────────────┼────────────┘
            │                       │                       │
            ▼                       ▼                       ▼
┌────────────────────────────────────────────────────────────────────────┐
│  OPENCOST METRICS ENGINE / EMITTER                                     │
│  • Calculates pod CPU/RAM/PV requests & utilization vs node cost       │
│  • Exposes Prometheus metric endpoints (/metrics)                      │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│  PROMETHEUS TIME-SERIES DATABASE                                       │
│  • Scrapes and stores cost allocation metrics                          │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│  KUBECOST ENTERPRISE ECOSYSTEM                                         │
│  • Front-end Dashboard & Cost Allocation UI                            │
│  • Savings Engine (Rightsizing & Idle Waste Detection)                 │
│  • Multi-cluster & Cloud Billing Reconciler                            │
└────────────────────────────────────────────────────────────────────────┘

2. Prerequisites & Architecture Planning

Before deploying cost monitoring tools, ensure your environment meets the following requirements:

  • Kubernetes Cluster: v1.22+ running on EKS, GKE, AKS, or bare-metal.
  • Helm: v3.8+ installed locally.
  • Prometheus: Existing Prometheus operator or capacity to deploy a bundled Prometheus instance (Kubecost includes a lightweight, pre-configured Prometheus setup by default).
  • Resource Ingestion Dependencies: kube-state-metrics (KSM) and cAdvisor running on nodes to collect container-level usage data.

3. Deployment Option A: Installing Standalone OpenCost via Helm

If you require an ultra-lightweight, purely open-source spec engine without complex frontend dependencies, deploy OpenCost directly.

Step 1: Add the OpenCost Helm Repository

Bash

helm repo add opencost https://opencost.github.io/opencost-helm/
helm repo update

Step 2: Configure Custom Cloud Pricing Engine

Create a configuration file named opencost-values.yaml. For cloud providers (AWS, Azure, GCP), OpenCost dynamically queries billing APIs or uses a pricing file fallback.

YAML

# opencost-values.yaml
opencost:
  exporter:
    extraEnv:
      # Enable cloud pricing reconciliation
      - name: CLOUD_PROVIDER
        value: "aws" # Options: aws, gcp, azure, csv
      - name: CUSTOM_PRICING_CONFIG_MAP
        value: "opencost-custom-pricing"
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 128Mi
  
  # Configure Prometheus scrape settings if using existing Prometheus
  prometheus:
    internal:
      enabled: true # Set to false if connecting to an external Prometheus instance

For custom or on-premise pricing (bare-metal), apply custom baseline pricing via a ConfigMap:

YAML

apiVersion: v1
kind: ConfigMap
metadata:
  name: opencost-custom-pricing
  namespace: opencost
data:
  default.json: |
    {
      "provider": "custom",
      "description": "On-Premises Bare Metal Baseline Pricing",
      "CPU": "0.03161",
      "RAM": "0.00423",
      "storage": "0.00004",
      "gpu": "0.95000"
    }

Step 3: Deploy OpenCost

Bash

kubectl create namespace opencost
helm install opencost opencost/opencost \
  --namespace opencost \
  -f opencost-values.yaml

4. Deployment Option B: Installing Full Kubecost Stack (Recommended)

Kubecost incorporates the OpenCost allocation engine alongside enterprise visualization, automated rightsizing recommendations, and cloud bill reconciliation.

1.Add Kubecost Helm Repository:Initialize repo and pull latest charts.

Execute the official Helm chart commands to register the repository:

Bash

helm repo add kubecost https://kubecost.github.io/cost-analyzer/
helm repo update

2.Configure Production Values (values.yaml):Tune storage, retention, and pricing reconciliation.

Create a production deployment configuration file kubecost-values.yaml. This setup enables persistent storage, configures retention policies, and sets up cloud pricing connectors.

YAML

# kubecost-values.yaml
global:
  prometheus:
    enabled: true
    fqdn: http://kubecost-prometheus-server.kubecost.svc.cluster.local

kubecostCostAnalyzer:
  ingress:
    enabled: true
    className: nginx
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    hosts:
      - host: kubecost.internal.yourdomain.com
        paths:
          - path: /
            type: ImplementationSpecific
  
  # Resource allocations for core cost-analyzer pod
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 2000m
      memory: 4Gi

  # Retain metric history
  persistence:
    enabled: true
    size: 32Gi
    storageClass: "gp3" # Adjust based on cloud provider (e.g., standard-rwd, managed-csi)

# Configure Prometheus retention settings
prometheus:
  server:
    retention: 30d
    persistentVolume:
      size: 64Gi
      storageClass: "gp3"

3.Deploy Kubecost Architecture:Execute Helm installation in targeted namespace.

Deploy the complete stack into a dedicated kubecost namespace:

Bash

kubectl create namespace kubecost
helm install kubecost kubecost/kubecost \
  --namespace kubecost \
  -f kubecost-values.yaml

4.Verify Service Deployment Status:Ensure pods and services are operational.

Check that all core services are running smoothly:

Bash

kubectl get pods -n kubecost

You should observe kubecost-cost-analyzer, kubecost-prometheus-server, and kube-state-metrics in a Running state.

5. Integrating Cloud Provider Billing Data

To account for actual cloud discounts, Enterprise Agreements (EA), AWS Savings Plans, and Spot Instance price fluctuations, integrate your cloud provider’s billing export APIs.

AWS Integration: CUR (Cost and Usage Report)

Create an IAM policy giving Kubecost permission to read your AWS CUR S3 bucket:

JSON

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "KubecostCURAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-kubecost-cur-bucket",
        "arn:aws:s3:::your-kubecost-cur-bucket/*"
      ]
    }
  ]
}

Provide the bucket configuration to Kubecost via a Kubernetes Secret:

YAML

apiVersion: v1
kind: Secret
metadata:
  name: cloud-cost-config
  namespace: kubecost
type: Opaque
stringData:
  aws-service-key.json: |
    {
      "bucket": "your-kubecost-cur-bucket",
      "region": "us-east-1",
      "serviceAccountHeader": "AWS",
      "projectARN": "arn:aws:iam::123456789012:role/KubecostCURRole"
    }

Update your kubecost-values.yaml to reference the secret:

YAML

kubecostProductConfigs:
  cloudIntegrationSecret: cloud-cost-config

6. Querying Metrics via Prometheus (PromQL)

Once deployed, OpenCost and Kubecost export real-time cost metrics to Prometheus. You can integrate these metrics directly into your Grafana dashboards or alerting systems using these baseline PromQL queries:

Pod Monthly Running Cost Rate

Calculates the real-time cost per hour of a specific pod multiplied by 730 hours (average hours in a month):

Code snippet

sum(node_cpu_hourly_cost * container_cpu_allocation) by (pod, namespace) * 730

Unallocated / Idle Capacity Cost by Node

Measures node capacity that is paid for but not requested by any running pod:

Code snippet

sum(kube_node_status_capacity{resource="cpu"}) by (node) 
- sum(kube_pod_container_resource_requests{resource="cpu"}) by (node)
* on(node) group_left node_cpu_hourly_cost

Namespace Cost Breakdown (CPU + RAM + Persistent Volumes)

Computes overall cost consumption grouped by Kubernetes Namespace over time:

Code snippet

sum(
  rate(container_cpu_allocation[1h]) * on(node) group_left node_cpu_hourly_cost +
  (rate(container_memory_allocation_bytes[1h]) / 1024 / 1024 / 1024) * on(node) group_left node_ram_hourly_cost
) by (namespace)

7. Actionable Cost Optimization: Pod Rightsizing

The primary operational benefit of deploying Kubecost is identifying over-provisioned workloads.

Accessing Local UI Rightsizing Dashboards

  1. Port-forward the Kubecost service locally:Bashkubectl port-forward --namespace kubecost service/kubecost-cost-analyzer 9090:9090
  2. Open http://localhost:9090 in your browser.
  3. Navigate to Savings $\rightarrow$ Request Right-Sizing Recommendations.
┌─────────────────────────────────────────────────────────────────────────────┐
│ WORKLOAD RIGHTSIZING RECOMMENDATION SUMMARY                                │
├───────────────────┬─────────────────────┬──────────────────┬────────────────┤
│ Workload / Pod    │ Current Requests    │ Recommended      │ Monthly Saving │
├───────────────────┼─────────────────────┼──────────────────┼────────────────┤
│ api-gateway       │ CPU: 2000m, RAM: 4G │ CPU: 350m, RAM: 1.2G │ ~$142.50 / mo │
│ payment-service   │ CPU: 1000m, RAM: 2G │ CPU: 150m, RAM: 512M │ ~$68.20 / mo   │
└───────────────────┴─────────────────────┴──────────────────┴────────────────┘

Applying Recommended Limits Programmatically

To resolve over-provisioning automatically, update your Deployment manifest specifications according to Kubecost’s target profile (aiming for standard 85th percentile utilization bounds):

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  namespace: production
spec:
  template:
    spec:
      containers:
        - name: api-gateway
          image: api-gateway:v2.4.1
          resources:
            requests:
              cpu: "350m"      # Downsized from 2000m based on 7-day P95 usage
              memory: "1200Mi"  # Downsized from 4Gi based on 7-day P95 usage
            limits:
              cpu: "1000m"
              memory: "2048Mi"

8. Verification and Continuous Governance Checkpoint

To ensure your cost-monitoring setup remains healthy long-term:

  1. Verify Metric Ingestion: Run kubectl logs -n kubecost deployment/kubecost-cost-analyzer and confirm there are no persistent database lock errors or API rate limit errors from your cloud provider.
  2. Setup Automated Alerts: Define Webhooks in the Kubecost UI to send daily budget notifications or alert Slack channels whenever a specific namespace experiences a spend spike greater than 20% over baseline.
  3. CI/CD Integration: Integrate OpenCost CLI tools (opencost CLI) into pull requests to check whether infrastructure changes modify workload resource requests prior to production merge.

Leave a Comment