Platform Engineering: The Evolution of DevOps and Why Every Company Needs an IDP

Platform Engineering hero — a developer navigating a modern internal platform dashboard

*Generated with Higgsfield GPT Image — 16:9*

Introduction

In 2012, a two-pizza team at Amazon published a manifesto that would reshape software delivery for a decade: "You build it, you run it." DevOps was born. Infrastructure-as-code, continuous integration, containerization, and Kubernetes democratized the ability to ship software fast. Any developer could deploy to production. Any team could own their pipeline.

The problem? Most of them didn't actually want to.

By 2024, the average software engineer at a mid-size tech company was spending nearly 40% of their time on infrastructure concerns — debugging Terraform state locks, wrangling Kubernetes YAML, chasing observability gaps, triaging Dependabot alerts, and navigating seven different internal tools just to ship a feature. DevOps gave developers power over their systems, but it also handed them an enormous cognitive tax that most were never hired to pay.

Platform Engineering is the answer. It asks a different question: instead of "who deploys software?", it asks "who owns the platform that developers use to deploy software?" The distinction sounds subtle, but it changes everything about how engineering organizations scale.

The CNCF Platforms Working Group formalized this shift in 2023 with a white paper defining platform engineering as the discipline of designing and building toolchains and workflows that enable self-service capabilities for software engineering organizations. Gartner went further, predicting that 80% of large software engineering organizations will have dedicated platform teams by 2026. That prediction is well on track.

This post is your complete guide to understanding platform engineering — what it is, why it emerged, what an Internal Developer Platform (IDP) contains, and how to start building one whether you have a team of five or five hundred.

What DevOps Got Wrong

DevOps did not fail. It succeeded enormously at its original goal: breaking down the wall between developers and operations, enabling continuous delivery, and moving organizations away from monthly release trains to daily or hourly deployments. DORA metrics improved industry-wide. Software quality went up. Release cycles shortened.

But success created a new problem. "You build it, you run it" was designed for teams with deep operational knowledge. As it spread across the industry, it was applied to every engineering team regardless of context, interest, or skill. The result was a phenomenon researchers now call DevOps tax — the hidden cost in developer time, energy, and cognitive load imposed by infrastructure ownership at scale.

Consider what a developer at a modern cloud-native company is expected to know and operate on a typical day:

CI/CD pipelines — GitHub Actions, Jenkins, CircleCI, or Tekton. Debugging flaky tests, managing secrets, caching builds.
Kubernetes — Writing Deployments, Services, Ingresses, HorizontalPodAutoscalers, resource limits, PodDisruptionBudgets.
Infrastructure as code — Terraform or Pulumi for VPCs, RDS instances, S3 buckets, IAM roles.
Observability — Setting up Prometheus metrics, Datadog dashboards, defining SLOs, creating PagerDuty escalation policies.
Security scanning — Trivy for container images, Snyk for dependencies, Semgrep for SAST.
Cost management — Understanding cloud billing, right-sizing instances, avoiding reserved instance waste.
Service mesh — Istio or Linkerd routing, mTLS, traffic splitting for canary releases.

Studies from DORA (2023 State of DevOps Report) found that developers at organizations without platform teams spend on average 1.5 days per week on infrastructure-related tasks. That is 30% of total working time — time that does not produce user-facing features.

The developer experience survey from Humanitec (2024) found that 83% of developers feel overwhelmed by the cognitive complexity of their tooling stack. More damning: 41% said they had made infrastructure decisions they later regretted because they simply did not know enough about the tradeoffs.

The DevOps dream of autonomous, self-sufficient teams ran into the reality of finite human attention. Not every developer wants to become a platform expert. And forcing them to become one is expensive.

pie title Developer Time Distribution (Without Platform Team)
    "Feature Development" : 42
    "Infrastructure & CI/CD" : 28
    "Debugging & Incidents" : 18
    "Meetings & Coordination" : 12

What Platform Engineering Is

Platform Engineering takes the operational complexity that DevOps distributed across all teams and re-centralizes it — but with a crucial difference from the old operations model. The platform team is not a gatekeeper. It is a product team, and developers are its customers.

The output of a platform team is an Internal Developer Platform (IDP): a curated set of self-service capabilities, workflows, and tools that developers can use without becoming infrastructure experts. The key word is self-service. Developers do not file tickets. They do not wait for approvals. They click a button (or run a CLI command) and get what they need.

Spotify popularized this model with Backstage, their internal developer portal, which they open-sourced in 2020. When Backstage launched internally at Spotify, it unified a fragmented tooling landscape into a single catalog. Developers could discover services, understand ownership, spin up new microservices from templates, and access documentation — all from one place. The result: onboarding time for new engineers dropped dramatically, and developer satisfaction scores climbed.

The platform engineering philosophy rests on three core ideas:

1. Paved roads, not guardrails. A golden path is the blessed way to do something — the approach that the platform team has tested, secured, optimized, and documented. Developers are not forced to use it, but it is so much easier than the alternative that most choose to. Think of it like a highway versus a dirt road: you can take the dirt road, but why would you?

2. The platform as a product. Platform teams use product management techniques: user research, roadmaps, feedback loops, versioning, deprecation policies. They measure adoption and developer satisfaction. They run office hours. They treat their internal users with the same respect a SaaS company treats its customers.

3. Developer self-service at every layer. From spinning up a new service to requesting a database, from provisioning a staging environment to rotating secrets — the IDP enables developers to do these things independently, without waiting for another team.

The ecosystem has matured rapidly. Tools like Backstage (open source, CNCF project), Port (API-first commercial platform), Humanitec (platform orchestrator), and Cortex (catalog and scorecard) have made it possible to build a production-quality IDP without writing everything from scratch.

Core IDP Components

An Internal Developer Platform is not a single tool — it is a system of integrated capabilities that spans the entire software development lifecycle. Understanding what goes into an IDP helps teams prioritize what to build first and what to buy.

*Generated with Higgsfield GPT Image — 16:9*

Service Catalog

The foundation of any IDP is a service catalog — a searchable registry of every service, library, API, data pipeline, and infrastructure component in the organization. The catalog answers questions like: Who owns this service? What does it do? What are its dependencies? What is its current health? Where is its documentation?

Without a catalog, developers waste time rediscovering existing solutions, reinventing the wheel, and reaching out to the wrong team when something breaks. With one, they can find and reuse internal components in minutes.

Here is an example catalog-info.yaml for Backstage, the most widely used catalog format:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-service
  description: "Handles payment processing via Stripe and internal billing APIs"
  annotations:
    github.com/project-slug: "acmecorp/payments-service"
    pagerduty.com/service-id: "P12345"
    datadoghq.com/service-name: "payments-service"
  tags:
    - payments
    - critical
    - backend
spec:
  type: service
  lifecycle: production
  owner: group:payments-team
  system: billing
  dependsOn:
    - component:user-service
    - component:fraud-detection-service
  providesApis:
    - payments-api

Self-Service Templates (Scaffolding)

Templates allow developers to spin up new services from a curated starting point that already includes security defaults, CI pipelines, observability hooks, and README documentation. Instead of copying and modifying an existing service (with all its baggage), developers run one command and get a production-ready skeleton.

CI/CD Abstraction

Rather than each team reinventing their pipeline, the platform team provides reusable CI/CD templates that teams opt into. GitHub Actions reusable workflows, shared Jenkins libraries, or Tekton task bundles — the implementation varies, but the goal is the same: secure, tested, and consistent pipelines without every team needing to become a CI expert.

Environment Provisioning

Developers need environments: development, staging, production, and often ephemeral preview environments for feature branches. IDPs enable on-demand environment provisioning — a developer can request a staging environment via a UI or CLI, and the IDP automatically provisions the Kubernetes namespace, network policies, secrets, and monitoring.

Observability Integration

Logging, metrics, tracing, and alerting should be available automatically, not configured manually per service. When a service is created from an IDP template, it gets a Datadog dashboard, a Prometheus scrape config, and a PagerDuty escalation policy by default.

Secrets Management

Vault, AWS Secrets Manager, or GCP Secret Manager access should be self-service. Developers should be able to request and rotate secrets through the IDP without involving a security team for routine operations.

Cost Visibility

Cloud cost attribution at the team or service level enables financial accountability. Developers should see what their services cost to run, identify anomalies, and make informed tradeoff decisions.

graph TB
    subgraph IDP["Internal Developer Platform"]
        SC[Service Catalog]
        ST[Self-Service Templates]
        CI[CI/CD Abstraction]
        EP[Environment Provisioning]
        OB[Observability]
        SM[Secrets Management]
        CV[Cost Visibility]
    end

    DEV[Developer] -->|Browse & discover| SC
    DEV -->|Scaffold new service| ST
    ST -->|Triggers| CI
    CI -->|Deploys to| EP
    EP -->|Wired to| OB
    EP -->|Accesses| SM
    OB -->|Reports to| CV

    SC -.->|Linked metadata| OB
    SC -.->|Ownership| SM

    style IDP fill:#f0f4ff,stroke:#4a6cf7,stroke-width:2px
    style DEV fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px

Building vs Buying

One of the first decisions platform teams face is whether to build their IDP in-house, use open-source tools, or buy a commercial solution. There is no universally correct answer — the right choice depends on team size, budget, engineering maturity, and how much customization you need.

Open Source: Backstage

Backstage is the most widely deployed IDP framework. Created by Spotify and donated to the CNCF in 2020, it has become the de facto industry standard. Over 2,700 companies use it in production, including Netflix, American Airlines, LinkedIn, Airbnb, and Zalando.

Pros: Completely free, infinitely extensible via plugins (600+ available), large community, strong ecosystem, no vendor lock-in.

Cons: Requires a React engineering team to maintain, significant initial setup investment (typically 2-4 weeks for a basic deployment, months for full customization), plugin quality varies widely, hosting burden falls on your team.

Commercial: Port

Port is the fastest-growing commercial IDP, built on an API-first catalog model. Developers define blueprints (data models for resources), connect integrations (GitHub, AWS, Kubernetes, PagerDuty), and build self-service actions — all without writing code.

Pros: Days to initial value (not months), no hosting burden, strong scorecard and compliance features, beautiful UI.

Cons: Ongoing subscription cost (pricing based on users), less customizable than Backstage, some vendor dependency.

Commercial: Humanitec

Humanitec takes a different approach with its Platform Orchestrator model — focusing less on the developer portal UI and more on the API layer that translates developer intent into infrastructure configurations. It integrates with any CI system and manages dynamic environment configuration.

Commercial: Cortex

Cortex focuses on the service catalog and engineering excellence scorecards. It is particularly strong for organizations that want to track service maturity, enforce standards, and gamify quality improvements.

flowchart TD
    A{What is your priority?} --> B{Budget available?}
    B -->|No budget| C{Engineering team size?}
    B -->|Budget available| D{Need quick time-to-value?}

    C -->|Small < 5 eng| E[Start with lightweight tooling\nGitHub + Actions templates]
    C -->|Medium 5-20 eng| F[Backstage self-hosted\nOpen source]
    C -->|Large > 20 eng| G[Backstage with platform team\nFull investment]

    D -->|Yes, within weeks| H[Port or Cortex\nCommercial SaaS]
    D -->|No, can invest months| I{Need full customization?}

    I -->|Yes| F
    I -->|No - orchestration focus| J[Humanitec\nPlatform Orchestrator]

    style E fill:#fff3cd
    style F fill:#d4edda
    style G fill:#d4edda
    style H fill:#cce5ff
    style J fill:#cce5ff

The Golden Path Pattern

The golden path is the most powerful pattern in platform engineering. It is the combination of tools, templates, and processes that the platform team has curated and recommended as the standard way to build and deploy services. When a developer starts a new service using the golden path, they get everything they need without having to make infrastructure decisions.

A mature golden path for a new microservice typically provisions:

1. GitHub repository — from a template with standard .github/workflows, Dockerfile, README.md, linting config, and branch protection rules

2. CI/CD pipeline — pre-configured GitHub Actions workflows for build, test, security scan, and deploy

3. Kubernetes namespace — with network policies, resource quotas, and service accounts

4. Service mesh entry — Istio virtual service and destination rule

5. Observability — Datadog APM auto-instrumentation, pre-built dashboard, SLO definition

6. PagerDuty service — escalation policy linked to the owning team

7. Catalog entry — catalog-info.yaml registered in Backstage or equivalent

The beauty of the golden path is that it encodes best practices invisibly. Security defaults, cost controls, and observability are not checklist items — they are automatic outcomes of using the standard tooling.

Here is a simplified example of what a golden path scaffolder template produces when invoked via Backstage:

# scaffolder-template.yaml (Backstage Template)
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: new-microservice
  title: New Microservice
  description: Creates a production-ready microservice with CI/CD, observability, and catalog registration
spec:
  owner: platform-team
  type: service
  parameters:
    - title: Service Details
      properties:
        name:
          title: Service Name
          type: string
          description: "Lowercase, hyphenated (e.g. 'payments-service')"
        owner:
          title: Owner Team
          type: string
          ui:field: OwnerPicker
        language:
          title: Language
          type: string
          enum: [go, python, typescript, java]
  steps:
    - id: create-repo
      name: Create GitHub Repository
      action: publish:github
      input:
        repoUrl: "github.com?repo=${{ parameters.name }}&owner=acmecorp"
        defaultBranch: main
        gitAuthorName: platform-bot
        sourcePath: templates/${{ parameters.language }}/service

    - id: register-catalog
      name: Register in Service Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['create-repo'].output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

    - id: create-namespace
      name: Provision Kubernetes Namespace
      action: kubernetes:create-namespace
      input:
        name: ${{ parameters.name }}
        labels:
          team: ${{ parameters.owner }}
          service: ${{ parameters.name }}

Golden path comparison — before and after platform engineering adoption

*Generated with Higgsfield GPT Image — 16:9*

sequenceDiagram
    participant Dev as Developer
    participant IDP as IDP / Backstage
    participant GH as GitHub
    participant K8s as Kubernetes
    participant DD as Datadog
    participant PD as PagerDuty

    Dev->>IDP: "Create new service: payment-processor"
    IDP->>GH: Fork template repo → payment-processor
    GH-->>IDP: Repo created ✓
    IDP->>GH: Configure branch protection + Actions workflows
    IDP->>K8s: kubectl create namespace payment-processor
    K8s-->>IDP: Namespace + RBAC created ✓
    IDP->>DD: Register APM service + create dashboard
    DD-->>IDP: Dashboard URL ✓
    IDP->>PD: Create service + link to payments-team escalation
    PD-->>IDP: PagerDuty service ID ✓
    IDP->>GH: Commit catalog-info.yaml to repo
    IDP-->>Dev: ✅ Service ready — links to repo, dashboard, runbook
    Note over Dev,PD: Total time: ~90 seconds vs. ~2 days manual

Measuring Platform Success

A platform team that cannot demonstrate its value will not survive the next budget cycle. Fortunately, the metrics for platform success are concrete and well-established.

DORA Metrics are the gold standard for measuring software delivery performance:

|--------|----------------|----------------|------------------|---------------|

| Change Failure Rate | 0–5% | 5–10% | 10–15% | 15–30% |

Platform teams track developer satisfaction separately via quarterly developer experience surveys. Key questions: How much time did you spend on infrastructure this week? How easy was it to deploy? How confident are you in your observability setup? Tracking these scores over time reveals whether the platform is actually reducing cognitive load.

Platform teams fit the enabling team pattern from Team Topologies — their job is to make stream-aligned teams (feature teams) more effective, not to be in the critical path of delivery. This distinction matters: if developers are waiting on the platform team, the platform has failed its own design goals.

Getting Started Without a Platform Team

Platform engineering can sound expensive and out of reach for smaller organizations. It does not have to be. The principles scale down as well as up.

Start with one small, high-value improvement. Do not try to build a full IDP from day one. Choose one of these entry points:

Option 1: Standardize CI templates. Create a GitHub Actions reusable workflow that handles build, test, security scan, and Docker build consistently. Make it a 5-line uses: reference in every service's workflow. This alone saves hours per developer per week and eliminates the "each repo has a different CI setup" problem.

Option 2: Create one golden path. Build a cookiecutter or Yeoman template (or a simple shell script) that scaffolds a new service with your standard directory structure, Dockerfile, CI workflow, and README. Socialize it with the team. Measure adoption.

Option 3: Start a service catalog. Even a simple GitHub repository with a services.yaml file listing every service, its owner, and its links is infinitely better than no catalog. If your team is ready for a real tool, deploy Backstage via the @backstage/create-app CLI in an afternoon — the basic deployment takes a few hours.

The worst thing you can do is wait until you have a dedicated platform team before starting. One engineer spending 20% of their time on platform concerns can deliver enormous value. The goal is not perfection — it is reducing friction, one paved road at a time.

Conclusion

Platform Engineering is not a replacement for DevOps — it is its maturation. DevOps gave developers the power to deploy. Platform Engineering gives them a platform that makes that power accessible without requiring every developer to become a site reliability engineer.

The Internal Developer Platform is the infrastructure upon which modern software engineering organizations run. It is the difference between developers spending their days fighting YAML and developers shipping features that matter to users. When done well, an IDP is nearly invisible — developers do not talk about the platform because it just works.

The CNCF's Platforms Working Group, Gartner's predictions, and the rapid growth of tools like Backstage, Port, and Humanitec all point to the same conclusion: platform engineering is no longer optional for organizations at scale. The question is not whether you will need an IDP, but how you will build one.

Start small. Pick one problem. Make one thing easier for developers today. That is how every great platform begins.

*Next in this series: [Internal Developer Platforms in 2026: Build vs Buy, and the Tools That Win](/blog/060-internal-developer-platforms)*

Enjoyed this post? Follow AmtocSoft for AI tutorials from beginner to professional.

☕ Buy Me a Coffee | 🔔 YouTube | 💼 LinkedIn | 🐦 X/Twitter

Search This Blog

AmtocSoft Tech Insights