Application Modernization on EKS: Moving a Database Out of Kubernetes

In Phase 1, I built the AWS foundation with ten Terraform modules. In Phase 2, the platform layer went on top: ArgoCD, Karpenter, cert-manager, ExternalDNS, External Secrets, Velero, all managed through an App-of-Apps. Phase 3 is where the actual application finally lands: the Instana Robot Shop, deployed to the cluster, with its MySQL databases pulled out of the cluster and onto managed RDS.

The headline is "deploy the app." The real work was everything that makes a stateful microservices app run properly on managed infrastructure: mapping which service talks to which datastore, moving two databases to RDS without leaving a single password in a manifest, bootstrapping schemas that a Docker image used to create for free, and wiring it all through External Secrets so the credentials only ever exist at runtime.

By the end, shop.devopsportfolio.com serves the storefront over HTTPS, ten of twelve services are healthy, and the two that aren't are down for a reason I'll get to: a reason that's actually a feature, not a bug.

Here's what got built.

Read the code first, not the README

The plan I'd written weeks earlier said: ratings uses MySQL; migrate it to RDS; everything else stays in-cluster. One database, one service, done.

I reviewed the service source instead of relying on my plan and found five issues. This is the single most valuable habit in the whole phase, so I'm putting it first.

• Shipping also uses MySQL. The plan had shipping down as a RabbitMQ-only service. Its JpaConfig.java builds a JDBC URL against a cities database. So two services move to RDS, not one, and cities is where essentially all the seed data lives (~948,000 rows).

• Ratings reads a full DSN, not a hostname. The plan assumed MYSQL_HOST. The code reads PDO_URL, a complete PDO connection string: mysql:host=...;port=3306;dbname=ratings;charset=utf8mb4. Setting MYSQL_HOST on that pod does nothing; it silently keeps resolving the default MySQL hostname and CrashLoops on a DNS failure that looks unrelated to configuration.

• Shipping's user and database don't match. The MySQL user is shipping; the database it owns is cities. That asymmetry comes straight from the source, and it's exactly the kind of thing that reads like a typo six months later.

• User depends on Redis too. The plan mapped user to MongoDB only. It opens a Redis client at startup as well, which matters later because it widens the blast radius of "delete the redis pod" in the chaos phase.

• Ratings listens on port 80, not 8080. It's a PHP/Apache image. Its health path is /_health, not /health. Service, readiness probe, and the nginx upstream all need that exception.

None of these are in the README. All five would have cost debugging time. The lesson that carried the whole phase: your project plan is a hypothesis; the source code is the truth.

The data tier: one managed service, two schemas

The decision matrix after the source read:

Datastore	Decision	Why
MySQL	RDS MySQL 8.0	Backups, patching, encryption, point-in-time recovery. This is the layer the disaster-recovery and secret-rotation stories are built on.
MongoDB	stays in-cluster	DocumentDB starts around $200/mo for the smallest instance. Not justifiable for a portfolio demo.
Redis	stays in-cluster	Cart state is disposable cache. Losing it is a cache miss, not data loss. ElastiCache isn't worth it.
RabbitMQ	stays in-cluster	Amazon MQ has no free tier, and the broker isn't on the critical path for the demo journey.

One managed data service, already provisioned in Phase 1, now carrying two databases: ratings and cities. Everything else stays in the cluster as a deliberate, documented cost trade-off.

Secrets: the master credential never touches the app namespace

Phase 1 left me a single robot-shop/rds-credentials secret, which is the RDS master. But ratings and cities are two schemas that want two separate least-privilege app users. So I created two more secrets, robot-shop/ratings-db and robot-shop/shipping-db, each with its own generated password, and left the master alone.

The important boundary is that nothing in the robot-shop application namespace can read the master credential. The two app ExternalSecrets sync into robot-shop. The master syncs into a separate robot-shop-bootstrap namespace, and the only thing that runs there is the schema bootstrap job, running exclusively during a pre-sync hook. No long-lived workload ever shares a namespace with the master password.

I want to be honest about the limit of this, because it's the kind of thing a reviewer should catch. External Secrets Operator's controller holds one IAM identity scoped to robot-shop/*, so the controller itself can technically read the master on behalf of any namespace. What's actually enforced is that no manifest in the app namespace asks it to, plus namespace and RBAC separation. A hard boundary needs a second ESO identity and a namespaced SecretStore; that's a documented Phase 10 hardening item, not something I'm pretending is already done.

One more decision worth flagging: I assemble the ratings DSN in the ExternalSecret's template, not in Secrets Manager. Secrets Manager holds atomic fields: host, port, database, user, password. The PDO_URL connection string gets built in the manifest. Connection string format is an application concern, so it lives in the GitOps repo. If the charset changes, that's a one-line manifest edit, not a terraform apply.

The gotcha that cost me an afternoon: a credential that lied about being wrong

This is the one that ate the most time, and the fix is a genuinely better pattern, so it earns its own section.

Phase 1 stored the RDS master password by generating it with random_password and writing it to Secrets Manager, while the instance took the same value through the write-only password_wo argument. Two resources, one value, kept in sync by hand.

Except password_wo only pushes to the instance when you bump password_wo_version. That version was pinned to 1. So when random_password regenerated for its own reasons, the secret got the new value and the instance kept the old one. Secret and instance drifted, and the bootstrap job authenticated with a password RDS no longer accepted.

The error was Access denied for user 'admin'. Straightforward enough, except while chasing it, the EBS side threw something far more misleading, which I'll cover below. The RDS fix was to stop hand-syncing two resources entirely:

manage_master_user_password = true

With this, RDS generates and owns the master password itself, writes it to an AWS-managed secret at instance-creation time, and the instance and secret are set by a single atomic operation. They cannot drift, because there's no second resource to fall out of step. Terraform reads that managed secret and mirrors it into the stable robot-shop/rds-credentials name that ESO already consumes, so nothing in the GitOps repo changed.

One catch worth knowing: master_user_secret_kms_key_id is immutable once the instance exists. If RDS already enabled managed passwords with the default aws/secretsmanager key, trying to point it at a CMK on a later apply fails with You can't change an existing master user secret KMS key. Since the mirror flow means ESO never reads the managed secret directly, the key on it doesn't matter, so I dropped the line and moved on.

That screenshot is the whole secrets pipeline proven end to end: the ratings MySQL user authenticating with the password that came out of the ratings-db Kubernetes secret, which came from AWS Secrets Manager, which matches what the app pod will present. One value, one source, verified at the database.

The schema bootstrap: replacing what a Docker image did for free

The in-cluster MySQL image used to do three invisible things: create the cities database, create the shipping user, and run the seed SQL. RDS has no entrypoint. When you move MySQL out of the cluster, you leave that bootstrap behind.

So I rebuilt it as an ArgoCD PreSync hook Job that runs before the app pods sync. It connects as the master (from the isolated bootstrap namespace), and idempotently creates both databases, both least-privilege users, and their grants:

Waiting for RDS to accept connections...
mysqld is alive
Connected. Applying schema (idempotent)...
Schema bootstrap complete.

Every statement is IF NOT EXISTS, and the ALTER USER reasserts each app password from its secret on every run. As a result, a rotated credential propagates on the next sync with no manual step, and re-running the hook (or restoring a seeded snapshot) is a no-op. That idempotency is what makes it safe to fire on every sync.

The grants are the part I care about:

GRANT ALL PRIVILEGES ON `ratings`.*  TO `ratings`@`%`
GRANT ALL PRIVILEGES ON `cities`.*   TO `shipping`@`%`

ratings can touch ratings and nothing else. shipping can touch cities and nothing else. The only other grant either user has is USAGE ON *.*, which is just "allowed to log in." That's real least privilege, verified in the database rather than asserted in a comment, and the shipping to cities asymmetry came through correctly, which is exactly the detail that was easy to get wrong.

The 64 MB decision: seed once, snapshot, restore forever

The cities table is ~948,000 rows, 64.5 MB uncompressed, with a FULLTEXT index that MySQL builds inline during load. On a db.t4g.micro (two burstable vCPUs), that index build dominates and burns CPU credits for minutes.

That collides directly with the Phase 8 target of rebuilding the whole platform in under 45 minutes. So I deliberately kept the seed out of the bootstrap job. The job does schema and users only. The 64 MB load is a documented one-time operation followed by an RDS snapshot, and from then on the recovery path is "restore the snapshot," not "re-import 64 MB into a burstable instance on every rebuild." Manual snapshots survive instance deletion, so a seeded snapshot persists across teardown.

It's a small dent in "everything reproducible from git" (one manual step on first bring-up), but it's a documented runbook step rather than tribal knowledge, and it protects the rebuild target. That's the trade I'd make in production too.

The gotcha that actually cost me the evening: an error message that lied

Redis is a StatefulSet with a PersistentVolumeClaim. The claim would not bind. The EBS CSI driver kept logging this, on a loop, creating a fresh orphaned volume each time:

Could not create volume "pvc-...": timed out waiting for volume to create:
InvalidVolume.NotFound: The volume 'vol-...' does not exist.

Create a volume, then immediately can't find it. I spent real time on region mismatches and duplicate controllers before finding the documented behaviour: when the CSI driver's role lacks KMS permissions, it reports InvalidVolume.NotFound instead of an access-denied error. The message names the wrong problem entirely.

My gp3 StorageClass specified an encrypted volume on a customer-managed key. The CSI driver's IAM role had no CreateGrant on that CMK, which is the same class of gap as the AWSServiceRoleForAutoScaling one from Phase 1, just a different principal missing from the same key policy. AWS created the volume, couldn't complete the encryption handshake, rolled it back, and the follow-up describe genuinely found nothing.

To unblock, I dropped the CMK and let the class use the AWS-managed aws/ebs key, which the driver can always use. Restoring the CMK by granting the CSI driver role on the EBS key policy is a tracked follow-up. The lesson stands on its own: an error message is a hypothesis, not a diagnosis. NotFound meant AccessDenied.

There were two smaller traps stacked underneath this one. A StorageClass's parameters are immutable: you can't edit the CMK out, you have to delete and recreate the class, and my ArgoCD sync kept restoring the old one from main faster than I could replace it by hand. GitOps selfHeal fights manual cluster edits; the fix has to land in git first. Furthermore, ArgoCD couldn't update the immutable field either, failing with updates to parameters are forbidden until I added Replace=true to that app's sync options.

The app: 22 manifests, rendered flat

Robot Shop ships as a Helm chart full of {{ .Values }} templating, PodSecurityPolicy gates, and OpenShift Route blocks. That can't go into a GitOps repo as-is. So I rendered it to flat, declarative Kubernetes manifests: one per object, no conditionals; the manifest is the truth

What changed from upstream: dropped the in-cluster mysql deployment entirely (it's RDS now), stripped the Instana tracing env (no agent in this cluster), removed the PSP and Route cruft, and modernized the datastore images. Upstream pins redis:4.0.6 and rabbitmq:3.7, both years old, so they moved to redis:7.2 and rabbitmq:3.13.

The only services that needed real modification were ratings and shipping, and the modification is a single block to inject the whole ESO secret as environment variables:

envFrom:
  - secretRef:
      name: ratings-db

No DB credentials in the manifest. None. That was the entire point.

The services also encode the source-read findings: ratings on port 80, mongodb on 27017, redis on 6379, rabbitmq on 5672 and 15672, and dispatch as a headless service because it only consumes from RabbitMQ and exposes nothing.

Why two pods are red on purpose

Here's the state after deploy: ten services Running. ratings and shipping are not ready, and that's by design, not broken.

The upstream rs-ratings and rs-shipping images hardcode their database passwords in source (iloveit and secret). They only read the host from the environment. But my bootstrap job set the MySQL user passwords to the ESO-generated random values, so an upstream image authenticates with a password RDS doesn't accept, and CrashLoops.

That's a deliberate checkpoint. The envFrom wiring is already correct; the moment Phase 4 builds a patched image that reads credentials from environment variables, these two pods go green with no manifest change. Rather than paper over the hardcoded-credential problem — or worse, weaken the app secrets to match the hardcoded values — I let it fail visibly and documented why. The before/after is a better story than "it just worked," and it sets up the CI/CD phase cleanly.

The storefront, over HTTPS

The last piece is exposure: an Ingress via the AWS Load Balancer Controller, TLS terminated at the ALB with the Phase 1 wildcard cert, and the DNS record created automatically by ExternalDNS. I made the web service a ClusterIP rather than the upstream LoadBalancer because the ALB fronts it, and gave the Ingress a shared ALB group so it can consolidate onto one load balancer with the ArgoCD ingress later, saving ~$20/month per avoided ALB.

Categories load: that's web to catalogue to MongoDB, working end to end. Cart shows empty: Redis, working. Login and Register are live: Mongo and Redis. The whole in-cluster path is proven through a real browser over HTTPS, on a real domain. Ratings widgets and shipping calculations are the two broken features, and now you know exactly why.

What I actually learned during this phase

The blog-friendly version is "I deployed a microservices app to EKS." The real version was a string of failures that each taught me something a tutorial wouldn't.

• Read the source, not the plan. Five wrong assumptions (a second MySQL service, a DSN masquerading as a hostname, a user/database name mismatch, a hidden Redis dependency, a non-standard port) were all caught by reading service code before writing a single manifest. Every one would have been a debugging session downstream.

• Error messages are hypotheses, not diagnoses. The EBS driver said InvalidVolume.NotFound and meant AccessDenied on KMS. The Terraform HCL parser pointed at the line after the real syntax error. RDS said Access denied when the deeper cause was two Terraform resources drifting apart. In every case, the message named a symptom two levels downstream of the cause. The fix was never where the error pointed.

• GitOps selfHeal fights you when you fix things by hand. Three separate times this phase, I edited the cluster directly and ArgoCD dutifully reverted me within seconds, restoring a broken StorageClass or re-applying a stale manifest. Once a resource is under ArgoCD, the change has to land in main. The cluster is not the source of truth; the repo is. That's the entire point of GitOps, and it's easy to forget at 9pm.

• Don't make persistent resources into hooks. I briefly turned the bootstrap ExternalSecrets into PreSync hooks to order them before the job. Hooks are ephemeral (they get cleaned up after the sync), so the secrets vanished and the job timed out waiting for them. ExternalSecrets and ServiceAccounts must persist; the right tool for ordering persistent resources is plain sync-waves, not hooks. Mixing the two lifecycles is a quiet trap.

• Immutable fields need a delete-and-recreate strategy. StorageClass parameters and the RDS managed-secret KMS key are both immutable, and both failed the naive "just change it" apply. Knowing which fields are immutable before you try to change them saves an hour of confusing Forbidden errors and, in ArgoCD's case, needing Replace=true.

• The one that ties it together: moving a database out of Kubernetes is 20% the database and 80% everything around it. The RDS instance was done in Phase 1. Phase 3 was the credential lifecycle, the schema bootstrap that an image used to do for free, the encrypted storage for the caches that stayed behind, the DNS and TLS to reach it, and the honest accounting of what's still hardcoded. That's the actual work of modernization, and it's the part the tutorials skip.

What's next

Continuous Integration and Continuous Delivery (CI/CD): GitHub Actions builds and scans the service images on every pull request, pushes to ECR over OIDC, and updates the GitOps repo — which finally produces the patched ratings and shipping images that read their credentials from the environment and turn those two red pods green. Then Argo Rollouts for canary deploys, so a bad image pauses at 20% traffic instead of taking down the storefront.

GitHub repos: robot-shop-infra · robot-shop-gitOps