Skip to main content

Major Database Update Guide

:::info Version Upgrade Gate Starting from version 7.1.2, an automatic gate mechanism is in place that blocks the update process if you attempt to upgrade from Grain versions < 6.6.0 to versions >= 7.1.2.

Please note that for versions between 7.0.0 and 7.1.1, this automatic control is absent. Exercise extreme caution and manually verify compatibility when upgrading within this specific version range. :::

:::warning README You should not copy & paste commands from this guide, they are provided as an example, but you have to update them to match you current context and database versions. :::

1. MongoDB​

General Description​

MongoDB is configured as a StatefulSet with data persistence and supports an automated version upgrade mechanism through Feature Compatibility Version (FCV) management.

New Implemented Features​

The ability to configure the Feature Compatibility Version directly from ArgoCD has been added, enabling a controlled and predictable upgrade of feature compatibility.

Update Procedure​

Detailed Steps​

Phase 1: Preparation​

  1. Complete database backup

    mongodump --uri="mongodb+srv://<user>:<pass>@mongodb-svc:27017/?authSource=admin" \
              --out=./backup_pre_upgrade

  2. Update the chart values.yaml

    mongodb:
      enabled: true
      image:
        repository: mongo
        tag: "7.0"  # New major version
      migration:
        enabled: true
        targetFCV: "7.0"  # Set target FCV

Phase 2: Deploy​

  1. Apply changes via ArgoCD
    • Synchronize the chart in the desired namespace
    • The new StatefulSet will have the new image

Phase 3: Automatic migration​

  1. Migration Job
    • A Kubernetes Job is automatically created thanks to the template configuration
    • The job executes the command: mongosh "$MONGO_URI" --eval 'db.adminCommand({setFeatureCompatibilityVersion: "7.0", confirm: true})'
    • The job has a TTL of 300 seconds after completion (automatic cleanup)

Phase 4: Validation​

  1. Verify the FCV 
    mongosh mongodb-svc:27017 -u <user> -p <pass> --authenticationDatabase admin \
             --eval "db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1})"

Rollback Strategies​

If the migration job fails:

  1. Quick diagnostics

    kubectl logs -l batch.kubernetes.io/job-name=mongodb-migration-job-<hash> -f

  2. Image rollback

    • Return to the previous tag in values.yaml
    • Apply the rollback via ArgoCD
    • The old StatefulSet will become active again

  3. Restore from backup (last resort)

    mongorestore --uri="mongodb+srv://<user>:<pass>@mongodb-svc:27017/?authSource=admin" \
                 ./backup_pre_upgrade

Important Notes​

  • ⚠️ FCV must be incremental: Move between consecutive versions (e.g., 5.0 → 6.0 → 7.0)
  • ✅ Idempotent job: Can be re-run without issues
  • 📊 The upgrade is zero-downtime for clients once the FCV is set
  • 🔄 Job TTL: The job is automatically deleted after 5 minutes from completion

2. PostgreSQL​

General Description​

PostgreSQL is configured as a StatefulSet with data persistence. Unlike MongoDB, it does not have an automatic compatibility mechanism for major changes and requires a more conservative and manual approach.

Current Configuration Features​

  • Init Container: Manages automatic data layout migration from PostgreSQL <18 to >=18
  • Storage: Supports both traditional layout and with subPath
  • Architecture duality: Requires different approaches depending on the target architecture

Major Update Procedure​

For non-Power Architectures (x86_64, ARM64)​

On non-Power architectures, it is possible to use pgautoupgrade, a tool that automatically performs the upgrade between PostgreSQL major versions using Debian images.

Detailed Steps:

  1. Pre-Upgrade Backup

    # Access the pod
    kubectl exec -it postgres-0 -- /bin/bash

    # Export everything
    pg_dumpall > /tmp/backup_pre_upgrade.sql

    # Extract the backup from the pod
    kubectl cp postgres-0:/tmp/backup_pre_upgrade.sql ./backup_pre_upgrade.sql

  2. Update values.yaml with the pgautoupgrade image

    postgres:
      enabled: true
      image:
        repository: pgautoupgrade/pgautoupgrade
        tag: "pgautoupgrade-debian-tag"

  3. Apply the change via ArgoCD

    • ArgoCD will synchronize the chart and Kubernetes automatically recreates the pod
    • pgautoupgrade will automatically execute the major version upgrade
    • The init container will automatically handle the data layout migration

    ⚠️ If you need to upgrade multiple major versions using the pgautoupgrade image, you should wait a few moments between updates to ensure the previous migration has completed.

  4. Monitor the upgrade process

    kubectl logs -f postgres-0

    This may take several minutes depending on the database size.

  5. Post-upgrade validation

    kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT version();"

    # Verify data integrity
    kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT relname FROM pg_class LIMIT 5;"

For Power Architectures (ppc64le)​

The pgautoupgrade images are not available for Power. A manual approach is required.

Detailed Steps:

  1. Make a complete backup (as above)

  2. Update values.yaml with the new version

    postgres:
      enabled: true
      image:
        repository: postgres  # Official image (supported on Power)
        tag: "16-alpine"      # New major version

  3. Apply the change via ArgoCD

    • ArgoCD will synchronize the chart and Kubernetes automatically recreates the pod with the new image
    • PostgreSQL will detect the change in the data layout
    • It will automatically execute the necessary upgrades on first start
    • This may take several minutes

  4. Monitor the logs

    kubectl logs -f postgres-0

  5. Post-upgrade validation

    kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT version();"

Rollback Strategy​

In case of problems during the upgrade:

  1. Return to the previous image in values.yaml

    postgres:
      image:
        tag: "15-alpine"  # Previous version

  2. Apply the rollback via ArgoCD

    • Kubernetes automatically recreates the pod with the previous version

  3. Complete restore from backup (last resort)

    # Access the pod
    kubectl exec -it postgres-0 -- /bin/bash

    # Copy the backup to the pod
    kubectl cp ./backup_pre_upgrade.sql postgres-0:/tmp/

    # Restore (with database downtime)
    psql -U postgres < /tmp/backup_pre_upgrade.sql

Important Notes​

  • ⚠️ Significant downtime: The upgrade requires database shutdown
  • 📦 pgautoupgrade not available for Power: Use official images
  • 🔄 Incremental upgrade: Preferably one major version at a time
  • 💾 Mandatory backup: Always backup before any upgrade
  • 🧹 Init Container: Automatically handles layout migration
  • 🔑 Credentials: Must remain consistent before and after the upgrade

3. Redis​

General Description​

Redis is configured as a simple Deployment (not StatefulSet) without data persistence. Therefore, major version upgrades are very simple and without complications.

Configuration Features​

# Minimal deployment, no persistence
kind: Deployment
replicas: 1
containers:
  - name: redis
    image: redis:7-alpine
    ports:
      - containerPort: 6379

Update Procedure​

Steps:

  1. Update values.yaml

    redis:
      enabled: true
      image:
        repository: redis
        tag: "8-alpine"  # New major version

  2. Apply the change via ArgoCD

    • Synchronize the chart
    • The old pod will be terminated and a new one will be started

  3. Validate the new version

    kubectl exec -it redis-<pod-id> -- redis-cli INFO server | grep redis_version

Considerations​

  • ✅ No backup necessary: Redis is used as a cache, data is not persistent
  • ⏱️ Brief downtime: Only the time needed to recreate the pod (seconds)
  • 🔄 Rolling update: The Deployment automatically handles the rollout
  • 💾 Acceptable cache loss: Clients must be prepared to recache data
  • 🧩 Compatibility: Redis maintains good compatibility between consecutive major versions

Rollback (not necessary)​

In the unlikely case of issues:

redis:
  image:
    tag: "7-alpine"  # Return to previous version

Recreate the pod with the previous version.

4. RabbitMQ​

General Description​

RabbitMQ is configured as a StatefulSet with persistence of schema and configuration data. It has an internal compatibility management system that makes major upgrades relatively safe.

Current Configuration Features​

  • StatefulSet with persistence: Message data and configuration are persistent
  • Health checks: Liveness and readiness probes to monitor status
  • Management UI: Available on port 15672
  • AMQP Protocol: Available on port 5672
  • Auto-discovery: Automatic cluster configuration via Kubernetes

Major Update Procedure​

Detailed Steps:

  1. Pre-upgrade Validation

    # Verify cluster status
    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics status

    # Verify active connections
    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics -q listconnections

  2. Update values.yaml

    rabbitmq:
      enabled: true
      image:
        repository: rabbitmq  # Official repository
        tag: "3.13-management-alpine"  # New major version

  3. Apply the change via ArgoCD

    • Synchronize the chart in the namespace
    • RabbitMQ will have a brief downtime during restart

  4. Validate the new version

    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics -q version

  5. Verify the internal schema

    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics -q status | grep schema

Automatic Compatibility Management​

RabbitMQ automatically manages compatibility between versions:

  • Schema Management: RabbitMQ updates the internal schema on first start
  • Feature Negotiation: Clients automatically negotiate available features
  • Backward Compatibility: Supports clients of different versions at the same time
  • Non-breaking Changes: The vast majority of major upgrades are non-breaking

Stability Considerations​

  • ✅ Automatic compatibility: No additional configuration needed
  • 📊 Health Checks: Constantly monitor service health
  • 🔄 Persistent Storage: Data is not lost during the upgrade
  • ⏱️ Brief downtime: Only during pod recreation
  • 👥 Multi-tenancy: Supports connection pooling and client multiplexing
  • 🛡️ Transactions: Supports AMQP transactions in all versions

Rollback Strategy​

In case of post-upgrade issues:

  1. Monitor the logs

    kubectl logs -f rabbitmq-0 | grep -i error

  2. If necessary, return to the previous version

    rabbitmq:
      image:
        tag: "3.12-management-alpine"  # Previous version

  3. RabbitMQ will automatically repeat the schema downgrade

    • No manual intervention required
    • Persistent storage still contains the data

Important Notes​

  • ✅ Zero configuration required: RabbitMQ manages everything automatically
  • 📈 Scalability: Supports multi-node clusters (future configuration)
  • 🔒 Security: Credentials managed via Secrets
  • 🎯 Performance: No performance change between consecutive versions
  • 📚 Management Plugin: Included in the rabbitmq:*-management-* image