Skip to main content

Major Database Update Guide

Version Upgrade Gate

Starting from version 7.1.2, an automatic gate mechanism is in place that blocks the update process if you attempt to upgrade from Grain versions < 6.6.0 to versions >= 7.1.2.

Please note that for versions between 7.0.0 and 7.1.1, this automatic control is absent. Exercise extreme caution and manually verify compatibility when upgrading within this specific version range.

README

You should not copy & paste commands from this guide, they are provided as an example, but you have to update them to match you current context and database versions.

1. MongoDB​

General Description​

MongoDB is configured as a StatefulSet with data persistence and supports an automated version upgrade mechanism through Feature Compatibility Version (FCV) management.

New Implemented Features​

The ability to configure the Feature Compatibility Version directly from ArgoCD has been added, enabling a controlled and predictable upgrade of feature compatibility.

Update Procedure​

Detailed Steps​

Phase 1: Preparation​

  1. Complete database backup

    mongodump --uri="mongodb+srv://<user>:<pass>@mongodb-svc:27017/?authSource=admin" \
              --out=./backup_pre_upgrade

  2. Update the chart values.yaml

    mongodb:
      enabled: true
      image:
        repository: mongo
        tag: "7.0"  # New major version
      migration:
        enabled: true
        targetFCV: "7.0"  # Set target FCV

Phase 2: Deploy​

  1. Apply changes via ArgoCD
    • Synchronize the chart in the desired namespace
    • The new StatefulSet will have the new image

Phase 3: Automatic migration​

  1. Migration Job
    • A Kubernetes Job is automatically created thanks to the template configuration
    • The job executes the command: mongosh "$MONGO_URI" --eval 'db.adminCommand({setFeatureCompatibilityVersion: "7.0", confirm: true})'
    • The job has a TTL of 300 seconds after completion (automatic cleanup)

Phase 4: Validation​

  1. Verify the FCV 
    mongosh mongodb-svc:27017 -u <user> -p <pass> --authenticationDatabase admin \
             --eval "db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1})"

Rollback Strategies​

If the migration job fails:

  1. Quick diagnostics

    kubectl logs -l batch.kubernetes.io/job-name=mongodb-migration-job-<hash> -f

  2. Image rollback

    • Return to the previous tag in values.yaml
    • Apply the rollback via ArgoCD
    • The old StatefulSet will become active again

  3. Restore from backup (last resort)

    mongorestore --uri="mongodb+srv://<user>:<pass>@mongodb-svc:27017/?authSource=admin" \
                 ./backup_pre_upgrade

Important Notes​

  • ⚠️ FCV must be incremental: Move between consecutive versions (e.g., 5.0 → 6.0 → 7.0)
  • ✅ Idempotent job: Can be re-run without issues
  • 📊 The upgrade is zero-downtime for clients once the FCV is set
  • 🔄 Job TTL: The job is automatically deleted after 5 minutes from completion

2. PostgreSQL​

General Description​

PostgreSQL is configured as a StatefulSet with data persistence. Unlike MongoDB, it does not have an automatic compatibility mechanism for major changes and requires a more conservative and manual approach.

Current Configuration Features​

  • Init Container: Manages automatic data layout migration from PostgreSQL <18 to >=18
  • Storage: Supports both traditional layout and with subPath
  • Architecture duality: Requires different approaches depending on the target architecture

Major Update Procedure​

For non-Power Architectures (x86_64, ARM64)​

On non-Power architectures, it is possible to use pgautoupgrade, a tool that automatically performs the upgrade between PostgreSQL major versions using Debian images.

Detailed Steps:

  1. Pre-Upgrade Backup

    # Access the pod
    kubectl exec -it postgres-0 -- /bin/bash

    # Export everything
    pg_dumpall > /tmp/backup_pre_upgrade.sql

    # Extract the backup from the pod
    kubectl cp postgres-0:/tmp/backup_pre_upgrade.sql ./backup_pre_upgrade.sql

  2. Update values.yaml with the pgautoupgrade image

    postgres:
      enabled: true
      image:
        repository: pgautoupgrade/pgautoupgrade
        tag: "pgautoupgrade-debian-tag"

  3. Apply the change via ArgoCD

    • ArgoCD will synchronize the chart and Kubernetes automatically recreates the pod
    • pgautoupgrade will automatically execute the major version upgrade
    • The init container will automatically handle the data layout migration

    ⚠️ If you need to upgrade multiple major versions using the pgautoupgrade image, you should wait a few moments between updates to ensure the previous migration has completed.

  4. Monitor the upgrade process

    kubectl logs -f postgres-0

    This may take several minutes depending on the database size.

  5. Post-upgrade validation

    kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT version();"

    # Verify data integrity
    kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT relname FROM pg_class LIMIT 5;"

For Power Architectures (ppc64le)​

The pgautoupgrade images are not available for Power. A manual approach is required.

Detailed Steps:

  1. Make a complete backup (as above)

  2. Update values.yaml with the new version

    postgres:
      enabled: true
      image:
        repository: postgres  # Official image (supported on Power)
        tag: "16-alpine"      # New major version

  3. Apply the change via ArgoCD

    • ArgoCD will synchronize the chart and Kubernetes automatically recreates the pod with the new image
    • PostgreSQL will detect the change in the data layout
    • It will automatically execute the necessary upgrades on first start
    • This may take several minutes

  4. Monitor the logs

    kubectl logs -f postgres-0

  5. Post-upgrade validation

    kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT version();"

Rollback Strategy​

In case of problems during the upgrade:

  1. Return to the previous image in values.yaml

    postgres:
      image:
        tag: "15-alpine"  # Previous version

  2. Apply the rollback via ArgoCD

    • Kubernetes automatically recreates the pod with the previous version

  3. Complete restore from backup (last resort)

    # Access the pod
    kubectl exec -it postgres-0 -- /bin/bash

    # Copy the backup to the pod
    kubectl cp ./backup_pre_upgrade.sql postgres-0:/tmp/

    # Restore (with database downtime)
    psql -U postgres < /tmp/backup_pre_upgrade.sql

Important Notes​

  • ⚠️ Significant downtime: The upgrade requires database shutdown
  • 📦 pgautoupgrade not available for Power: Use official images
  • 🔄 Incremental upgrade: Preferably one major version at a time
  • 💾 Mandatory backup: Always backup before any upgrade
  • 🧹 Init Container: Automatically handles layout migration
  • 🔑 Credentials: Must remain consistent before and after the upgrade

3. Redis​

General Description​

Redis is configured as a simple Deployment (not StatefulSet) without data persistence. Therefore, major version upgrades are very simple and without complications.

Configuration Features​

# Minimal deployment, no persistence
kind: Deployment
replicas: 1
containers:
  - name: redis
    image: redis:7-alpine
    ports:
      - containerPort: 6379

Update Procedure​

Steps:

  1. Update values.yaml

    redis:
      enabled: true
      image:
        repository: redis
        tag: "8-alpine"  # New major version

  2. Apply the change via ArgoCD

    • Synchronize the chart
    • The old pod will be terminated and a new one will be started

  3. Validate the new version

    kubectl exec -it redis-<pod-id> -- redis-cli INFO server | grep redis_version

Considerations​

  • ✅ No backup necessary: Redis is used as a cache, data is not persistent
  • ⏱️ Brief downtime: Only the time needed to recreate the pod (seconds)
  • 🔄 Rolling update: The Deployment automatically handles the rollout
  • 💾 Acceptable cache loss: Clients must be prepared to recache data
  • 🧩 Compatibility: Redis maintains good compatibility between consecutive major versions

Rollback (not necessary)​

In the unlikely case of issues:

redis:
  image:
    tag: "7-alpine"  # Return to previous version

Recreate the pod with the previous version.

4. RabbitMQ​

General Description​

RabbitMQ is configured as a StatefulSet with persistence of schema and configuration data. It has an internal compatibility management system that makes major upgrades relatively safe.

Current Configuration Features​

  • StatefulSet with persistence: Message data and configuration are persistent
  • Health checks: Liveness and readiness probes to monitor status
  • Management UI: Available on port 15672
  • AMQP Protocol: Available on port 5672
  • Auto-discovery: Automatic cluster configuration via Kubernetes

Major Update Procedure​

Detailed Steps:

  1. Pre-upgrade Validation

    # Verify cluster status
    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics status

    # Verify active connections
    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics -q listconnections

  2. Update values.yaml

    rabbitmq:
      enabled: true
      image:
        repository: rabbitmq  # Official repository
        tag: "3.13-management-alpine"  # New major version

  3. Apply the change via ArgoCD

    • Synchronize the chart in the namespace
    • RabbitMQ will have a brief downtime during restart

  4. Validate the new version

    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics -q version

  5. Verify the internal schema

    kubectl exec -it rabbitmq-0 -- rabbitmq-diagnostics -q status | grep schema

Automatic Compatibility Management​

RabbitMQ automatically manages compatibility between versions:

  • Schema Management: RabbitMQ updates the internal schema on first start
  • Feature Negotiation: Clients automatically negotiate available features
  • Backward Compatibility: Supports clients of different versions at the same time
  • Non-breaking Changes: The vast majority of major upgrades are non-breaking

Stability Considerations​

  • ✅ Automatic compatibility: No additional configuration needed
  • 📊 Health Checks: Constantly monitor service health
  • 🔄 Persistent Storage: Data is not lost during the upgrade
  • ⏱️ Brief downtime: Only during pod recreation
  • 👥 Multi-tenancy: Supports connection pooling and client multiplexing
  • 🛡️ Transactions: Supports AMQP transactions in all versions

Rollback Strategy​

In case of post-upgrade issues:

  1. Monitor the logs

    kubectl logs -f rabbitmq-0 | grep -i error

  2. If necessary, return to the previous version

    rabbitmq:
      image:
        tag: "3.12-management-alpine"  # Previous version

  3. RabbitMQ will automatically repeat the schema downgrade

    • No manual intervention required
    • Persistent storage still contains the data

Important Notes​

  • ✅ Zero configuration required: RabbitMQ manages everything automatically
  • 📈 Scalability: Supports multi-node clusters (future configuration)
  • 🔒 Security: Credentials managed via Secrets
  • 🎯 Performance: No performance change between consecutive versions
  • 📚 Management Plugin: Included in the rabbitmq:*-management-* image