Upgrading RDI

Learn how to upgrade an existing RDI installation

Upgrading a VM installation

Follow the steps below to upgrade an existing VM installation of RDI:

1. Download the RDI installer from the Redis download center (in the Modules, Tools & Integration category) and extract it to your preferred installation folder.

   export RDI_VERSION=2.2.0
   wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-$RDI_VERSION.tar.gz
   tar -xvf rdi-installation-$RDI_VERSION.tar.gz
   

2. Go to the installation folder:

   cd rdi_install/$RDI_VERSION
   

3. Run the upgrade.sh script as a privileged user. Note that you must pass your RDI password to the script unless the password is empty.

   sudo ./upgrade.sh --rdi-password <redis-rdi-password>
   

Recovering from failure during a VM upgrade

If the previous version is v1.4.4 or later, go to the rdi_install/<PREVIOUS_VERSION> directory and run sudo ./upgrade.sh to revert to that version, as described in the section Upgrading a VM installation above.

If the version you are replacing is earlier than v1.4.4, follow these steps. These steps restore and run the CLI binary of the previous RDI version, which still provided the redis-di upgrade command. (On current versions, upgrades are performed with the upgrade.sh script as described above, and redis-di upgrade is no longer a CLI command.)

1. Run redis-di --version to check the current version.

   If the version is the new one, copy the previous version of the RDI CLI to /usr/local/bin with the following command:

   sudo cp rdi_install/<PREVIOUS_VERSION>/deps/rdi-cli/<OS>/redis-di usr/local/bin
   

2. Check that the CLI version is correct by running redis-di --version.

   Then, go to the rdi_install/<PREVIOUS_VERSION> directory and run the following command;

   sudo redis-di upgrade --rdi-host <RDI_REDIS_HOST> --rdi-port <RDI_REDIS_PORT>
   

Upgrading a VM installation with High Availability

If there is an active pipeline, upgrade RDI on the active VM first. This will cause a short pipeline downtime of up to two minutes. Afterwards, wait until redis-di info reports the upgraded VM as the leader, then upgrade RDI on the passive VM. This will not cause any downtime.

Upgrading a Kubernetes installation

Follow the steps below to upgrade an existing Kubernetes installation of RDI:

1. If you are using a private registry, pull the new versions of all images listed in Using a private image registry and add them to your local registry.

2. Download the RDI Helm chart tar file from the Redis download center (in the Modules, Tools & Integration category).

   export RDI_VERSION=2.2.0
   wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-$RDI_VERSION.tgz
   

3. Adapt your rdi-values.yaml file to any changes in the new RDI version if needed. See also Upgrading to RDI 1.8.0 or later from an earlier version. Before making any changes, save your existing rdi-values.yaml if you need to revert to the old RDI version for any reason.

4. Run the helm upgrade command:

   helm upgrade --install rdi rdi-<tag>.tar.gz -f rdi-values.yaml -n rdi
   

Note that you don't need to deploy the RDI configuration again after this step.

Upgrading to RDI 1.8.0 or later from an earlier version

When upgrading to RDI 1.8.0 or later from an earlier version you must adapt your rdi-values.yaml file to the following changes:

• All collector and processor values that were previously under collector, collectorSourceMetricsExporter, and processor have been moved to operator.dataPlane.collector and operator.dataPlane.processor.
• global.collectorApiEnabled has been moved to operator.dataPlane.collectorApi.enabled, and is now a boolean value, not "0" or "1".
• api.authEnabled is also now a boolean value, not "0" or "1".
• The following values have been deprecated: rdiMetricsExporter.service.protocol, rdiMetricsExporter.service.port, rdiMetricsExporter.serviceMonitor.path, api.service.name.

Verifying the upgrade

Check that all pods have Running status:

kubectl get all -n rdi

If you find that the upgrade did not work as expected for any reason, then run the helm upgrade command again (as described in the section Upgrading a Kubernetes installation above), but this time with the previous version you were upgrading from, and using your saved rdi-values.yaml for that version. This will restore your previous working state.

Upgrading to RDI 2.2.0

Check the following changes before you upgrade from RDI 2.0.0. If you upgrade from RDI 1.19.x or earlier, also review Upgrading to RDI 2.0.0.

Upgrade from a version before 1.8.0 in two steps

You cannot upgrade an installation of a version before 1.8.0 directly to RDI 2.2.0. Upgrade to RDI 2.0.0 first, following the RDI 2.0.0 upgrade instructions. Check that the default pipeline runs, and then upgrade to RDI 2.2.0.

Upgrade the leader first

If you run two RDI instances with leader election, upgrade the leader first. Upgrade the other instance only after redis-di info on the upgraded instance shows Leader Election Mode: leader. See Upgrading a VM installation with High Availability. Finish upgrading all instances before creating a pipeline other than default.

The first upgraded instance that becomes the leader saves its pipelines as the backup that RDI restores at every later failover. This matters most when you upgrade from a version before 2.0.0: the other instance does not keep a copy of your pipelines, so if it became the leader first, RDI would delete your pipelines at the next failover.

Validate your configurations before you redeploy them

RDI 2.2.0 rejects some configurations that earlier versions accepted. After the upgrade, check each configuration with redis-di deploy --dry-run before you deploy it again, and fix the following:

• A source named api, or a source name that contains consecutive dashes (--). Rename the source, and update its secrets, its secret references in config.yaml, and the server_name of its jobs. See Name your sources for the naming rules, and Add or remove a source for what renaming a source implies.
• A secret provider name that contains consecutive dashes. Rename the provider, and update the references to it in config.yaml.
• The internal key overrides described in Remove overrides of internal Redis keys.

Remove overrides of internal Redis keys

RDI 2.2.0 always uses the Redis keys and consumer groups that it generates, and rejects the following properties. This example shows where to remove them from your configuration:

sources:
  <source-name>:
    advanced:
      source:
        topic.delimiter: <value> # Rejected since RDI 2.2.0
        offset.storage.redis.key: <value> # Rejected since RDI 2.2.0
        schema.history.internal.redis.key: <value> # Rejected since RDI 2.2.0
      riotx:
        streamPrefix: <value> # Rejected since RDI 2.2.0
processors:
  advanced:
    source:
      stream.name.pattern: <value> # Rejected since RDI 2.2.0
      consumer.group: <value> # Rejected since RDI 2.2.0
    processor:
      counters.key: <value> # Rejected since RDI 2.2.0

The operator uses the generated keys for your existing pipelines as soon as you upgrade, even before you redeploy them, and RDI does not migrate the state stored under custom keys or consumer groups. As a result:

• A Debezium collector that used custom offset or schema history keys takes a new snapshot.
• A Flink processor that used a custom consumer group replays the entries retained in its streams.
• The entries left in streams with custom names or prefixes are no longer consumed.

Remove these properties from your configurations before you deploy them again.

Check which certificate authorities RDI trusts

Debezium collectors and the Collector API now trust the system certificate authorities (CAs) in addition to the CAs of your secrets. To trust only the CAs of your secrets, as before, set operator.dataPlane.collector.useSystemTruststore and operator.dataPlane.collectorApi.useSystemTruststore to false in your Helm values.

Update tools that read the VM log files

On a VM installation, RDI 2.2.0 stores the log files of each pipeline in /opt/rdi/logs/pipelines/<pipeline>/ and those of the control plane in /opt/rdi/logs/control-plane/, with new file names, such as pipelines/default/collector-mysql-collector-source.log. Update any scripts or tools that read the log files by path. RDI does not delete the log files at the previous paths, so delete them once you no longer need them. See RDI logs for the new file names.

Reset MariaDB and SQL Server sources after you add tables

RDI 2.2.0 limits the schema history of MariaDB and SQL Server sources to the tables that they capture, as it already did for MySQL and Oracle sources. If you add a table to the tables list of an existing MariaDB or SQL Server source, or add a database or schema to the ones it captures, reset the source. Otherwise, the source stops with an unknown schema error when the new table first changes. To keep the previous behavior, set schema.history.internal.store.only.captured.tables.ddl to false in the advanced.source section of the source.

Downgrade from RDI 2.2.0

Downgrading to a version earlier than 2.0.0 is not supported. Do not downgrade after you create a pipeline other than default.

Upgrading to RDI 2.0.0

RDI 2.0.0 changes several behaviors that affect a pipeline upgraded from RDI 1.19.x or earlier.

The Flink processor becomes the default

RDI 2.0.0 changes the default processor from classic to flink. This default applies when the pipeline's config.yaml omits processors.type.

For an existing pipeline that uses the classic processor, choose one of these options before upgrading:

• To migrate to the Flink processor, first follow Migrate from the classic processor to the Flink processor on RDI 1.19.0. This stops the collector and drains the input streams before switching processors. Then upgrade RDI.
• To keep the classic processor, set processors.type: classic in the pipeline's config.yaml and deploy it before upgrading.

If your pipeline already uses processors.type: flink, no processor change is needed. Continue with the upgrade instructions for your installation.

Clearing a pipeline is different from deleting it

DELETE /api/v2/pipelines/{name} deletes a pipeline rather than emptying its configuration. In RDI 2.2.0, use... redis-di delete to delete the pipeline, or clear the configuration with redis-di deploy --empty, or with PUT /api/v2/pipelines/{name} and an empty configuration, instead. See Clear a pipeline.

Clearing a pipeline is not the same as resetting it or flushing the target database. Clearing removes the configuration and the pipeline's data from the RDI database, while resetting keeps the configuration and re-snapshots the sources; neither one deletes any records from the target database.

Redeploying a configuration after clearing a pipeline

A configuration of an upgraded pipeline still references the names from before the upgrade, so deploying it again after clearing the pipeline fails. See Redeploying a configuration after clearing a pipeline for what you have to change, with a before and after example.

API changes

RDI 2.0.0 changes several API v2 query parameters and response fields. Most of them affect only an API client, and are listed in the RDI 2.0.0 release notes.

The one to check for is the metric collections endpoint, which now keys data_streams.streams by the source-qualified table name, such as mysql.inventory.addresses, instead of by the Redis stream name. Update any dashboard or script that reads the previous form. See Observability for the metrics RDI reports.

Enabling the Flink processor

The Apache Flink-based stream processor is fully supported on both VM and Kubernetes installations after upgrading to RDI 1.19.0. Once the upgrade completes, it is always available — no opt-in is required, and the defaults are sized for typical workloads.

On Kubernetes, to override the Flink processor defaults, add an operator.dataPlane.flinkProcessor block to your rdi-values.yaml file as described in Configure the Flink processor. On VMs, see Configure the Flink processor. For the per-pipeline migration steps, see Migrate from the classic processor to the Flink processor.

What happens during the upgrade?

The upgrade process replaces the current RDI components with their new versions:

• Firstly, the control plane components are replaced. At this point, the pipeline is still active but monitoring will be disconnected.
• Secondly, the pipeline data plane components are replaced. If a pipeline is active while upgrading, the collector and processor pods will be restarted. The pipeline will pause for up to two minutes but it will catch up very quickly after restarting. The pipeline data and state are both stored in Redis, so data will not be lost during the upgrade.