MySQL Process Definition

This document describes the definition of bulk transfer, batch compare, and CDC processes for MySQL databases in the RDRS Dashboard. Each process type corresponds to a different data access method and use case.

Prerequisites

Before defining processes, MySQL table structures must be imported into the RDRS repository. For more information, refer MySQL Import Table Structure. A MySQL data source with valid connectivity must be configured. For more information, refer Configuring MySQL Access.

Bulk Transfer — Direct Access

RDRS reads MySQL tables directly from the active database using the native MySQL client interface. The agent connects through the data source definition configured in the RDRS repository. This is the primary method for initial data load and full-table refresh operations.

Multiple tables can be selected in a single process definition. Only tables that have been imported into the RDRS repository for the selected data source are available for selection.

To create a new bulk transfer process, navigate to Process Definitions in the Dashboard, select Bulk transfer and batch compare. Select MySQL tables with direct access as the source type.

Process definition parameters:
  • Data source: All defined data sources of type MySQL are shown in the list. A new data source can be created by selecting new Data Source without leaving the process definition. The eye icon shows the properties of the selected data source.
  • Data source agent: The agent used for the input part of the process is derived automatically from the selected data source.
  • Selected tables: The list shows all MySQL tables imported into the repository for the selected data source. Additional tables can be added by selecting Import of metadata from the list without leaving the process definition. For bulk transfer, multiple tables can be selected.
    CDC input

This process type requires the RDRS MySQL Bulk license feature.

Batch Compare — Direct Access

Batch compare reads a single MySQL table directly and compares each record against a snapshot file from a previous run. Only the differences (inserts, updates, deletes) are forwarded to the output. This is useful for tables where CDC is not available or not desired.

  • To create a new batch compare process, navigate to Process Definitions, select Bulk transfer and batch compare.
  • Select MySQL tables with direct access as the source type.
  • Select Batch Compare as the process type.

The common input parameters (data source, data source agent, table selection) are the same as for bulk transfer, except that batch compare processes a single table per process definition.

In addition to the common parameters, batch compare requires snapshot file configuration:
  • Automatic snapshot file management: When enabled, RDRS manages snapshot file names and rotation automatically. Configure the snapshot directory where files are stored.
  • Output snapshot file name: The snapshot file to write for this run. After completion, this file becomes the input snapshot for the next run.
  • Input snapshot file name: The snapshot file from a previous run to compare against. On the first run, leave empty — RDRS treats all records as inserts.

This process type requires the RDRS MySQL Compare license feature.

CDC — MySQL CDC Function

Change data capture from MySQL databases. RDRS reads change records from MySQL binary log files using the mysqlbinlog (or mysqlbinlog) utility. This is the primary method for continuous replication from MySQL.

CDC Prerequisites

The MySQL server must be configured for binary logging before CDC processes can be defined. For a full list of database prerequisites (binary logging, ROW format, binlog_row_image, user privileges), refer Configuring MySQL Access.

Process Definition Parameters

To create a new CDC process, navigate to Process Definitions, select Active logs / realtime CDC, and select MySQL CDC function as the source type.

Process definition parameters:
  • Data source: All defined data sources of type MySQL are shown in the list. A new data source can be created by selecting new Data Source without leaving the process definition. The eye icon shows the properties of the selected data source.
  • Data source agent: The agent used for the input part of the process is derived automatically from the selected data source.
  • Logreader method: Select the method for reading binary logs:
    • Log directory derived from database: RDRS reads the log_bin_basename variable from the MySQL server and uses the result as the path to the binary log files. This is the default method.
    • Given log directory: The binary log directory is specified manually. Use this when the binary logs are not in the default location.
    • Logreader directory: When "Given log directory" is selected, enter the path to the directory containing the binary log files.
  • Beginning of the replication (position): Select from the available options:
    • Start newly at runtime: CDC begins from the current binary log position at process start time.
    • Use saved start point: Resumes from a previously saved position (if status information exists from a prior run).
    • Manual specification of start point: Allows entering a specific position to start from.
  • Manual specification of position: Defines the restart point of data retrieval when starting the process. The position identifies the location in the MySQL binary log. The value must have the following format:

    Format: xxxxxxxx,yyyyyyyyyyyy

    • Comma‑separated values
    • First part: Binary log file identifier
    • Second part: Position within that log file

    Use one of the following SQL commands:

    SHOW MASTER STATUS

    The format is f,p where f is the log file number (up to 8 digits) and p is the position in the log file (up to 12 digits).

    To find the position for a specific point in time, use the mysql-binlog utility (formerly mysqlbinlog) with the --start-datetime option: 
    mysql-binlog --start-datetime='2025-06-15 08:00:00' mysql-bin.000005
    The first event in the output shows the file position to use. Alternatively, use SHOW BINLOG EVENTS IN ''mysql-bin.000005' to scan events and identify the position closest to the desired time.
    Note: MySQL does not provide a built-in SQL function to convert a timestamp directly into a binary log position.

If this process has already been executed and status information has been written, processing will automatically continue at the correct position in the binary logs. If no status information exists and no explicit start position is defined, CDC data capture begins at the time of process start.

Additional controls:

  • Show saved status information: Displays the current restart state saved by a previous run.
  • Remove saved status information: Deletes any saved restart state, causing the next run to start fresh.

This process type requires the RDRS MySQL Log or DBMS license feature.

Realtime Specific Settings

Realtime specific settings section is an optional section that controls the CDC process behavior during periods of inactivity when no new change data is available.

  • To add this section, click the More settings button at the bottom of the process definition.
  • Select Realtime specific settings from the list. The section is not shown by default if all values are at their defaults.

    For more information, refer Real-time Specific Settings – Other Data Sources.

Known limitations and constraints

  • MySQL version: CDC via binary log reading is supported for MySQL 5.7 through 9.0. Bulk transfer and batch compare require MySQL 8.0 or later.
  • Binary log format: The binary log format must be set to ROW. STATEMENT and MIXED formats are not supported for CDC.
  • binlog_row_image: Must be set to "full" to ensure complete before and after images for UPDATE and DELETE operations.
  • mysqlbinlog (or mysqlbinlog) utility: Must be installed and accessible on the agent host. The version must be compatible with the MySQL server version.
  • Table selection: Only tables that have been imported into the RDRS repository are available for selection in the process definition. Use "Import of metadata" from within the process definition to add new tables.
  • Loopback prevention: In bidirectional CDC environments, configure the loopback prevention database name in the output target definition to prevent processing of changes made by RDRS itself.
  • GTID support: MySQL uses its own GTID format (domain-id–server-id–sequence). RDRS handles the MySQL GTID format internally for position tracking.
  • Platform: All MySQL process types (bulk, compare, CDC) can run on any supported platform (Windows, Linux, Unix) where the MySQL client library and mysqlbinlog utility are accessible to the RDRS agent.