Configure Message Deserialization in Redpanda Console

The Redpanda Operator deploys Redpanda Console v2.x, not v3.x.

Redpanda Console v3 is not yet available when deploying with the Redpanda Operator. The Redpanda Operator continues to deploy Redpanda Console v2. To try Redpanda Console v3 in Kubernetes, you can deploy Redpanda using the Redpanda Helm chart instead of the Redpanda Operator.

Redpanda Console configuration syntax varies by major version. Before configuring, determine which version you’re using:

# Check console version from deployment
kubectl get deployment -n <namespace> redpanda-console -o jsonpath='{.spec.template.spec.containers[0].image}'

# Or check from running pod
kubectl get pod -n <namespace> -l app.kubernetes.io/name=console -o jsonpath='{.items[0].spec.containers[0].image}'

# Or check from console logs
kubectl logs -n <namespace> -l app.kubernetes.io/name=console | grep "started Redpanda Console"

If you see output like redpandadata/console:v2.8.0, you’re using Redpanda Console v2.x. If you see redpandadata/console:v3.0.0, you’re using Redpanda Console v3.x.

Redpanda Console provides tools for deserializing and inspecting messages in Kafka topics. This topic explains how to configure Redpanda Console to use Schema Registry, Protobuf files, and other deserialization methods to ensure your data is correctly interpreted and displayed.

Use Schema Registry

The Schema Registry allows Redpanda Console to dynamically retrieve schemas for deserializing Avro, Protobuf, and JSON messages. This setup is important to ensure that messages are correctly interpreted based on your schema definitions.

See Configure access to the Schema Registry.

Protobuf configuration

Redpanda Console supports several methods for providing Protobuf schemas, including the Schema Registry, local file system, and GitHub repositories.

You don’t need to provide standard types, such as Google’s timestamp, in your schemas. These standard types are included by default.

Most Kafka clients that serialize Protobuf messages put the serialized byte array into a binary wrapper that contains meta information, like the schema ID or the used prototypes, so the application that deserializes the Kafka records must recognize the format. The deserialization process requires Redpanda Console to be aware of the used Protobuf files as well as a mapping of what prototype should be used for each topic. This information can either be sourced from the Schema Registry or it can be provided with additional configuration so the files can be pulled from the local file system or a GitHub repository.

Use Schema Registry for Protobuf

If you use a Schema Registry for Protobuf deserialization, Redpanda Console can automatically fetch and use the required schemas without the need for manual topic mappings or additional configuration.

When using Schema Registry for Protobuf, you must not configure serde.protobuf. Redpanda Console detects and uses the Protobuf schemas from the Schema Registry automatically.

If you configure serde.protobuf, it enables manual deserialization mode, which requires you to specify topic mappings and source providers. Without those, Redpanda Console fails to start due to validation errors.

When using Schema Registry for Protobuf, you do not need to provide specific topic mappings, as the schemas will be fetched dynamically.

Topic mapping

If you’re not using a Schema Registry for Protobuf deserialization, you must manually provide mappings between Kafka topics and their corresponding Protobuf types. This is necessary to inform Redpanda Console of the correct types to use for deserialization.

Consider a Kafka topic called address-v1 and a corresponding address.proto file with the following structure:

address.proto
syntax = "proto3";
package fake_models;

option go_package = "pkg/protobuf";

message Address {
  int32 version = 1;
  string id = 2;
  message Customer {
    string customer_id = 1;
    string customer_type = 2;
  }
}

To map this topic to the Protobuf schema, use the following configuration:

  • Standalone

  • Kubernetes embedded

  • Kubernetes standalone

serde:
  protobuf:
    enabled: true
    mappings:
    - topicName: address-v1
      valueProtoType: fake_models.Address # The full Protobuf type name
      # keyProtoType: Not specified because the key is a plain string

When Redpanda Console is part of the Redpanda Helm chart or Operator:

  • Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    console:
      enabled: true
      console:
        config:
          serde:
            protobuf:
              enabled: true
              mappings:
              - topicName: address-v1
                valueProtoType: fake_models.Address # The full Protobuf type name
                # keyProtoType: Not specified because the key is a plain string
redpanda-values.yaml
console:
  enabled: true
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
          - topicName: address-v1
            valueProtoType: fake_models.Address # The full Protobuf type name
            # keyProtoType: Not specified because the key is a plain string

When using the standalone Redpanda Console Helm chart:

console-values.yaml
config:
  serde:
    protobuf:
      enabled: true
      mappings:
      - topicName: address-v1
        valueProtoType: fake_models.Address # The full Protobuf type name
        # keyProtoType: Not specified because the key is a plain string

  • serde.protobuf.enabled: Set to true to enable Protobuf deserialization.

  • serde.protobuf.mappings.topicName: The name of the Kafka topic.

  • serde.protobuf.mappings.valueProtoType: The fully-qualified Protobuf type for the message value.

  • serde.protobuf.mappings.keyProtoType: Specify the key Protobuf type if the key is not a plain string.

Local file system

You can mount Protobuf files directly from your local file system. Redpanda Console will search the specified paths for Protobuf files and build a registry with all the available types.

Configuration example:

  • Standalone

  • Kubernetes embedded

  • Kubernetes standalone

serde:
  protobuf:
    enabled: true
    mappings:
      - topicName: orders
        valueProtoType: fake_models.Order
        keyProtoType: fake_models.OrderKey
    fileSystem:
      enabled: true
      # How often to refresh the Protobuf files from the filesystem
      refreshInterval: 5m
      # Directories containing the Protobuf files
      paths:
        - /etc/protos

When using the Redpanda Operator or the Redpanda Helm chart, configure Protobuf deserialization through the cluster configuration:

  • Operator

  • Helm

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    console:
      enabled: true
      console:
        config:
          serde:
            protobuf:
              enabled: true
              mappings:
                - topicName: orders
                  valueProtoType: fake_models.Order
                  keyProtoType: fake_models.OrderKey
              fileSystem:
                enabled: true
                refreshInterval: 5m
                paths:
                  - /etc/protos
console:
  enabled: true
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: orders
              valueProtoType: fake_models.Order
              keyProtoType: fake_models.OrderKey
          fileSystem:
            enabled: true
            refreshInterval: 5m
            paths:
              - /etc/protos

When using the standalone Redpanda Console Helm chart:

config:
  serde:
    protobuf:
      enabled: true
      mappings:
        - topicName: orders
          valueProtoType: fake_models.Order
          keyProtoType: fake_models.OrderKey
      fileSystem:
        enabled: true
        refreshInterval: 5m
        paths:
          - /etc/protos

Apply with:

helm upgrade --install redpanda-console redpanda/console -f console-values.yaml

  • serde.protobuf.enabled: Set to true to enable Protobuf deserialization.

  • serde.protobuf.fileSystem.paths: Paths to directories where Protobuf files are stored.

  • serde.protobuf.fileSystem.refreshInterval: The frequency at which Redpanda Console checks for updates to these files.

GitHub repository

If your Protobuf files are stored in a GitHub repository, Redpanda Console can fetch and use them directly. This is particularly useful if your organization maintains Protobuf definitions in version control.

Configuration example:

  • Standalone

  • Kubernetes embedded

  • Kubernetes standalone

serde:
  protobuf:
    enabled: true
    mappings:
      - topicName: orders
        valueProtoType: fake_models.Order
        keyProtoType: fake_models.OrderKey
    git:
      enabled: true
      repository:
        url: https://github.com/myorg/kafka-proto-files.git
        branch: master
      # How often to pull the git repository to refresh the schema files
      refreshInterval: 10m
      # Where all .proto files are stored in the git repository
      paths:
        - ./

When using the Redpanda Operator or the Redpanda Helm chart, configure GitHub repository access through the cluster configuration:

  • Operator

  • Helm

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    console:
      enabled: true
      console:
        config:
          serde:
            protobuf:
              enabled: true
              mappings:
                - topicName: orders
                  valueProtoType: fake_models.Order
                  keyProtoType: fake_models.OrderKey
              git:
                enabled: true
                repository:
                  url: https://github.com/myorg/kafka-proto-files.git
                  branch: master
                refreshInterval: 10m
                paths:
                  - ./
console:
  enabled: true
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: orders
              valueProtoType: fake_models.Order
              keyProtoType: fake_models.OrderKey
          git:
            enabled: true
            repository:
              url: https://github.com/myorg/kafka-proto-files.git
              branch: master
            refreshInterval: 10m
            paths:
              - ./

When using the standalone Redpanda Console Helm chart:

config:
  serde:
    protobuf:
      enabled: true
      mappings:
        - topicName: orders
          valueProtoType: fake_models.Order
          keyProtoType: fake_models.OrderKey
      git:
        enabled: true
        repository:
          url: https://github.com/myorg/kafka-proto-files.git
          branch: master
        refreshInterval: 10m
        paths:
          - ./

Apply with:

helm upgrade --install redpanda-console redpanda/console -f console-values.yaml

  • serde.protobuf.enabled: Set to true to enable Protobuf deserialization.

  • serde.protobuf.git.repository.url: The URL of the GitHub repository containing your Protobuf files.

  • serde.protobuf.git.basicAuth: Basic authentication credentials, often an API token for private repositories.

  • serde.protobuf.git.refreshInterval: Frequency at which the repository is polled for updates.

Private Git repositories

If Protobuf files are stored in a private GitHub repository, Redpanda Console must authenticate using one of the following methods:

Authenticate using a GitHub Personal Access Token (PAT)

Use this method to authenticate to GitHub over HTTPS using a personal access token.

  • Standalone

  • Kubernetes embedded

  • Kubernetes standalone

  1. Set environment variables:

    SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME=token
SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD=<github-pat>

  2. Configure Redpanda Console:

    serde:
  protobuf:
    enabled: true
    mappings:
      - topicName: <kafka-topic-name>
        valueProtoType: <protobuf-message-type>
    git:
      enabled: true
      repository:
        url: https://github.com/<github-organization>/<repository-name>.git
        branch: <branch-name>
      refreshInterval: 10m
      paths:
        - ./
      basicAuth:
        enabled: true

  1. Create a secret:

    apiVersion: v1
kind: Secret
metadata:
  name: protobuf-git-auth
  namespace: redpanda
type: Opaque
stringData:
  SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD: <github-pat>

  2. Reference the secret and set the username:

    • Operator

    • Helm

    apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    console:
      enabled: true
      extraEnv:
        - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME
          value: token
      extraEnvFrom:
        - secretRef:
            name: protobuf-git-auth
      console:
        config:
          serde:
            protobuf:
              enabled: true
              mappings:
                - topicName: <kafka-topic-name>
                  valueProtoType: <protobuf-message-type>
              git:
                enabled: true
                repository:
                  url: https://github.com/<github-organization>/<repository-name>.git
                  branch: <branch-name>
                refreshInterval: 10m
                paths:
                  - ./
                basicAuth:
                  enabled: true
    console:
  enabled: true
  extraEnv:
    - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME
      value: token
  extraEnvFrom:
    - secretRef:
        name: protobuf-git-auth
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: <kafka-topic-name>
              valueProtoType: <protobuf-message-type>
          git:
            enabled: true
            repository:
              url: https://github.com/<github-organization>/<repository-name>.git
              branch: <branch-name>
            refreshInterval: 10m
            paths:
              - ./
            basicAuth:
              enabled: true

  1. Create a secret:

    apiVersion: v1
kind: Secret
metadata:
  name: protobuf-git-auth
  namespace: redpanda
type: Opaque
stringData:
  SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD: <github-pat>

  2. Update Helm values:

    config:
  serde:
    protobuf:
      enabled: true
      mappings:
        - topicName: <kafka-topic-name>
          valueProtoType: <protobuf-message-type>
      git:
        enabled: true
        repository:
          url: https://github.com/<github-organization>/<repository-name>.git
          branch: <branch-name>
        refreshInterval: 10m
        paths:
          - ./
        basicAuth:
          enabled: true

extraEnv:
  - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME
    value: token

extraEnvFrom:
  - secretRef:
      name: protobuf-git-auth

Replace the following values:

  • <github-pat>: A GitHub personal access token that has repo scope

  • <kafka-topic-name>: Kafka topic to be deserialized

  • <protobuf-message-type>: Fully qualified Protobuf message type (for example, com.example.Order)

  • <github-organization>: GitHub organization or user that owns the repository

  • <repository-name>: Name of the repository containing .proto files

  • <branch-name>: Git branch to clone (for example, main)

Authenticate using SSH

Use this method to authenticate with GitHub over SSH using a private key.

  • Standalone

  • Kubernetes embedded

  • Kubernetes standalone

  1. Save the SSH private key on the local filesystem (for example, /etc/redpanda/ssh/id_rsa).

  2. Set environment variables:

    SERDE_PROTOBUF_GIT_SSH_ENABLED=true
SERDE_PROTOBUF_GIT_SSH_USERNAME=git
SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH=/etc/redpanda/ssh/id_rsa
SERDE_PROTOBUF_GIT_SSH_PASSPHRASE=<private-key-passphrase>

  3. Configure Redpanda Console:

    serde:
  protobuf:
    enabled: true
    mappings:
      - topicName: <kafka-topic-name>
        valueProtoType: <protobuf-message-type>
    git:
      enabled: true
      repository:
        url: git@github.com:<github-organization>/<repository-name>.git
        branch: <branch-name>
      refreshInterval: 10m
      paths:
        - ./
      ssh:
        enabled: true

  1. Create a secret with the SSH key:

    apiVersion: v1
kind: Secret
metadata:
  name: protobuf-git-ssh
  namespace: redpanda
type: Opaque
stringData:
  privateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    <ssh-private-key>
    -----END OPENSSH PRIVATE KEY-----
  passphrase: <private-key-passphrase>

  2. Mount the secret and configure environment variables:

    • Operator

    • Helm

    apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    console:
      enabled: true
      extraVolumeMounts:
        - name: git-ssh
          mountPath: /etc/git-ssh
          readOnly: true
      extraVolumes:
        - name: git-ssh
          secret:
            secretName: protobuf-git-ssh
      extraEnv:
        - name: SERDE_PROTOBUF_GIT_SSH_ENABLED
          value: "true"
        - name: SERDE_PROTOBUF_GIT_SSH_USERNAME
          value: git
        - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH
          value: /etc/git-ssh/privateKey
        - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE
          value: <private-key-passphrase>
    console:
  enabled: true
  extraVolumeMounts:
    - name: git-ssh
      mountPath: /etc/git-ssh
      readOnly: true
  extraVolumes:
    - name: git-ssh
      secret:
        secretName: protobuf-git-ssh
  extraEnv:
    - name: SERDE_PROTOBUF_GIT_SSH_ENABLED
      value: "true"
    - name: SERDE_PROTOBUF_GIT_SSH_USERNAME
      value: git
    - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH
      value: /etc/git-ssh/privateKey
    - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE
      value: <private-key-passphrase>

  1. Create a secret:

    apiVersion: v1
kind: Secret
metadata:
  name: protobuf-git-ssh
  namespace: redpanda
type: Opaque
stringData:
  privateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    <ssh-private-key>
    -----END OPENSSH PRIVATE KEY-----
  passphrase: <private-key-passphrase>

  2. Update Helm values:

    config:
  serde:
    protobuf:
      enabled: true
      mappings:
        - topicName: <kafka-topic-name>
          valueProtoType: <protobuf-message-type>
      git:
        enabled: true
        repository:
          url: git@github.com:<github-organization>/<repository-name>.git
          branch: <branch-name>
        refreshInterval: 10m
        paths:
          - ./
        ssh:
          enabled: true

extraVolumeMounts:
  - name: git-ssh
    mountPath: /etc/git-ssh
    readOnly: true

extraVolumes:
  - name: git-ssh
    secret:
      secretName: protobuf-git-ssh

extraEnv:
  - name: SERDE_PROTOBUF_GIT_SSH_ENABLED
    value: "true"
  - name: SERDE_PROTOBUF_GIT_SSH_USERNAME
    value: git
  - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH
    value: /etc/git-ssh/privateKey
  - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE
    value: <private-key-passphrase>

Replace the following values:

  • <kafka-topic-name>: Kafka topic to be deserialized

  • <protobuf-message-type>: Fully qualified Protobuf message type (for example, com.example.Order)

  • <github-organization>: GitHub organization or user that owns the repository

  • <repository-name>: Name of the repository containing .proto files

  • <branch-name>: Git branch to clone (for example, main)

  • <ssh-private-key>: SSH private key content used to authenticate to GitHub (must be base64-safe if stored in secrets)

  • <private-key-passphrase>: Passphrase used to decrypt the SSH private key, if applicable

MessagePack deserialization

If your data is serialized using MessagePack, Redpanda Console can be configured to deserialize it.

  • Standalone

  • Kubernetes embedded

  • Kubernetes standalone

serde:
  messagePack:
    enabled: true
    # Define which topics use MessagePack serialization
    # Regex to match all topics by default
    topicNames: ["/.*/"]

When using the Redpanda Operator or the Redpanda Helm chart, configure MessagePack deserialization through the cluster configuration:

  • Operator

  • Helm

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    console:
      enabled: true
      console:
        config:
          serde:
            messagePack:
              enabled: true
              topicNames: ["/.*/"]
console:
  enabled: true
  console:
    config:
      serde:
        messagePack:
          enabled: true
          topicNames: ["/.*/"]

When using the standalone Redpanda Console Helm chart:

config:
  serde:
    messagePack:
      enabled: true
      topicNames: ["/.*/"]

Apply with:

helm upgrade --install redpanda-console redpanda/console -f console-values.yaml

  • serde.messagePack.enabled: Enables MessagePack deserialization.

  • serde.messagePack.topicNames: A list of topic name regex patterns that specify which topics use MessagePack serialization. The default pattern (/.*/) matches all topics.

Best practices

  • Use Schema Registry when possible.

    Schema Registry simplifies schema management and ensures that all messages are serialized and deserialized consistently across your Kafka ecosystem.

  • Organize Protobuf files.

    Whether using a local file system or a GitHub repository, keep your Protobuf files organized and use consistent naming conventions to avoid confusion.

  • Monitor deserialization performance.

    Regularly check the performance impact of deserialization, especially when using complex Protobuf schemas or large numbers of messages. Adjust refresh intervals and schema caching as needed.

  • Secure access.

    Ensure that credentials for accessing the Schema Registry or GitHub repositories are securely managed and rotated regularly.

Troubleshooting

If you encounter issues with deserialization:

  • Ensure that the Schema Registry URL and credentials are correctly configured and accessible.

  • Check your topic mappings and Protobuf type names for accuracy.

  • Review the Redpanda Console for insights into any errors occurring during deserialization.