Connect Redpanda Console to Kafka Connect Clusters

Kafka Connect is community-supported on Redpanda Community Slack. Redpanda Data does not provide enterprise support for Kafka Connect with Redpanda Console. For a supported and scalable Kafka Connect alternative, try Redpanda Connect.
The Redpanda Operator deploys Redpanda Console v2.x, not v3.x.

Redpanda Console v3 is not yet available when deploying with the Redpanda Operator. The Redpanda Operator continues to deploy Redpanda Console v2. To try Redpanda Console v3 in Kubernetes, you can deploy Redpanda using the Redpanda Helm chart instead of the Redpanda Operator.

Redpanda Console configuration syntax varies by major version. Before configuring, determine which version you’re using:

# Check console version from deployment
kubectl get deployment -n <namespace> redpanda-console -o jsonpath='{.spec.template.spec.containers[0].image}'

# Or check from running pod
kubectl get pod -n <namespace> -l app.kubernetes.io/name=console -o jsonpath='{.items[0].spec.containers[0].image}'

# Or check from console logs
kubectl logs -n <namespace> -l app.kubernetes.io/name=console | grep "started Redpanda Console"

If you see output like redpandadata/console:v2.8.0, you’re using Redpanda Console v2.x. If you see redpandadata/console:v3.0.0, you’re using Redpanda Console v3.x.

Redpanda Console provides a user interface that lets you manage multiple Kafka Connect clusters. You can inspect or patch connectors; restart, pause, and resume connector tasks; and delete connectors. Redpanda Console queries all configured Kafka Connect clusters for their status, so you have an overview of all your Kafka Connect clusters.

Prerequisites

You must deploy a Kafka Connect cluster separately before configuring Redpanda Console to connect to it.

Configure a connection to Kafka Connect

For each cluster, provide a unique name, the HTTP address of the cluster, and the authentication settings, if required. The name can be any unique string that helps you to identify the Kafka Connect cluster. See all available configuration options in the Redpanda Console Configuration.

  • Standalone

  • Kubernetes embedded

  • Kubernetes standalone

kafkaConnect:
  enabled: true
  clusters:
    - name: datawarehouse # Required field, will be used as identifier in the frontend
      url: http://dwh-connect.mycompany.com:8083
      tls:
        enabled: false # Trusted certs are still allowed by default
      username: admin
      # password: # Set using flag --kafkaConnect.clusters.0.password=secret
    - name: analytics # Required field, will be used as identifier in the frontend
      url: http://analytics.mycompany.com:8083
      # No auth configured on that cluster, hence no username/password set

When Redpanda Console is part of the Redpanda Helm chart or Operator:

  • Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    console:
      enabled: true
      console:
        config:
          kafkaConnect:
            enabled: true
            clusters:
              - name: datawarehouse # Required field, will be used as identifier in the frontend
                url: http://dwh-connect.mycompany.com:8083
                tls:
                  enabled: false # Trusted certs are still allowed by default
                username: admin
                # password:
              - name: analytics # Required field, will be used as identifier in the frontend
                url: http://analytics.mycompany.com:8083
                # No auth configured on that cluster, hence no username/password set
redpanda-values.yaml
console:
  enabled: true
  console:
    config:
      kafkaConnect:
        enabled: true
        clusters:
          - name: datawarehouse # Required field, will be used as identifier in the frontend
            url: http://dwh-connect.mycompany.com:8083
            tls:
              enabled: false # Trusted certs are still allowed by default
            username: admin
            # password:
          - name: analytics # Required field, will be used as identifier in the frontend
            url: http://analytics.mycompany.com:8083
            # No auth configured on that cluster, hence no username/password set

When using the standalone Redpanda Console Helm chart:

console-values.yaml
config:
  kafkaConnect:
    enabled: true
    clusters:
      - name: datawarehouse # Required field, will be used as identifier in the frontend
        url: http://dwh-connect.mycompany.com:8083
        tls:
          enabled: false # Trusted certs are still allowed by default
        username: admin
        # password:
      - name: analytics # Required field, will be used as identifier in the frontend
        url: http://analytics.mycompany.com:8083
        # No auth configured on that cluster, hence no username/password set