Manage Secrets

Learn how to manage secrets in Redpanda Connect, and how to add them to your data pipelines without exposing them.

Secrets are stored in the secret management solution of your cloud provider and are retrieved when you run a pipeline configuration that references them.

Manage secrets

You can manage secrets from the Cloud UI or the Data Plane API.

Create a secret

You can create a secret and reference it in multiple data pipelines on the same cluster.

  • Cloud UI

  • Data Plane API

  1. Log in to Redpanda Cloud.

  2. Go to the Secrets Store page.

  3. Click Create secret.

  4. For ID, enter a name for the secret. You cannot rename the secret once it is created.

  5. For Value, enter the secret you need to add.

  6. For Scopes, select Redpanda Connect.

  7. Optionally, add labels to help organize your secrets.

  8. Click Create.

You can now add the secret to your data pipeline.

You must use a Base64-encoded secret.

  1. Authenticate and get the base URL for the Data Plane API.

  2. Make a request to POST /v1/secrets.

curl -X POST "https://<dataplane-api-url>/v1/secrets" \
 -H 'accept: application/json'\
 -H 'authorization: Bearer <token>'\
 -H 'content-type: application/json' \
 -d '{"id":"<secret-name>","scopes":["SCOPE_REDPANDA_CONNECT"],"secret_data":"<secret-value>"}'

You must include the following values:

  • <dataplane-api-url>: The base URL for the Data Plane API.

  • <token>: The API key you generated during authentication.

  • <secret-name>: The ID or name of the secret you want to add. Use only the following characters: ^[A-Z][A-Z0-9_]*$.

  • <secret-value>: The Base64-encoded secret.

  • This scope: "SCOPE_REDPANDA_CONNECT".

    The response returns the name of the secret and the scope "SCOPE_REDPANDA_CONNECT".

You can now add the secret to your data pipeline.

Update a secret

You can only update the secret value, not its name.

Changes to secret values do not take effect until a pipeline is restarted.

  • Cloud UI

  • Data Plane API

  1. Log in to Redpanda Cloud.

  2. Go to the Secrets Store page.

  3. Find the secret you want to update, and click the edit icon.

  4. Enter the new secret value or labels, and click Update.

  5. Start and stop any pipelines that reference the secret.

You must use a Base64-encoded secret.

  1. Authenticate and get the base URL for the Data Plane API.

  2. Make a request to PUT /v1/secrets/{id}.

    curl -X PUT "https://<dataplane-api-url>/v1/secrets/<secret-name>" \
 -H 'accept: application/json'\
 -H 'authorization: Bearer <token>'\
 -H 'content-type: application/json' \
 -d '{"scopes":["SCOPE_REDPANDA_CONNECT"],"secret_data":"<secret-value>"}'

    You must include the following values:

    • <dataplane-api-url>: The base URL for the Data Plane API.

    • <secret-name>: The name of the secret you want to update.

    • <token>: The API key you generated during authentication.

    • This scope: "SCOPE_REDPANDA_CONNECT".

    • <secret-value>: Your new Base64-encoded secret.

    The response returns the name of the secret and the scope "SCOPE_REDPANDA_CONNECT".

Delete a secret

Before you delete a secret, make sure that you remove references to it from your data pipelines.

Changes do not affect pipelines that are already running.

  • Cloud UI

  • Data Plane API

  1. Log in to Redpanda Cloud.

  2. Go to the Secrets Store page.

  3. Find the secret you want to remove, and click the delete icon.

  4. Confirm your deletion.

  1. Authenticate and get the base URL for the Data Plane API.

  2. Make a request to DELETE /v1/secrets/{id}.

    curl -X DELETE "https://<dataplane-api-url>/v1/secrets/<secret-name>" \
 -H 'accept: application/json'\
 -H 'authorization: Bearer <token>'\

    You must include the following values:

    • <dataplane-api-url>: The base URL for the Data Plane API.

    • <secret-name>: The name of the secret you want to delete.

    • <token>: The API key you generated during authentication.

Add a secret to a data pipeline

  • Cloud UI

  • Data Plane API

  1. Go to the Connect page, and create a pipeline (or open an existing pipeline to edit).

  2. Click the Secret button to add a new or existing secret to the pipeline.

You can add a secret to any pipeline in your cluster using the notation ${secrets.SECRET_NAME}. For example:

sasl:
  - mechanism: SCRAM-SHA-256
    username: "user"
    password: "${secrets.PASSWORD}"