Deploy and configure IBM Cloudant’s Kafka Connector in IBM Event Streams.

Learning objectives

In this tutorial, you will learn how to configure IBM Cloudant Kafka Source Connector and deploy it into Event Streams that running in IBM Cloud Pak for Integration.

Prerequisites

To complete this tutorial, you need

· IBM Cloud account.

· Red Hat OpenShift cluster in your preferred Cloud. In this tutorial, I’m using IBM Red Hat OpenShift Cluster.

· Cloud Pak for Integration in you preferred Cloud. In this tutorial, I’m using Cloud Pak for Integration in IBM Cloud.

· Event Streams Cluster. (To Setup your Cluster. Please refer to this tutorial.)

. Kafka Connect Cluster. (To Setup your Cluster please refer to this tutorial.)

. OpenShift CLI.

Steps:

1. First you need to download kafka-cloudant-connect jar. It is available to download from Maven Central.

2. Navigate to the directory where you extracted the Kafka Connect ZIP file. You can refer to Set up a Kafka Connect tutorial. Copy Cloudant connector jar files to the my-plugins folder. If the connector has multiple jar files, you will need to compile all of the jars in one folder under the main folder “my-plugins”.

3. Build of KafkaConnect using the below command:

oc start-build YOUR_CONTAINER_NAME --from-dir my-plugins/ -n YOUR_NAMESPACE

The result will be something similar to the below result:

Uploading directory "my-plugins" as binary input for the build ...
...
  Uploading finished
  build.build.openshift.io/demo-connect-cluster-connect-2 started

4. Next step is to configure your connector using a KafkaConnector resource. Create a cloudant-source-demo.yaml. You can refer to the below KafkaConnector template.

apiVersion: eventstreams.ibm.com/v1alpha1
kind: KafkaConnector
metadata:
  name: my-connector
  labels:
    eventstreams.ibm.com/cluster: my-connect-cluster
spec:
  class: <connector-class>
  tasksMax: 1
  config:
    key: value
    # ...

5. Enter your Cloudant Connect name.

6. Enter the Event Streams Cluster name.

7. Enter Cloudant source Connector class. Please refer to Cloudant connector documentations.

8. Enter the Topic name. The topic will be created if your topic is not exist before.

9. Enter your Cloudant database’s url. The url should contain the hostname and database name.

10. Enter your Cloudant database’s username.

11. Enter your Cloudant database’s password.

The demo is designed for simplicity for illustration purposes. You can refer to Kafka Connect and Cloudant connector to build any complex transformations.

12. Run The below command to start your connector.

oc apply -f cloudant-source-demo.yaml -n YOUR_NAMESPACE

13. Validate your connector deployment by running the below command.

oc describe kafkaconnector YOUR_CONNECTOR_NAME -n YOUR_NAMESPACE

YOUR_CONNECTOR_NAME is the name that you used in this Part in Step 5.

14. Now the connector is ready to watch Cloudant database and stream the changes to Kafka.

15. Open Your Cloudant database and create or update existing document.

16. Navigate to your Event Streams dashboard. Click Topics.

17. You will find the topic is created. Click topic name.

18. Click Messages.

19. You will find the messages that you updated or created sit in the topic. Click the message row.

20. The message’s payload will be displayed on the right side.

Now, you setup Kafka Connect to capture any insert/update in Cloudant database.