A command line client for Kafka Connect

Overview

🧸 kcctl – Your Cuddly CLI for Apache Kafka Connect

This project is a command-line client for Kafka Connect. Relying on the idioms and semantics of kubectl, it allows you to register and examine connectors, delete them, restart them, etc. You can see what kcctl is about in this short video:

kcctl Intro

Installation

You can obtain early access binaries of kcctl (x86) for Linux, macOS, and Windows from here. This is a rolling release, new binaries are published upon each commit pushed to the kcctl repository.

Note: on macOS, you need to remove the quarantine flag after downloading, as the distribution currently is not signed:

xattr -r -d com.apple.quarantine path /to/kcctl-1.0.0-SNAPSHOT-osx-x86_64/

We're planning to publish kcctl binaries via SDKMAN! soon, too.

It is recommended to install the bash/zsh completion script kcctl_completion:

. kcctl_completion

Usage

Display the help to learn about using kcctl:

kcctl help
Usage: kcctl [COMMAND]
A command-line interface for Kafka Connect
Commands:
  info      Displays information about the Kafka Connect cluster
  config    Sets or retrieves the configuration of this client
  get       Displays information about connector plug-ins, connectors, and
              loggers
  describe  Displays detailed information about the specified resource
  apply     Applies the given file for registering or updating a connector
  patch     Modifies the configuration of a connector or logger
  restart   Restarts a connector or task
  delete    Deletes the specified connector
  help      Displays help information about the specified command

Development

This project uses Quarkus, the Supersonic Subatomic Java Framework.

If you want to learn more about Quarkus, please visit its website: https://quarkus.io/ .

Running the application in dev mode

You can run your application in dev mode that enables live coding using:

./mvnw compile quarkus:dev

To seed the command line arguments, pass the -Dquarkus.args option:

./mvnw compile quarkus:dev -Dquarkus.args='patch get connectors'

In dev mode, remote debuggers can connect to the running application on port 5005. In order to wait for a debugger to connect, pass the -Dsuspend option.

Packaging and running the application

The application can be packaged using:

./mvnw package

It produces the quarkus-run.jar file in the target/quarkus-app/ directory. Be aware that it’s not an über-jar as the dependencies are copied into the target/quarkus-app/lib/ directory.

The application is now runnable using java -jar target/quarkus-app/quarkus-run.jar. You should define an alias kcctl:

alias kcctl="java -jar target/quarkus-app/quarkus-run.jar"

Creating a native executable

You can create a native executable using:

./mvnw package -Pnative

You can then execute your native executable with: ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner

As above, either define an alias kcctl or rename the resulting executable accordingly.

Updating the Completion Script

Grep for usages of DummyCompletions and uncomment them (they are used as placeholders in the generated completion script).

Build the application in JVM mode.

Recreate the completion script:

java -cp "target/quarkus-app/app/*:target/quarkus-app/lib/main/*:target/quarkus-app/quarkus-run.jar" \
  picocli.AutoComplete -n kcctl --force dev.morling.kccli.command.KcCtlCommand

Edit the completion scrpt kcctl_completion, replace all the dummy completion placeholders with invocations of one of the (hidden) completion candidate commands, e.g. like so:

--- local CONNECTOR_NAME_pos_param_args="connector-1 connector-2 connector-3" # 0-0 values
+++ local CONNECTOR_NAME_pos_param_args=`kcctl connector-name-completions` # 0-0 values

Related Quarkus Guides

  • Picocli (guide): Develop command line applications with Picocli
  • Quarkus native apps (guide): Develop native applications with Quarkus and GraalVM

License

This code base is available ander the Apache License, version 2.

Issues
  • Expand notion of configuration context

    Expand notion of configuration context

    In order to support display (#2) and reset (#8) of connector offsets, the concept of the configuration context should be expanded. We need to have the information about the Kafka bootstrap server(s) as well as the name of the offset topic. The suggestion is to introduce a notion of named context which take all the required configuration and allowing the the user to switch between them (#19).

    # local is the name of the context
    $ kcctl config set-context local --cluster=my-connect:8083 --bootstrap-servers=my-kafka:9092 --offset-topic=my-offsets
    Using context local
    

    The information about all the configured contexts would be stored in JSON format in the ~/.kcctl configuration file. Commands that don't need the additional attributes, should gracefully handle the current format of ~/.kcctl (solely containing the KC URL). Commands which do need the additional attributes, should raise an error when encountering a file in the legacy format, indicating how to use set-context for updating it.

    opened by gunnarmorling 15
  • Add support for multiple configuration contexts

    Add support for multiple configuration contexts

    Hi @gunnarmorling,

    here comes the PR! I hope I've got the the requirements right 😄.

    One thing I would like to challenge is supporting the old, properties-like config format. Because we are still pre 1.0.0, I guess it would be acceptable to not support the current (properties-like) configuration format anymore. Wdyt?

    The current, context-aware format is very flexible, except for the context-name and cluster URL, every other property is optional.

    So both

    {
        "currentContext": "local",
        "local": {
            "cluster": "http://localhost:8083"
        }
    }
    

    and

    {
        "currentContext": "local",
        "local": {
            "cluster": "http://localhost:8083",
            "boostrapServers": "localhost:9092",
            "offsetTopic": "connect-offsets"
        }
    }
    

    are valid.

    Fixes #24.

    opened by helpermethod 13
  • added config get-context command and description in set-context #3

    added config get-context command and description in set-context #3

    added config get-context command and description in set-context

    opened by SiddiqueAhmad 9
  • Move repo to kcctl organization

    Move repo to kcctl organization

    Having a separate org is nicer in terms of collaboration.

    @aalmiray, I've already created the org https://github.com/kcctl and am planning to move this repo over there. I assume this will require some changes to the release infra as well? Should I first do the move and you then do the required updates? Or the other way around? Thx a lot!

    opened by gunnarmorling 9
  • Add describe tasks-config command

    Add describe tasks-config command

    This maps to /connectors//tasks-config and returns the configuration of all tasks of a connector. Usage: kcctl describe tasks-config

    opened by mimaison 8
  • 🦉 Configure JReleaser via its Maven plugin

    🦉 Configure JReleaser via its Maven plugin

    Fix for #26

    opened by aalmiray 8
  • Boostrapping Patch command

    Boostrapping Patch command

    https://github.com/gunnarmorling/kcctl/issues/15

    Notes:

    • Added API, helper class and patch command.
    • Tested this locally:
      • mvn compile quarkus:dev -Dquarkus.args='patch logger com.snowflake.kafka.connector -l INFO' -Dsuspend
    • Can add more comments and actual logic in next commits.
    • API for reference: Link
    opened by sfc-gh-japatel 8
  • handled exception, and displayed list of available connectors #32

    handled exception, and displayed list of available connectors #32

    handled exception, and displayed list of available connectors

    opened by SiddiqueAhmad 7
  • added version command #7

    added version command #7

    added version command, application version and name is used for print.

    Screenshot from 2021-08-13 19-41-33

    opened by SiddiqueAhmad 6
  • Bump quarkus.platform.version from 2.1.4.Final to 2.2.3.Final

    Bump quarkus.platform.version from 2.1.4.Final to 2.2.3.Final

    Bumps quarkus.platform.version from 2.1.4.Final to 2.2.3.Final. Updates quarkus-universe-bom from 2.1.4.Final to 2.2.3.Final

    Commits
    • 9723371 [maven-release-plugin] prepare release 2.2.3.Final
    • d28406c Merge pull request #314 from quarkusio/2.2-backports
    • 34d062a Upgrade to Quarkus 2.2.3.Final
    • 0060b9e Kogito 1.11.1.Final + OptaPlanner 8.11.1.Final
    • 35090e9 Include kubernetes as dev-support and opentracing as deprecated
    • e8a57ea Use com.redhat.quarkus groupId for the quarkus-universe-bom in the rhproduct ...
    • ff8a0c8 Updating JSON file to include jdbc-oracle support
    • bf4d6e9 Kogito 1.11.0.Final + OptaPlanner 8.11.0.Final
    • 9cc0e81 [maven-release-plugin] prepare for next development iteration
    • 653fb36 [maven-release-plugin] prepare release 2.2.2.Final
    • Additional commits viewable in compare view

    Updates quarkus-maven-plugin from 2.1.4.Final to 2.2.3.Final

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies java 
    opened by dependabot[bot] 0
  • Feature/kcctl 17

    Feature/kcctl 17

    Added a simple implementation for basic authentication.

    This implementation follows roughly how kubectl formerly handled basic auth (I believe basic auth is now deprecated in Kubernetes, but don't quote me on that). Credentials may be added directly to a context in .kcctl or set using the set-context command.

    Documentation was added to README and tests added for some relevant areas.

    I chose to keep this implementation very simple. Additional items should be able to be added via KafkaConnectClientHeadersFactory in the future.

    opened by tonyfosterdev 3
  • dev.morling.kccli.service.KafkaConnectException: Request timed out

    dev.morling.kccli.service.KafkaConnectException: Request timed out

    Hi Team,

    Am getting timeouts for kcctl describe connector * or kcctl get connectors Let me know how I can increase timeout

    good first issue 
    opened by gcpcloudreddy 11
  • Suppress stack trace in case of invalid configuration

    Suppress stack trace in case of invalid configuration

    When trying to register a connector whose configuration is invalid, let's only display the actual validation error message(s), but not the stacktrace.

    good first issue 
    opened by gunnarmorling 0
  • completion not working for ZSH using oh-my-zsh

    completion not working for ZSH using oh-my-zsh

    When copying the completlion file to ${ZSH}/completions using oh-my-zsh plugins no completion is available.

    sourcing the file directly in .zshrc file works.

    opened by hguerrero 0
  • Have tests for kcctl

    Have tests for kcctl

    Currently, no tests exist for kcctl at all. We should add tests for the different commands, so to avoid regressions down the road. As part of this task, we need to

    • Understand how PicoCLI applications are tested; is their any guidance in their docs?
    • Set up the infrastructure for integration tests via Testcontainers; we'll need to start up ZooKeeper, Kafka, and Kafka Connect for testing kcctl; as inspiration or starting point, here's a Docker Compose file which shows the required components; as a test connector, the built-in file connector could probably be used
    opened by gunnarmorling 6
  • Simplify generation of completion script

    Simplify generation of completion script

    The idea is:

    • Un-comment the current dummy completions
    • Replace them with two actual completion implementations (one for connector list, one for task list), which return the corresponding kcctl command (e.g. "kcctl", "get-connectors-list")

    Then, when running the generate command as described in the README, only one slight manual adjustment will be needed for replacing the quotes from the command with backticks, so to make this an actual command invocation.

    Subsquently, there should be a command in kcctl for completion script generation (as described in the PicoCLI docs) instead of checking in the script into the kcctl repo. This command implementation ideally will take care of that backtick adjustment above.

    opened by gunnarmorling 4
  • Allow to update connector configuration via patch

    Allow to update connector configuration via patch

    E.g. like this:

    kcctl patch connector etcd-test -p '{"tasks.max":3}'
    kcctl patch connector etcd-test -p '{"tasks.max":3, "database.name":"foo"}'
    
    opened by gunnarmorling 7
  • 📦 Publish a container image for kcctl to DockerHub

    📦 Publish a container image for kcctl to DockerHub

    Hey @aalmiray, is this something JReleaser would help with as well?

    opened by gunnarmorling 8
  • Rename artifactId to kcctl

    Rename artifactId to kcctl

    opened by aalmiray 4
Owner
kcctl
Your cuddly command-line tool for working with Kafka Connect
kcctl
Publish Kafka messages from HTTP

Kafka Bridge Publish Kafka messages from HTTP Configuration Example configuration for commonly used user + password authentication: kafka-bridge: ka

neuland - Büro für Informatik 3 Sep 15, 2021
Mirror of Apache Kafka

Apache Kafka See our web site for details on the project. You need to have Java installed. We build and test Apache Kafka with Java 8, 11 and 15. We s

The Apache Software Foundation 19.8k Sep 10, 2021
A command line client for Kafka Connect

?? kcctl – Your Cuddly CLI for Apache Kafka Connect This project is a command-line client for Kafka Connect. Relying on the idioms and semantics of ku

kcctl 77 Sep 10, 2021
A command line client for Kafka Connect

kcctl -- A CLI for Apache Kafka Connect This project is a command-line client for Kafka Connect. Relying on the idioms and semantics of kubectl, it al

Gunnar Morling 79 Sep 16, 2021
Java client for NATS

NATS - Java Client A Java client for the NATS messaging system. A Note on Versions This is version 2.x of the java-nats library. This version is a gro

NATS - The Cloud Native Messaging System 361 Sep 13, 2021
Dataflow template which read data from Kafka (Support SSL), transform, and outputs the resulting records to BigQuery

Kafka to BigQuery Dataflow Template The pipeline template read data from Kafka (Support SSL), transform the data and outputs the resulting records to

DoiT International 12 Jun 1, 2021
A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

Nakadi Event Broker Nakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues, which can be used

Zalando SE 777 Sep 19, 2021
EventStoreDB is the database for Event Sourcing. This repository provides a sample of event sourced system that uses EventStoreDB as event store.

Event Sourcing with EventStoreDB Introduction Example Domain Event Sourcing and CQRS 101 State-Oriented Persistence Event Sourcing CQRS Advantages of

Evgeniy Khyst 15 Aug 16, 2021
2021年更新开源数字货币合约交易所,基于Java开发的比特币交易所 | BTC交易所 | ETH交易所 | 数字货币交易所 | 交易平台 | 撮合交易引擎。本项目有完整的撮合交易引擎源码、后台管理(后端+前端)、前台(交易页面、活动页面、个人中心等)、安卓APP源码、苹果APP源码、币种钱包RPC源码。开源项目仅供学习参考,请勿用于非法用途

一、项目目标 安卓+IOS+PC+钱包 2021年新版更新 ZTuo数字资产交易平台(以下简称“ZTuo”)是ZTuo团队历经了多个数字资产交易系统开发之后,通过不断的优化精简,首家开源的Java语言开发的数字货币交易所平台,支持当前多种主流的数字货币交易(测试环境受限于服务器的成本,目前支持USD

null 120 Sep 10, 2021
RabbitMQ Java client

RabbitMQ Java Client This repository contains source code of the RabbitMQ Java client. The client is maintained by the RabbitMQ team at Pivotal. Depen

RabbitMQ 1k Sep 16, 2021
Fast and reliable message broker built on top of Kafka.

Hermes Hermes is an asynchronous message broker built on top of Kafka. We provide reliable, fault tolerant REST interface for message publishing and a

Allegro Tech 688 Aug 27, 2021
Flink Demo

flink-demo minimum code just run flink-ds-connector DataStream API usage kafka es jdbc file row string parquet avro avro custom avro flink-sql-connect

hiscat 25 Sep 17, 2021
Apache Pulsar - distributed pub-sub messaging system

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. Learn more about Pulsar at https:

The Apache Software Foundation 9.6k Sep 16, 2021
A high available,high performance distributed messaging system.

#新闻 MetaQ 1.4.6.2发布。更新日志 MetaQ 1.4.6.1发布。更新日志 MetaQ 1.4.5.1发布。更新日志 MetaQ 1.4.5发布。更新日志 Meta-ruby 0.1 released: a ruby client for metaq. SOURCE #介绍 Meta

dennis zhuang 1.3k Sep 1, 2021
Efficient reliable UDP unicast, UDP multicast, and IPC message transport

Aeron Efficient reliable UDP unicast, UDP multicast, and IPC message transport. Java and C++ clients are available in this repository, and a .NET clie

Real Logic 5.7k Sep 15, 2021
A JVM library to use RabbitMQ as an embedded service

Embedded RabbitMQ Compatibility: Builds: Linux OS X Windows Reports: Dist: Social: This library allows for the use of various RabbitMQ versions as if

Alejandro Rivera 85 Sep 2, 2021
RocketMQ-on-Pulsar - A protocol handler that brings native RocketMQ protocol to Apache Pulsar

RocketMQ on Pulsar(RoP) RoP stands for RocketMQ on Pulsar. Rop broker supports RocketMQ-4.6.1 protocol, and is backed by Pulsar. RoP is implemented as

StreamNative 65 Sep 7, 2021
CookieClient is a utility client for anarchy servers

CookieClient CookieClient is a utility client for anarchy servers. Its a forge mod so you need forge to run it. Currently supported versions: 1.12.2 D

null 38 Sep 15, 2021
A modular and portable open source XMPP client library written in Java for Android and Java (SE) VMs

Smack About Smack is an open source, highly modular, easy to use, XMPP client library written in Java for Java SE compatible JVMs and Android. A pure

Ignite Realtime 2.3k Sep 16, 2021