Compare commits

..

63 Commits
v3.8.50 ... v4

Author SHA1 Message Date
vtolstov
c4f8de4aca Apply Code Coverage Badge 2025-07-15 13:02:27 +00:00
63d82be92a add additional metadata to message, closes #194
Some checks failed
sync / sync (push) Failing after 12m3s
coverage / build (push) Failing after 13m41s
test / test (push) Successful in 4m0s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-07-15 15:59:05 +03:00
cced3b84c1 remove noisy logs from event hooks and add tests (#193)
Some checks failed
coverage / build (push) Failing after 1m50s
test / test (push) Failing after 19m17s
sync / sync (push) Successful in 7s
2025-07-10 12:51:26 +03:00
vtolstov
82c1bda50f Apply Code Coverage Badge 2025-07-02 13:21:33 +00:00
4b0b70e7d0 fixup message options and content-type header
Some checks failed
coverage / build (push) Successful in 4m34s
test / test (push) Failing after 17m28s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-07-02 16:20:02 +03:00
vtolstov
da88718e03 Apply Code Coverage Badge 2025-06-30 13:11:01 +00:00
aed304b7e3 rework message options
Some checks failed
sync / sync (push) Successful in 1m2s
coverage / build (push) Successful in 3m31s
test / test (push) Failing after 18m28s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-06-30 16:08:59 +03:00
b796c43257 [v4] hide access to internal mutex (#185)
Some checks failed
coverage / build (push) Failing after 2m4s
test / test (push) Successful in 3m53s
sync / sync (push) Successful in 8s
* changed embedded mutex to private field

* update ci

* fix tests
2025-06-10 16:28:17 +03:00
b0691edf82 disable message pool by default
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-06-10 16:28:16 +03:00
d43b59f2f3 disable message pool by default
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-06-10 16:28:03 +03:00
f976516d6b new kgo version
update deps

fixup race conditions

add kfake usage

Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-06-10 16:27:40 +03:00
8f5c03ad60 [v4] hide access to internal mutex (#185)
* changed embedded mutex to private field

* update ci

* fix tests
2025-06-10 16:26:52 +03:00
vtolstov
9e9c1434ff Apply Code Coverage Badge 2025-06-09 14:30:14 +00:00
fe81a14413 update for latest micro
Some checks failed
coverage / build (push) Successful in 3m22s
test / test (push) Failing after 19m17s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-06-09 17:27:45 +03:00
eb56a48dda update kgo
Some checks failed
coverage / build (push) Has been cancelled
test / test (push) Has been cancelled
sync / sync (push) Failing after 28s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-05-25 01:25:56 +03:00
71e8a13c70 new kgo version
Some checks failed
sync / sync (push) Has been cancelled
coverage / build (push) Failing after 1m31s
test / test (push) Failing after 18m29s
update deps

fixup race conditions

add kfake usage

Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-05-25 01:24:50 +03:00
b538ef82b5 [v4] hide access to internal mutex (#185)
* changed embedded mutex to private field

* update ci

* fix tests
2025-05-25 01:23:46 +03:00
44e2d5f9a4 update ci (#188) 2025-05-25 01:23:11 +03:00
vtolstov
9426208e1c Apply Code Coverage Badge 2025-05-21 08:14:38 +00:00
d6e73a3419 minimize checks on publish
Some checks failed
coverage / build (push) Successful in 4m51s
sync / sync (push) Has been skipped
test / test (push) Failing after 15m31s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-05-21 09:54:16 +03:00
969e459e3d add tracer enabled status
Some checks failed
test / test (push) Failing after 17m28s
coverage / build (push) Failing after 17m37s
sync / sync (push) Has been cancelled
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-05-19 09:34:24 +03:00
vtolstov
b2cd7d8b8e Apply Code Coverage Badge 2025-05-18 16:02:55 +00:00
6d85d3ee41 disable message pool by default
Some checks failed
sync / sync (push) Has been skipped
test / test (push) Successful in 4m47s
coverage / build (push) Failing after 18m8s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-05-18 18:58:57 +03:00
d0978fb314 Обновить README.md
Some checks failed
coverage / build (push) Failing after 18m14s
test / test (push) Successful in 3m43s
sync / sync (push) Has been skipped
2025-05-12 11:05:30 +03:00
vtolstov
0bd81d1253 Apply Code Coverage Badge 2025-05-10 22:53:42 +00:00
e34f57a515 new kgo version
All checks were successful
sync / sync (push) Has been skipped
coverage / build (push) Successful in 4m56s
test / test (push) Successful in 4m59s
update deps

fixup race conditions

add kfake usage

Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-05-11 01:49:38 +03:00
ccd912adb2 dont modify connected state on errors
Some checks failed
coverage / build (push) Failing after 1m1s
test / test (push) Successful in 5m12s
sync / sync (push) Has been skipped
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-05-02 19:08:54 +03:00
db97b24904 improve sync
All checks were successful
sync / sync (push) Successful in 47s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-04-27 21:02:58 +03:00
7410408fc4 update deps
Some checks failed
coverage / build (push) Failing after 1m8s
test / test (push) Successful in 3m27s
sync / sync (push) Successful in 10s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-04-27 16:12:32 +03:00
4d28a6ebb2 update actions
Some checks failed
coverage / build (push) Failing after 1m24s
sync / sync (push) Successful in 13s
test / test (push) Successful in 2m8s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-04-27 16:09:17 +03:00
f73d9d7347 fixup group lag metric
All checks were successful
test / test (push) Successful in 3m49s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-04-23 10:51:39 +03:00
53f72f1246 fixup panic
All checks were successful
test / test (push) Successful in 1m57s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-03-12 20:34:30 +03:00
ac454693a6 update to v4
All checks were successful
test / test (push) Successful in 3m29s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-03-12 16:57:25 +03:00
f4f8793686 fixup connected status
All checks were successful
test / test (push) Successful in 2m25s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-03-07 11:45:36 +03:00
ec4922ad8b fixup connected status
Some checks failed
test / test (push) Has been cancelled
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-03-07 11:44:56 +03:00
eaea14e5a8 fixup connected status
All checks were successful
test / test (push) Successful in 3m3s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-03-07 11:26:34 +03:00
f6f7139d2f fixup panic
All checks were successful
test / test (push) Successful in 2m4s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-03-03 10:43:41 +03:00
51e4118dfc cleanup
All checks were successful
test / test (push) Successful in 2m19s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-02-27 12:41:42 +03:00
aaf8c43e04 add ability to fail probes and fatal on broker errors
All checks were successful
test / test (push) Successful in 3m12s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-02-26 11:06:11 +03:00
cfecb4afd0 export broker state
All checks were successful
test / test (push) Successful in 3m16s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2025-02-25 17:30:48 +03:00
ae4ae64694 added set ct for records && updated setHeaders (#150)
All checks were successful
test / test (push) Successful in 3m45s
Reviewed-on: #150
Co-authored-by: Evstigneev Denis <danteevstigneev@yandex.ru>
Co-committed-by: Evstigneev Denis <danteevstigneev@yandex.ru>
2025-01-21 15:29:02 +03:00
2ca4e5c74e Merge pull request 'update gomod' (#149) from devstigneev/micro-broker-kgo:v3 into v3
All checks were successful
test / test (push) Successful in 3m26s
Reviewed-on: #149
2025-01-16 16:15:13 +03:00
2809ebcaeb update gomod
All checks were successful
lint / lint (pull_request) Successful in 2m11s
test / test (pull_request) Successful in 4m8s
2025-01-16 16:13:05 +03:00
3f04403319 Merge pull request 'update deps && structs && hooks' (#147) from devstigneev/micro-broker-kgo:v3 into v3
All checks were successful
test / test (push) Successful in 3m47s
Reviewed-on: #147
2024-12-20 00:02:24 +03:00
8edfc11eef rebase v3
All checks were successful
lint / lint (pull_request) Successful in 1m7s
test / test (pull_request) Successful in 2m49s
2024-12-19 23:42:51 +03:00
cb5ef75e09 Merge remote-tracking branch 'origin/v3' into v3
# Conflicts:
#	go.sum
#	subscriber.go
2024-12-19 23:42:16 +03:00
d5bd105cc6 undo comments 2024-12-19 23:41:57 +03:00
1ba02ed8ad resolve conflict 2024-12-19 23:41:55 +03:00
7c5da60556 update deps && structs && hooks 2024-12-19 23:39:40 +03:00
07fa36d704 update go.mod
All checks were successful
test / test (push) Successful in 4m32s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2024-12-19 21:58:34 +03:00
849bbd7a09 undo comments
Some checks failed
test / test (pull_request) Failing after 1m17s
lint / lint (pull_request) Successful in 1m52s
2024-12-19 13:42:25 +03:00
f461bb6876 resolve conflict
Some checks failed
lint / lint (pull_request) Has been cancelled
test / test (pull_request) Has been cancelled
2024-12-19 13:36:55 +03:00
201d22d1c4 Merge remote-tracking branch 'fork/v3' into v3
# Conflicts:
#	.gitignore
#	event.go
#	go.mod
#	go.sum
#	kgo.go
#	kgo_test.go
#	subscriber.go
2024-12-19 13:33:40 +03:00
ff6a272594 update deps && structs && hooks 2024-12-19 13:31:34 +03:00
1018abe7bd update deps && structs && hooks
Some checks failed
codeql / analyze (go) (pull_request) Has been cancelled
prbuild / test (pull_request) Has been cancelled
prbuild / lint (pull_request) Has been cancelled
2024-12-19 13:22:30 +03:00
398c3c81cb Merge pull request 'Upd workflows' (#146) from atolstikhin/micro-broker-kgo:v3 into v3
All checks were successful
test / test (push) Successful in 12m59s
Reviewed-on: #146
2024-12-11 23:12:57 +03:00
Aleksandr Tolstikhin
786d03b521 Upd workflows
Some checks failed
autoapprove / autoapprove (pull_request) Successful in 16s
automerge / automerge (pull_request) Has been skipped
lint / lint (pull_request) Failing after 1m19s
test / test (pull_request) Successful in 1m15s
dependabot-automerge / automerge (pull_request) Failing after 14m40s
2024-12-11 22:17:33 +07:00
937c9d5720 Merge pull request 'Update script of jobs' (#142) from atolstikhin/micro-broker-kgo:v3 into v3
Some checks failed
codeql / analyze (go) (push) Failing after 46s
test / test (push) Failing after 5m33s
Reviewed-on: #142
2024-12-09 19:20:31 +03:00
Aleksandr Tolstikhin
951fba55fa Merge remote-tracking branch 'rem/v3' into v3
Some checks failed
automerge / automerge (pull_request) Has been skipped
dependabot-automerge / automerge (pull_request) Has been skipped
autoapprove / autoapprove (pull_request) Successful in 11s
codeql / analyze (go) (pull_request) Failing after 49s
test / test (pull_request) Failing after 5m38s
lint / lint (pull_request) Failing after 15m2s
2024-12-09 22:21:40 +07:00
Aleksandr Tolstikhin
ec5238ed14 Fix by code review 2024-12-09 22:17:44 +07:00
d8f44a924e Merge pull request 'Update workflows' (#141) from atolstikhin/micro-broker-kgo:v3 into v3
Some checks failed
codeql / analyze (go) (push) Failing after 48s
test / test (push) Failing after 13m41s
Reviewed-on: #141
2024-12-09 16:25:10 +03:00
Aleksandr Tolstikhin
ffe9e5d952 Update workflows
Some checks failed
automerge / automerge (pull_request) Has been skipped
dependabot-automerge / automerge (pull_request) Has been skipped
autoapprove / autoapprove (pull_request) Successful in 6s
codeql / analyze (go) (pull_request) Failing after 52s
lint / lint (pull_request) Failing after 10m30s
test / test (pull_request) Has been cancelled
2024-12-09 18:35:02 +07:00
8c362fd6ae fixup panic
Some checks failed
codeql / analyze (go) (push) Failing after 49s
build / test (push) Failing after 4m53s
build / lint (push) Successful in 9m29s
Signed-off-by: Vasiliy Tolstov <v.tolstov@unistack.org>
2024-12-05 09:53:40 +03:00
29 changed files with 1519 additions and 750 deletions

View File

@@ -1,19 +0,0 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2
updates:
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
# Maintain dependencies for Golang
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: "daily"

View File

@@ -1,20 +0,0 @@
name: "autoapprove"
on:
pull_request_target:
types: [assigned, opened, synchronize, reopened]
permissions:
pull-requests: write
contents: write
jobs:
autoapprove:
runs-on: ubuntu-latest
steps:
- name: approve
uses: hmarr/auto-approve-action@v3
if: github.actor == 'vtolstov' || github.actor == 'dependabot[bot]'
id: approve
with:
github-token: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -1,21 +0,0 @@
name: "automerge"
on:
pull_request_target:
types: [assigned, opened, synchronize, reopened]
permissions:
pull-requests: write
contents: write
jobs:
automerge:
runs-on: ubuntu-latest
if: github.actor == 'vtolstov'
steps:
- name: merge
id: merge
run: gh pr merge --auto --merge "$PR_URL"
env:
PR_URL: ${{github.event.pull_request.html_url}}
GITHUB_TOKEN: ${{secrets.TOKEN}}

View File

@@ -1,47 +0,0 @@
name: build
on:
push:
branches:
- master
- v3
jobs:
test:
name: test
runs-on: ubuntu-latest
steps:
- name: setup
uses: actions/setup-go@v3
with:
go-version: 1.17
- name: checkout
uses: actions/checkout@v3
- name: cache
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ runner.os }}-go-
- name: deps
run: go get -v -t -d ./...
- name: test
env:
INTEGRATION_TESTS: yes
run: go test -mod readonly -v ./...
lint:
name: lint
runs-on: ubuntu-latest
steps:
- name: checkout
uses: actions/checkout@v3
- name: lint
uses: golangci/golangci-lint-action@v3.4.0
continue-on-error: true
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.30
# Optional: working directory, useful for monorepos
# working-directory: somedir
# Optional: golangci-lint command line arguments.
# args: --issues-exit-code=0
# Optional: show only new issues if it's a pull request. The default value is `false`.
# only-new-issues: true

View File

@@ -1,78 +0,0 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "codeql"
on:
workflow_run:
workflows: ["prbuild"]
types:
- completed
push:
branches: [ master, v3 ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ master, v3 ]
schedule:
- cron: '34 1 * * 0'
jobs:
analyze:
name: analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: [ 'go' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]
# Learn more:
# https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed
steps:
- name: checkout
uses: actions/checkout@v3
- name: setup
uses: actions/setup-go@v3
with:
go-version: 1.17
# Initializes the CodeQL tools for scanning.
- name: init
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: autobuild
uses: github/codeql-action/autobuild@v2
# Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl
# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
# and modify them (or add more) to build your code if your project
# uses a compiled language
#- run: |
# make bootstrap
# make release
- name: analyze
uses: github/codeql-action/analyze@v2

View File

@@ -1,27 +0,0 @@
name: "dependabot-automerge"
on:
pull_request_target:
types: [assigned, opened, synchronize, reopened]
permissions:
pull-requests: write
contents: write
jobs:
automerge:
runs-on: ubuntu-latest
if: github.actor == 'dependabot[bot]'
steps:
- name: metadata
id: metadata
uses: dependabot/fetch-metadata@v1.3.6
with:
github-token: "${{ secrets.TOKEN }}"
- name: merge
id: merge
if: ${{contains(steps.metadata.outputs.dependency-names, 'go.unistack.org')}}
run: gh pr merge --auto --merge "$PR_URL"
env:
PR_URL: ${{github.event.pull_request.html_url}}
GITHUB_TOKEN: ${{secrets.TOKEN}}

53
.github/workflows/job_coverage.yml vendored Normal file
View File

@@ -0,0 +1,53 @@
name: coverage
on:
push:
branches: [ main, v3, v4 ]
paths-ignore:
- '.github/**'
- '.gitea/**'
pull_request:
branches: [ main, v3, v4 ]
jobs:
build:
if: github.server_url != 'https://github.com'
runs-on: ubuntu-latest
steps:
- name: checkout code
uses: actions/checkout@v4
with:
filter: 'blob:none'
- name: setup go
uses: actions/setup-go@v5
with:
cache-dependency-path: "**/*.sum"
go-version: 'stable'
- name: test coverage
run: |
go test -v -cover ./... -covermode=count -coverprofile coverage.out -coverpkg ./...
go tool cover -func coverage.out -o coverage.out
- name: coverage badge
uses: tj-actions/coverage-badge-go@v2
with:
green: 80
filename: coverage.out
- uses: stefanzweifel/git-auto-commit-action@v4
name: autocommit
with:
commit_message: Apply Code Coverage Badge
skip_fetch: false
skip_checkout: false
file_pattern: ./README.md
- name: push
if: steps.auto-commit-action.outputs.changes_detected == 'true'
uses: ad-m/github-push-action@master
with:
github_token: ${{ github.token }}
branch: ${{ github.ref }}

29
.github/workflows/job_lint.yml vendored Normal file
View File

@@ -0,0 +1,29 @@
name: lint
on:
pull_request:
types: [opened, reopened, synchronize]
branches: [ master, v3, v4 ]
paths-ignore:
- '.github/**'
- '.gitea/**'
jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: checkout code
uses: actions/checkout@v4
with:
filter: 'blob:none'
- name: setup go
uses: actions/setup-go@v5
with:
cache-dependency-path: "**/*.sum"
go-version: 'stable'
- name: setup deps
run: go get -v ./...
- name: run lint
uses: golangci/golangci-lint-action@v6
with:
version: 'latest'

94
.github/workflows/job_sync.yml vendored Normal file
View File

@@ -0,0 +1,94 @@
name: sync
on:
schedule:
- cron: '*/5 * * * *'
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
sync:
if: github.server_url != 'https://github.com'
runs-on: ubuntu-latest
steps:
- name: init
run: |
git config --global user.email "vtolstov <vtolstov@users.noreply.github.com>"
git config --global user.name "github-actions[bot]"
echo "machine git.unistack.org login vtolstov password ${{ secrets.TOKEN_GITEA }}" >> /root/.netrc
echo "machine github.com login vtolstov password ${{ secrets.TOKEN_GITHUB }}" >> /root/.netrc
- name: check master
id: check_master
run: |
src_hash=$(git ls-remote https://github.com/${GITHUB_REPOSITORY} refs/heads/master | cut -f1)
dst_hash=$(git ls-remote ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY} refs/heads/master | cut -f1)
echo "src_hash=$src_hash"
echo "dst_hash=$dst_hash"
if [ "$src_hash" != "$dst_hash" ]; then
echo "sync_needed=true" >> $GITHUB_OUTPUT
else
echo "sync_needed=false" >> $GITHUB_OUTPUT
fi
- name: sync master
if: steps.check_master.outputs.sync_needed == 'true'
run: |
git clone --filter=blob:none --filter=tree:0 --branch master --single-branch ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY} repo
cd repo
git remote add --no-tags --fetch --track master upstream https://github.com/${GITHUB_REPOSITORY}
git pull --rebase upstream master
git push upstream master --progress
git push origin master --progress
cd ../
rm -rf repo
- name: check v3
id: check_v3
run: |
src_hash=$(git ls-remote https://github.com/${GITHUB_REPOSITORY} refs/heads/v3 | cut -f1)
dst_hash=$(git ls-remote ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY} refs/heads/v3 | cut -f1)
echo "src_hash=$src_hash"
echo "dst_hash=$dst_hash"
if [ "$src_hash" != "$dst_hash" ]; then
echo "sync_needed=true" >> $GITHUB_OUTPUT
else
echo "sync_needed=false" >> $GITHUB_OUTPUT
fi
- name: sync v3
if: steps.check_v3.outputs.sync_needed == 'true'
run: |
git clone --filter=blob:none --filter=tree:0 --branch v3 --single-branch ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY} repo
cd repo
git remote add --no-tags --fetch --track v3 upstream https://github.com/${GITHUB_REPOSITORY}
git pull --rebase upstream v3
git push upstream v3 --progress
git push origin v3 --progress
cd ../
rm -rf repo
- name: check v4
id: check_v4
run: |
src_hash=$(git ls-remote https://github.com/${GITHUB_REPOSITORY} refs/heads/v4 | cut -f1)
dst_hash=$(git ls-remote ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY} refs/heads/v4 | cut -f1)
echo "src_hash=$src_hash"
echo "dst_hash=$dst_hash"
if [ "$src_hash" != "$dst_hash" ]; then
echo "sync_needed=true" >> $GITHUB_OUTPUT
else
echo "sync_needed=false" >> $GITHUB_OUTPUT
fi
- name: sync v4
if: steps.check_v4.outputs.sync_needed == 'true'
run: |
git clone --filter=blob:none --filter=tree:0 --branch v4 --single-branch ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY} repo
cd repo
git remote add --no-tags --fetch --track v4 upstream https://github.com/${GITHUB_REPOSITORY}
git pull --rebase upstream v4
git push upstream v4 --progress
git push origin v4 --progress
cd ../
rm -rf repo

31
.github/workflows/job_test.yml vendored Normal file
View File

@@ -0,0 +1,31 @@
name: test
on:
pull_request:
types: [opened, reopened, synchronize]
branches: [ master, v3, v4 ]
push:
branches: [ master, v3, v4 ]
paths-ignore:
- '.github/**'
- '.gitea/**'
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: checkout code
uses: actions/checkout@v4
with:
filter: 'blob:none'
- name: setup go
uses: actions/setup-go@v5
with:
cache-dependency-path: "**/*.sum"
go-version: 'stable'
- name: setup deps
run: go get -v ./...
- name: run test
env:
INTEGRATION_TESTS: yes
run: go test -mod readonly -v ./...

50
.github/workflows/job_tests.yml vendored Normal file
View File

@@ -0,0 +1,50 @@
name: test
on:
pull_request:
types: [opened, reopened, synchronize]
branches: [ master, v3, v4 ]
push:
branches: [ master, v3, v4 ]
paths-ignore:
- '.github/**'
- '.gitea/**'
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: checkout code
uses: actions/checkout@v4
with:
filter: 'blob:none'
- name: checkout tests
uses: actions/checkout@v4
with:
ref: master
filter: 'blob:none'
repository: unistack-org/micro-tests
path: micro-tests
- name: setup go
uses: actions/setup-go@v5
with:
cache-dependency-path: "**/*.sum"
go-version: 'stable'
- name: setup go work
env:
GOWORK: ${{ github.workspace }}/go.work
run: |
go work init
go work use .
go work use micro-tests
- name: setup deps
env:
GOWORK: ${{ github.workspace }}/go.work
run: go get -v ./...
- name: run tests
env:
INTEGRATION_TESTS: yes
GOWORK: ${{ github.workspace }}/go.work
run: |
cd micro-tests
go test -mod readonly -v ./... || true

View File

@@ -1,47 +0,0 @@
name: prbuild
on:
pull_request:
branches:
- master
- v3
jobs:
test:
name: test
runs-on: ubuntu-latest
steps:
- name: setup
uses: actions/setup-go@v3
with:
go-version: 1.17
- name: checkout
uses: actions/checkout@v3
- name: cache
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ runner.os }}-go-
- name: deps
run: go get -v -t -d ./...
- name: test
env:
INTEGRATION_TESTS: yes
run: go test -mod readonly -v ./...
lint:
name: lint
runs-on: ubuntu-latest
steps:
- name: checkout
uses: actions/checkout@v3
- name: lint
uses: golangci/golangci-lint-action@v3.4.0
continue-on-error: true
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.30
# Optional: working directory, useful for monorepos
# working-directory: somedir
# Optional: golangci-lint command line arguments.
# args: --issues-exit-code=0
# Optional: show only new issues if it's a pull request. The default value is `false`.
# only-new-issues: true

5
.golangci.yml Normal file
View File

@@ -0,0 +1,5 @@
run:
concurrency: 8
timeout: 5m
issues-exit-code: 1
tests: true

View File

@@ -1,2 +1,2 @@
# broker-kgo
# micro-broker-kgo
![Coverage](https://img.shields.io/badge/Coverage-66.9%25-yellow)

View File

@@ -1,8 +1,12 @@
package kgo
import (
"net/http"
"slices"
"strings"
"github.com/twmb/franz-go/pkg/kgo"
"go.unistack.org/micro/v3/metadata"
"go.unistack.org/micro/v4/metadata"
)
// RecordCarrier injects and extracts traces from a kgo.Record.
@@ -53,24 +57,37 @@ func (c RecordCarrier) Keys() []string {
return out
}
func setHeaders(r *kgo.Record, md metadata.Metadata) {
func setHeaders(r *kgo.Record, md metadata.Metadata, exclude ...string) {
seen := make(map[string]struct{})
loop:
for k, v := range md {
for i := 0; i < len(r.Headers); i++ {
if r.Headers[i].Key == k {
// Key exist, update the value.
r.Headers[i].Value = []byte(v)
continue loop
} else if _, ok := seen[k]; ok {
k = http.CanonicalHeaderKey(k)
if _, ok := seen[k]; ok {
continue loop
}
if slices.ContainsFunc(exclude, func(s string) bool {
return strings.EqualFold(s, k)
}) {
continue loop
}
for i := 0; i < len(r.Headers); i++ {
if strings.EqualFold(r.Headers[i].Key, k) {
// Key exist, update the value.
r.Headers[i].Value = []byte(strings.Join(v, ","))
continue loop
}
}
// Key does not exist, append new header.
r.Headers = append(r.Headers, kgo.RecordHeader{
Key: k,
Value: []byte(v),
Value: []byte(strings.Join(v, ",")),
})
seen[k] = struct{}{}
}
}
}

10
errors.go Normal file
View File

@@ -0,0 +1,10 @@
package kgo
import (
"context"
"errors"
)
func isContextError(err error) bool {
return errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)
}

View File

@@ -1,48 +0,0 @@
package kgo
import (
"context"
"sync"
"go.unistack.org/micro/v3/broker"
)
type event struct {
ctx context.Context
topic string
err error
sync.RWMutex
msg *broker.Message
ack bool
}
func (p *event) Context() context.Context {
return p.ctx
}
func (p *event) Topic() string {
return p.topic
}
func (p *event) Message() *broker.Message {
return p.msg
}
func (p *event) Ack() error {
p.ack = true
return nil
}
func (p *event) Error() error {
return p.err
}
func (p *event) SetError(err error) {
p.err = err
}
var eventPool = sync.Pool{
New: func() interface{} {
return &event{msg: &broker.Message{}}
},
}

38
go.mod
View File

@@ -1,25 +1,27 @@
module go.unistack.org/micro-broker-kgo/v3
module go.unistack.org/micro-broker-kgo/v4
go 1.22.7
toolchain go1.23.3
go 1.23.8
require (
github.com/google/uuid v1.6.0
github.com/twmb/franz-go v1.18.0
github.com/twmb/franz-go/pkg/kadm v1.14.0
github.com/twmb/franz-go/pkg/kmsg v1.9.0
go.opentelemetry.io/otel v1.32.0
go.unistack.org/micro/v3 v3.11.0
github.com/stretchr/testify v1.10.0
github.com/twmb/franz-go v1.19.5
github.com/twmb/franz-go/pkg/kadm v1.16.0
github.com/twmb/franz-go/pkg/kfake v0.0.0-20250508175730-72e1646135e3
github.com/twmb/franz-go/pkg/kmsg v1.11.2
go.opentelemetry.io/otel v1.36.0
go.unistack.org/micro/v4 v4.1.17
)
require (
github.com/klauspost/compress v1.17.11 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
go.unistack.org/micro-proto/v3 v3.4.1 // indirect
golang.org/x/crypto v0.29.0 // indirect
golang.org/x/sys v0.27.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241202173237-19429a94021a // indirect
google.golang.org/grpc v1.68.0 // indirect
google.golang.org/protobuf v1.35.2 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/matoous/go-nanoid v1.5.1 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/spf13/cast v1.9.2 // indirect
go.unistack.org/micro-proto/v4 v4.1.0 // indirect
golang.org/x/crypto v0.39.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

73
go.sum
View File

@@ -1,28 +1,49 @@
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/twmb/franz-go v1.18.0 h1:25FjMZfdozBywVX+5xrWC2W+W76i0xykKjTdEeD2ejw=
github.com/twmb/franz-go v1.18.0/go.mod h1:zXCGy74M0p5FbXsLeASdyvfLFsBvTubVqctIaa5wQ+I=
github.com/twmb/franz-go/pkg/kadm v1.14.0 h1:nAn1co1lXzJQocpzyIyOFOjUBf4WHWs5/fTprXy2IZs=
github.com/twmb/franz-go/pkg/kadm v1.14.0/go.mod h1:XjOPz6ZaXXjrW2jVCfLuucP8H1w2TvD6y3PT2M+aAM4=
github.com/twmb/franz-go/pkg/kmsg v1.9.0 h1:JojYUph2TKAau6SBtErXpXGC7E3gg4vGZMv9xFU/B6M=
github.com/twmb/franz-go/pkg/kmsg v1.9.0/go.mod h1:CMbfazviCyY6HM0SXuG5t9vOwYDHRCSrJJyBAe5paqg=
go.opentelemetry.io/otel v1.32.0 h1:WnBN+Xjcteh0zdk01SVqV55d/m62NJLJdIyb4y/WO5U=
go.opentelemetry.io/otel v1.32.0/go.mod h1:00DCVSB0RQcnzlwyTfqtxSm+DRr9hpYrHjNGiBHVQIg=
go.unistack.org/micro-proto/v3 v3.4.1 h1:UTjLSRz2YZuaHk9iSlVqqsA50JQNAEK2ZFboGqtEa9Q=
go.unistack.org/micro-proto/v3 v3.4.1/go.mod h1:okx/cnOhzuCX0ggl/vToatbCupi0O44diiiLLsZ93Zo=
go.unistack.org/micro/v3 v3.11.0 h1:usQ+8wQuOWpQd4+DGhFXSgZ+e+wOBjuT3W5GJZ02bSs=
go.unistack.org/micro/v3 v3.11.0/go.mod h1:YzMldzHN9Ei+zy5t/Psu7RUWDZwUfrNYiStSQtTz90g=
golang.org/x/crypto v0.29.0 h1:L5SG1JTTXupVV3n6sUqMTeWbjAyfPwoda2DLX8J8FrQ=
golang.org/x/crypto v0.29.0/go.mod h1:+F4F4N5hv6v38hfeYwTdx20oUvLLc+QfrE9Ax9HtgRg=
golang.org/x/sys v0.27.0 h1:wBqf8DvsY9Y/2P8gAfPDEYNuS30J4lPHJxXSb/nJZ+s=
golang.org/x/sys v0.27.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
google.golang.org/genproto/googleapis/rpc v0.0.0-20241202173237-19429a94021a h1:hgh8P4EuoxpsuKMXX/To36nOFD7vixReXgn8lPGnt+o=
google.golang.org/genproto/googleapis/rpc v0.0.0-20241202173237-19429a94021a/go.mod h1:5uTbfoYQed2U9p3KIj2/Zzm02PYhndfdmML0qC3q3FU=
google.golang.org/grpc v1.68.0 h1:aHQeeJbo8zAkAa3pRzrVjZlbz6uSfeOXlJNQM0RAbz0=
google.golang.org/grpc v1.68.0/go.mod h1:fmSPC5AsjSBCK54MyHRx48kpOti1/jRfOlwEWywNjWA=
google.golang.org/protobuf v1.35.2 h1:8Ar7bF+apOIoThw1EdZl0p1oWvMqTHmpA2fRTyZO8io=
google.golang.org/protobuf v1.35.2/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/matoous/go-nanoid v1.5.1 h1:aCjdvTyO9LLnTIi0fgdXhOPPvOHjpXN6Ik9DaNjIct4=
github.com/matoous/go-nanoid v1.5.1/go.mod h1:zyD2a71IubI24efhpvkJz+ZwfwagzgSO6UNiFsZKN7U=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
github.com/spf13/cast v1.9.2 h1:SsGfm7M8QOFtEzumm7UZrZdLLquNdzFYfIbEXntcFbE=
github.com/spf13/cast v1.9.2/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/twmb/franz-go v1.19.5 h1:W7+o8D0RsQsedqib71OVlLeZ0zI6CbFra7yTYhZTs5Y=
github.com/twmb/franz-go v1.19.5/go.mod h1:4kFJ5tmbbl7asgwAGVuyG1ZMx0NNpYk7EqflvWfPCpM=
github.com/twmb/franz-go/pkg/kadm v1.16.0 h1:STMs1t5lYR5mR974PSiwNzE5TvsosByTp+rKXLOhAjE=
github.com/twmb/franz-go/pkg/kadm v1.16.0/go.mod h1:MUdcUtnf9ph4SFBLLA/XxE29rvLhWYLM9Ygb8dfSCvw=
github.com/twmb/franz-go/pkg/kfake v0.0.0-20250508175730-72e1646135e3 h1:p24opKWPySAy8xSl8NqRgOv7Q+bX7kdrQirBVRJzQfo=
github.com/twmb/franz-go/pkg/kfake v0.0.0-20250508175730-72e1646135e3/go.mod h1:7uQs3Ae6HkWT1Y9elMbqtAcNFCI0y6+iS+Phw49L49U=
github.com/twmb/franz-go/pkg/kmsg v1.11.2 h1:hIw75FpwcAjgeyfIGFqivAvwC5uNIOWRGvQgZhH4mhg=
github.com/twmb/franz-go/pkg/kmsg v1.11.2/go.mod h1:CFfkkLysDNmukPYhGzuUcDtf46gQSqCZHMW1T4Z+wDE=
go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg=
go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E=
go.unistack.org/micro-proto/v4 v4.1.0 h1:qPwL2n/oqh9RE3RTTDgt28XK3QzV597VugQPaw9lKUk=
go.unistack.org/micro-proto/v4 v4.1.0/go.mod h1:ArmK7o+uFvxSY3dbJhKBBX4Pm1rhWdLEFf3LxBrMtec=
go.unistack.org/micro/v4 v4.1.17 h1:26QDtRSYVpozYuassyvLP4sEQRo3dxgD3sVILRXmIPo=
go.unistack.org/micro/v4 v4.1.17/go.mod h1:xleO2M5Yxh4s6I+RUcLrEpUjobefh+71ctrdIfn7TUs=
golang.org/x/crypto v0.39.0 h1:SHs+kF4LP+f+p14esP5jAoDpHU8Gu/v9lFRK6IT5imM=
golang.org/x/crypto v0.39.0/go.mod h1:L+Xg3Wf6HoL4Bn4238Z6ft6KfEpN0tJGo53AAPC632U=
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

108
hook_event.go Normal file
View File

@@ -0,0 +1,108 @@
package kgo
import (
"context"
"net"
"sync/atomic"
"time"
"github.com/twmb/franz-go/pkg/kgo"
"go.unistack.org/micro/v4/logger"
)
type hookEvent struct {
log logger.Logger
fatalOnError bool
connected *atomic.Uint32
}
var (
_ kgo.HookBrokerConnect = &hookEvent{}
_ kgo.HookBrokerDisconnect = &hookEvent{}
_ kgo.HookBrokerRead = &hookEvent{}
_ kgo.HookBrokerWrite = &hookEvent{}
_ kgo.HookGroupManageError = &hookEvent{}
_ kgo.HookProduceRecordUnbuffered = &hookEvent{}
)
func (m *hookEvent) OnGroupManageError(err error) {
switch {
case err == nil || isContextError(err) || kgo.IsRetryableBrokerErr(err):
return
default:
ctx := context.TODO()
logMsg := "kgo.OnGroupManageError"
if m.fatalOnError {
m.log.Fatal(ctx, logMsg, err)
} else {
m.log.Error(ctx, logMsg, err)
}
}
}
func (m *hookEvent) OnBrokerConnect(_ kgo.BrokerMetadata, _ time.Duration, _ net.Conn, err error) {
switch {
case err == nil || isContextError(err) || kgo.IsRetryableBrokerErr(err):
return
default:
ctx := context.TODO()
logMsg := "kgo.OnBrokerConnect"
if m.fatalOnError {
m.log.Fatal(ctx, logMsg, err)
} else {
m.log.Error(ctx, logMsg, err)
}
}
}
func (m *hookEvent) OnBrokerDisconnect(_ kgo.BrokerMetadata, _ net.Conn) {}
func (m *hookEvent) OnBrokerWrite(_ kgo.BrokerMetadata, _ int16, _ int, _ time.Duration, _ time.Duration, err error) {
switch {
case err == nil || isContextError(err) || kgo.IsRetryableBrokerErr(err):
return
default:
ctx := context.TODO()
logMsg := "kgo.OnBrokerWrite"
if m.fatalOnError {
m.log.Fatal(ctx, logMsg, err)
} else {
m.log.Error(ctx, logMsg, err)
}
}
}
func (m *hookEvent) OnBrokerRead(_ kgo.BrokerMetadata, _ int16, _ int, _ time.Duration, _ time.Duration, err error) {
switch {
case err == nil || isContextError(err) || kgo.IsRetryableBrokerErr(err):
return
default:
ctx := context.TODO()
logMsg := "kgo.OnBrokerRead"
if m.fatalOnError {
m.log.Fatal(ctx, logMsg, err)
} else {
m.log.Error(ctx, logMsg, err)
}
}
}
func (m *hookEvent) OnProduceRecordUnbuffered(_ *kgo.Record, err error) {
switch {
case err == nil || isContextError(err) || kgo.IsRetryableBrokerErr(err):
return
default:
ctx := context.TODO()
logMsg := "kgo.OnProduceRecordUnbuffered"
if m.fatalOnError {
m.log.Fatal(ctx, logMsg, err)
} else {
m.log.Error(ctx, logMsg, err)
}
}
}

471
hook_event_test.go Normal file
View File

@@ -0,0 +1,471 @@
package kgo
import (
"context"
"errors"
"io"
"net"
"os"
"testing"
"github.com/stretchr/testify/require"
"github.com/twmb/franz-go/pkg/kgo"
"go.unistack.org/micro/v4/logger"
)
func TestHookEvent_OnGroupManageError(t *testing.T) {
tests := []struct {
name string
inputErr error
fatalOnError bool
expectedErrorIsCalled bool
expectedErrorMsg string
expectedFatalIsCalled bool
expectedFatalMsg string
}{
{
name: "error is nil",
inputErr: nil,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context canceled",
inputErr: context.Canceled,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context deadline exceeded",
inputErr: context.DeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: deadline exceeded (os package)",
inputErr: os.ErrDeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: EOF (io package)",
inputErr: io.EOF,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: closed network connection (net package)",
inputErr: net.ErrClosed,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "some error (non-fatal)",
inputErr: errors.New("some error"),
fatalOnError: false,
expectedErrorIsCalled: true,
expectedErrorMsg: "kgo.OnGroupManageError",
expectedFatalIsCalled: false,
},
{
name: "some error (fatal)",
inputErr: errors.New("some error"),
fatalOnError: true,
expectedErrorIsCalled: false,
expectedFatalIsCalled: true,
expectedFatalMsg: "kgo.OnGroupManageError",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
log := &mockLogger{}
he := &hookEvent{log: log, fatalOnError: tt.fatalOnError}
he.OnGroupManageError(tt.inputErr)
require.Equal(t, tt.expectedErrorIsCalled, log.errorIsCalled)
require.Equal(t, tt.expectedErrorMsg, log.errorMsg)
require.Equal(t, tt.expectedFatalIsCalled, log.fatalIsCalled)
require.Equal(t, tt.expectedFatalMsg, log.fatalMsg)
})
}
}
func TestHookEvent_OnBrokerConnect(t *testing.T) {
tests := []struct {
name string
inputErr error
fatalOnError bool
expectedErrorIsCalled bool
expectedErrorMsg string
expectedFatalIsCalled bool
expectedFatalMsg string
}{
{
name: "error is nil",
inputErr: nil,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context canceled",
inputErr: context.Canceled,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context deadline exceeded",
inputErr: context.DeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: deadline exceeded (os package)",
inputErr: os.ErrDeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: EOF (io package)",
inputErr: io.EOF,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: closed network connection (net package)",
inputErr: net.ErrClosed,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "some error (non-fatal)",
inputErr: errors.New("some error"),
fatalOnError: false,
expectedErrorIsCalled: true,
expectedErrorMsg: "kgo.OnBrokerConnect",
expectedFatalIsCalled: false,
},
{
name: "some error (fatal)",
inputErr: errors.New("some error"),
fatalOnError: true,
expectedErrorIsCalled: false,
expectedFatalIsCalled: true,
expectedFatalMsg: "kgo.OnBrokerConnect",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
log := &mockLogger{}
he := &hookEvent{log: log, fatalOnError: tt.fatalOnError}
he.OnBrokerConnect(kgo.BrokerMetadata{}, 0, nil, tt.inputErr)
require.Equal(t, tt.expectedErrorIsCalled, log.errorIsCalled)
require.Equal(t, tt.expectedErrorMsg, log.errorMsg)
require.Equal(t, tt.expectedFatalIsCalled, log.fatalIsCalled)
require.Equal(t, tt.expectedFatalMsg, log.fatalMsg)
})
}
}
func TestHookEvent_OnBrokerWrite(t *testing.T) {
tests := []struct {
name string
inputErr error
fatalOnError bool
expectedErrorIsCalled bool
expectedErrorMsg string
expectedFatalIsCalled bool
expectedFatalMsg string
}{
{
name: "error is nil",
inputErr: nil,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context canceled",
inputErr: context.Canceled,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context deadline exceeded",
inputErr: context.DeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: deadline exceeded (os package)",
inputErr: os.ErrDeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: EOF (io package)",
inputErr: io.EOF,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: closed network connection (net package)",
inputErr: net.ErrClosed,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "some error (non-fatal)",
inputErr: errors.New("some error"),
fatalOnError: false,
expectedErrorIsCalled: true,
expectedErrorMsg: "kgo.OnBrokerWrite",
expectedFatalIsCalled: false,
},
{
name: "some error (fatal)",
inputErr: errors.New("some error"),
fatalOnError: true,
expectedErrorIsCalled: false,
expectedFatalIsCalled: true,
expectedFatalMsg: "kgo.OnBrokerWrite",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
log := &mockLogger{}
he := &hookEvent{log: log, fatalOnError: tt.fatalOnError}
he.OnBrokerWrite(kgo.BrokerMetadata{}, 0, 0, 0, 0, tt.inputErr)
require.Equal(t, tt.expectedErrorIsCalled, log.errorIsCalled)
require.Equal(t, tt.expectedErrorMsg, log.errorMsg)
require.Equal(t, tt.expectedFatalIsCalled, log.fatalIsCalled)
require.Equal(t, tt.expectedFatalMsg, log.fatalMsg)
})
}
}
func TestHookEvent_OnBrokerRead(t *testing.T) {
tests := []struct {
name string
inputErr error
fatalOnError bool
expectedErrorIsCalled bool
expectedErrorMsg string
expectedFatalIsCalled bool
expectedFatalMsg string
}{
{
name: "error is nil",
inputErr: nil,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context canceled",
inputErr: context.Canceled,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context deadline exceeded",
inputErr: context.DeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: deadline exceeded (os package)",
inputErr: os.ErrDeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: EOF (io package)",
inputErr: io.EOF,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: closed network connection (net package)",
inputErr: net.ErrClosed,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "some error (non-fatal)",
inputErr: errors.New("some error"),
fatalOnError: false,
expectedErrorIsCalled: true,
expectedErrorMsg: "kgo.OnBrokerRead",
expectedFatalIsCalled: false,
},
{
name: "some error (fatal)",
inputErr: errors.New("some error"),
fatalOnError: true,
expectedErrorIsCalled: false,
expectedFatalIsCalled: true,
expectedFatalMsg: "kgo.OnBrokerRead",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
log := &mockLogger{}
he := &hookEvent{log: log, fatalOnError: tt.fatalOnError}
he.OnBrokerRead(kgo.BrokerMetadata{}, 0, 0, 0, 0, tt.inputErr)
require.Equal(t, tt.expectedErrorIsCalled, log.errorIsCalled)
require.Equal(t, tt.expectedErrorMsg, log.errorMsg)
require.Equal(t, tt.expectedFatalIsCalled, log.fatalIsCalled)
require.Equal(t, tt.expectedFatalMsg, log.fatalMsg)
})
}
}
func TestHookEvent_OnProduceRecordUnbuffered(t *testing.T) {
tests := []struct {
name string
inputErr error
fatalOnError bool
expectedErrorIsCalled bool
expectedErrorMsg string
expectedFatalIsCalled bool
expectedFatalMsg string
}{
{
name: "error is nil",
inputErr: nil,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context canceled",
inputErr: context.Canceled,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "context deadline exceeded",
inputErr: context.DeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: deadline exceeded (os package)",
inputErr: os.ErrDeadlineExceeded,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: EOF (io package)",
inputErr: io.EOF,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "retryable error: closed network connection (net package)",
inputErr: net.ErrClosed,
expectedErrorIsCalled: false,
expectedFatalIsCalled: false,
},
{
name: "some error (non-fatal)",
inputErr: errors.New("some error"),
fatalOnError: false,
expectedErrorIsCalled: true,
expectedErrorMsg: "kgo.OnProduceRecordUnbuffered",
expectedFatalIsCalled: false,
},
{
name: "some error (fatal)",
inputErr: errors.New("some error"),
fatalOnError: true,
expectedErrorIsCalled: false,
expectedFatalIsCalled: true,
expectedFatalMsg: "kgo.OnProduceRecordUnbuffered",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
log := &mockLogger{}
he := &hookEvent{log: log, fatalOnError: tt.fatalOnError}
he.OnProduceRecordUnbuffered(&kgo.Record{}, tt.inputErr)
require.Equal(t, tt.expectedErrorIsCalled, log.errorIsCalled)
require.Equal(t, tt.expectedErrorMsg, log.errorMsg)
require.Equal(t, tt.expectedFatalIsCalled, log.fatalIsCalled)
require.Equal(t, tt.expectedFatalMsg, log.fatalMsg)
})
}
}
// Mocks
type mockLogger struct {
errorIsCalled bool
errorMsg string
fatalIsCalled bool
fatalMsg string
}
func (m *mockLogger) Init(...logger.Option) error {
panic("implement me")
}
func (m *mockLogger) Clone(...logger.Option) logger.Logger {
panic("implement me")
}
func (m *mockLogger) V(logger.Level) bool {
panic("implement me")
}
func (m *mockLogger) Level(logger.Level) {
panic("implement me")
}
func (m *mockLogger) Options() logger.Options {
panic("implement me")
}
func (m *mockLogger) Fields(...interface{}) logger.Logger {
panic("implement me")
}
func (m *mockLogger) Info(context.Context, string, ...interface{}) {
panic("implement me")
}
func (m *mockLogger) Trace(context.Context, string, ...interface{}) {
panic("implement me")
}
func (m *mockLogger) Debug(context.Context, string, ...interface{}) {
panic("implement me")
}
func (m *mockLogger) Warn(context.Context, string, ...interface{}) {
panic("implement me")
}
func (m *mockLogger) Error(ctx context.Context, msg string, args ...interface{}) {
m.errorIsCalled = true
m.errorMsg = msg
}
func (m *mockLogger) Fatal(ctx context.Context, msg string, args ...interface{}) {
m.fatalIsCalled = true
m.fatalMsg = msg
}
func (m *mockLogger) Log(context.Context, logger.Level, string, ...interface{}) {
panic("implement me")
}
func (m *mockLogger) Name() string {
panic("implement me")
}
func (m *mockLogger) String() string {
panic("implement me")
}

View File

@@ -1,65 +0,0 @@
//go:build ignore
package main
import (
"context"
"fmt"
"os"
"time"
"github.com/twmb/franz-go/pkg/kadm"
"github.com/twmb/franz-go/pkg/kgo"
"github.com/twmb/franz-go/pkg/kversion"
//"github.com/twmb/franz-go/pkg/sasl/scram"
"github.com/twmb/franz-go/pkg/sasl/plain"
)
func die(msg string, args ...any) {
fmt.Fprintf(os.Stderr, msg, args...)
os.Exit(1)
}
func main() {
seeds := []string{"vm-kafka-ump01tn.mbrd.ru:9092", "vm-kafka-ump02tn.mbrd.ru:9092", "vm-kafka-ump03tn.mbrd.ru:9092"}
pass := "XXXXX"
user := "XXXXX"
var adminClient *kadm.Client
{
client, err := kgo.NewClient(
kgo.SeedBrokers(seeds...),
// kgo.SASL((scram.Auth{User: user, Pass: pass}).AsSha512Mechanism()),
kgo.SASL((plain.Auth{User: user, Pass: pass}).AsMechanism()),
// Do not try to send requests newer than 2.4.0 to avoid breaking changes in the request struct.
// Sometimes there are breaking changes for newer versions where more properties are required to set.
kgo.MaxVersions(kversion.V2_4_0()),
)
if err != nil {
panic(err)
}
defer client.Close()
adminClient = kadm.NewClient(client)
}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
dg, err := adminClient.DescribeGroups(ctx, "interestrate_loader")
if err != nil {
die("failed to describe group: %v", err)
}
for _, m := range dg["interestrate_loader"].Members {
mc, _ := m.Assigned.AsConsumer()
for _, mt := range mc.Topics {
for _, p := range mt.Partitions {
fmt.Printf("client:%s\tpartitions: %d\n", m.ClientID, p)
}
}
}
}

315
kgo.go
View File

@@ -5,31 +5,37 @@ import (
"context"
"errors"
"fmt"
"math/rand"
"net/http"
"math/rand/v2"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/google/uuid"
"github.com/twmb/franz-go/pkg/kgo"
"github.com/twmb/franz-go/pkg/kmsg"
"go.unistack.org/micro/v3/broker"
"go.unistack.org/micro/v3/logger"
"go.unistack.org/micro/v3/metadata"
"go.unistack.org/micro/v3/semconv"
"go.unistack.org/micro/v3/tracer"
mrand "go.unistack.org/micro/v3/util/rand"
"go.unistack.org/micro/v4/broker"
"go.unistack.org/micro/v4/codec"
"go.unistack.org/micro/v4/logger"
"go.unistack.org/micro/v4/metadata"
"go.unistack.org/micro/v4/options"
"go.unistack.org/micro/v4/semconv"
"go.unistack.org/micro/v4/tracer"
"go.unistack.org/micro/v4/util/id"
mrand "go.unistack.org/micro/v4/util/rand"
)
var _ broker.Broker = (*Broker)(nil)
var messagePool = sync.Pool{
New: func() interface{} {
return &kgoMessage{}
},
}
var ErrLostMessage = errors.New("message not marked for offsets commit and will be lost in next iteration")
var DefaultRetryBackoffFn = func() func(int) time.Duration {
var rngMu sync.Mutex
rng := rand.New(rand.NewSource(time.Now().UnixNano()))
return func(fails int) time.Duration {
const (
min = 100 * time.Millisecond
@@ -45,7 +51,7 @@ var DefaultRetryBackoffFn = func() func(int) time.Duration {
backoff := min * time.Duration(1<<(fails-1))
rngMu.Lock()
jitter := 0.8 + 0.4*rng.Float64()
jitter := 0.8 + 0.4*rand.Float64()
rngMu.Unlock()
backoff = time.Duration(float64(backoff) * jitter)
@@ -58,13 +64,18 @@ var DefaultRetryBackoffFn = func() func(int) time.Duration {
}()
type Broker struct {
init bool
funcPublish broker.FuncPublish
funcSubscribe broker.FuncSubscribe
c *kgo.Client
kopts []kgo.Opt
connected *atomic.Uint32
sync.RWMutex
opts broker.Options
kopts []kgo.Opt
subs []*Subscriber
opts broker.Options
mu sync.RWMutex
init bool
}
func (r *Broker) Live() bool {
@@ -91,6 +102,74 @@ func (k *Broker) Client() *kgo.Client {
return k.c
}
type kgoMessage struct {
c codec.Codec
topic string
ctx context.Context
body []byte
hdr metadata.Metadata
opts broker.MessageOptions
ack bool
}
func (m *kgoMessage) Ack() error {
m.ack = true
return nil
}
func (m *kgoMessage) Body() []byte {
return m.body
}
func (m *kgoMessage) Header() metadata.Metadata {
return m.hdr
}
func (m *kgoMessage) Context() context.Context {
return m.ctx
}
func (m *kgoMessage) Topic() string {
return ""
}
func (m *kgoMessage) Unmarshal(dst interface{}, opts ...codec.Option) error {
return m.c.Unmarshal(m.body, dst)
}
func (b *Broker) newCodec(ct string) (codec.Codec, error) {
if idx := strings.IndexRune(ct, ';'); idx >= 0 {
ct = ct[:idx]
}
b.mu.RLock()
c, ok := b.opts.Codecs[ct]
b.mu.RUnlock()
if ok {
return c, nil
}
return nil, codec.ErrUnknownContentType
}
func (b *Broker) NewMessage(ctx context.Context, hdr metadata.Metadata, body interface{}, opts ...broker.MessageOption) (broker.Message, error) {
options := broker.NewMessageOptions(opts...)
if options.ContentType == "" {
options.ContentType = b.opts.ContentType
}
m := &kgoMessage{ctx: ctx, hdr: hdr.Copy(), opts: options}
c, err := b.newCodec(m.opts.ContentType)
if err == nil {
m.body, err = c.Marshal(body)
}
if err != nil {
return nil, err
}
m.hdr.Set(metadata.HeaderContentType, m.opts.ContentType)
return m, nil
}
func (k *Broker) connect(ctx context.Context, opts ...kgo.Opt) (*kgo.Client, *hookTracer, error) {
var c *kgo.Client
var err error
@@ -108,10 +187,18 @@ func (k *Broker) connect(ctx context.Context, opts ...kgo.Opt) (*kgo.Client, *ho
}
}
var fatalOnError bool
if k.opts.Context != nil {
if v, ok := k.opts.Context.Value(fatalOnErrorKey{}).(bool); ok && v {
fatalOnError = v
}
}
htracer := &hookTracer{group: group, clientID: clientID, tracer: k.opts.Tracer}
opts = append(opts,
kgo.WithHooks(&hookMeter{meter: k.opts.Meter}),
kgo.WithHooks(htracer),
kgo.WithHooks(&hookEvent{log: k.opts.Logger, fatalOnError: fatalOnError, connected: k.connected}),
)
select {
@@ -133,8 +220,9 @@ func (k *Broker) connect(ctx context.Context, opts ...kgo.Opt) (*kgo.Client, *ho
}
return nil, nil, err
}
}
k.connected.Store(1)
return c, htracer, nil
}
}
func (k *Broker) Connect(ctx context.Context) error {
@@ -152,10 +240,10 @@ func (k *Broker) Connect(ctx context.Context) error {
return err
}
k.Lock()
k.mu.Lock()
k.c = c
k.connected.Store(1)
k.Unlock()
k.mu.Unlock()
return nil
}
@@ -173,8 +261,8 @@ func (k *Broker) Disconnect(ctx context.Context) error {
ctx, span = k.opts.Tracer.Start(ctx, "Disconnect")
defer span.Finish()
k.Lock()
defer k.Unlock()
k.mu.Lock()
defer k.mu.Unlock()
select {
case <-nctx.Done():
return nctx.Err()
@@ -198,8 +286,8 @@ func (k *Broker) Disconnect(ctx context.Context) error {
}
func (k *Broker) Init(opts ...broker.Option) error {
k.Lock()
defer k.Unlock()
k.mu.Lock()
defer k.mu.Unlock()
if len(opts) == 0 && k.init {
return nil
@@ -228,6 +316,18 @@ func (k *Broker) Init(opts ...broker.Option) error {
}
}
k.funcPublish = k.fnPublish
k.funcSubscribe = k.fnSubscribe
k.opts.Hooks.EachPrev(func(hook options.Hook) {
switch h := hook.(type) {
case broker.HookPublish:
k.funcPublish = h(k.funcPublish)
case broker.HookSubscribe:
k.funcSubscribe = h(k.funcSubscribe)
}
})
k.init = true
return nil
@@ -237,99 +337,93 @@ func (k *Broker) Options() broker.Options {
return k.opts
}
func (k *Broker) BatchPublish(ctx context.Context, msgs []*broker.Message, opts ...broker.PublishOption) error {
return k.publish(ctx, msgs, opts...)
func (b *Broker) Publish(ctx context.Context, topic string, messages ...broker.Message) error {
return b.funcPublish(ctx, topic, messages...)
}
func (k *Broker) Publish(ctx context.Context, topic string, msg *broker.Message, opts ...broker.PublishOption) error {
msg.Header.Set(metadata.HeaderTopic, topic)
return k.publish(ctx, []*broker.Message{msg}, opts...)
func (b *Broker) fnPublish(ctx context.Context, topic string, messages ...broker.Message) error {
return b.publish(ctx, topic, messages...)
}
func (k *Broker) publish(ctx context.Context, msgs []*broker.Message, opts ...broker.PublishOption) error {
k.Lock()
if k.connected.Load() == 0 {
c, _, err := k.connect(ctx, k.kopts...)
if err != nil {
k.Unlock()
return err
}
k.c = c
k.connected.Store(1)
}
k.Unlock()
func (b *Broker) publish(ctx context.Context, topic string, messages ...broker.Message) error {
var records []*kgo.Record
for _, msg := range messages {
rec := &kgo.Record{
Context: msg.Context(),
Topic: topic,
Value: msg.Body(),
}
options := broker.NewPublishOptions(opts...)
records := make([]*kgo.Record, 0, len(msgs))
var errs []string
var err error
var key []byte
var promise func(*kgo.Record, error)
if options.Context != nil {
if k, ok := options.Context.Value(publishKey{}).([]byte); ok && k != nil {
key = k
if rec.Context != nil {
if k, ok := rec.Context.Value(messageKey{}).([]byte); ok && k != nil {
rec.Key = k
}
if p, ok := options.Context.Value(publishPromiseKey{}).(func(*kgo.Record, error)); ok && p != nil {
if p, ok := rec.Context.Value(messagePromiseKey{}).(func(*kgo.Record, error)); ok && p != nil {
promise = p
}
}
for _, msg := range msgs {
rec := &kgo.Record{Context: ctx, Key: key}
rec.Topic, _ = msg.Header.Get(metadata.HeaderTopic)
msg.Header.Del(metadata.HeaderTopic)
k.opts.Meter.Counter(semconv.PublishMessageInflight, "endpoint", rec.Topic, "topic", rec.Topic).Inc()
if options.BodyOnly || k.opts.Codec.String() == "noop" {
rec.Value = msg.Body
for k, v := range msg.Header {
rec.Headers = append(rec.Headers, kgo.RecordHeader{Key: http.CanonicalHeaderKey(k), Value: []byte(v)})
kmsg, ok := msg.(*kgoMessage)
if !ok {
continue
}
} else {
rec.Value, err = k.opts.Codec.Marshal(msg)
if err != nil {
return err
if kmsg.opts.Context != nil {
if k, ok := kmsg.opts.Context.Value(messageKey{}).([]byte); ok && k != nil {
rec.Key = k
}
if p, ok := kmsg.opts.Context.Value(messagePromiseKey{}).(func(*kgo.Record, error)); ok && p != nil {
promise = p
}
}
records = append(records, rec)
}
setHeaders(rec, msg.Header())
if promise != nil {
ts := time.Now()
for _, rec := range records {
k.c.Produce(ctx, rec, func(r *kgo.Record, err error) {
b.opts.Meter.Counter(semconv.PublishMessageInflight, "endpoint", rec.Topic, "topic", rec.Topic).Inc()
b.c.Produce(ctx, rec, func(r *kgo.Record, err error) {
te := time.Since(ts)
k.opts.Meter.Counter(semconv.PublishMessageInflight, "endpoint", rec.Topic, "topic", rec.Topic).Dec()
k.opts.Meter.Summary(semconv.PublishMessageLatencyMicroseconds, "endpoint", rec.Topic, "topic", rec.Topic).Update(te.Seconds())
k.opts.Meter.Histogram(semconv.PublishMessageDurationSeconds, "endpoint", rec.Topic, "topic", rec.Topic).Update(te.Seconds())
b.opts.Meter.Counter(semconv.PublishMessageInflight, "endpoint", rec.Topic, "topic", rec.Topic).Dec()
b.opts.Meter.Summary(semconv.PublishMessageLatencyMicroseconds, "endpoint", rec.Topic, "topic", rec.Topic).Update(te.Seconds())
b.opts.Meter.Histogram(semconv.PublishMessageDurationSeconds, "endpoint", rec.Topic, "topic", rec.Topic).Update(te.Seconds())
if err != nil {
k.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", rec.Topic, "topic", rec.Topic, "status", "failure").Inc()
b.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", rec.Topic, "topic", rec.Topic, "status", "failure").Inc()
} else {
k.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", rec.Topic, "topic", rec.Topic, "status", "success").Inc()
b.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", rec.Topic, "topic", rec.Topic, "status", "success").Inc()
}
promise(r, err)
})
continue
} else {
records = append(records, rec)
}
return nil
}
if len(records) > 0 {
var errs []string
ts := time.Now()
results := k.c.ProduceSync(ctx, records...)
b.opts.Meter.Counter(semconv.PublishMessageInflight, "endpoint", topic, "topic", topic).Set(uint64(len(records)))
results := b.c.ProduceSync(ctx, records...)
te := time.Since(ts)
for _, result := range results {
k.opts.Meter.Summary(semconv.PublishMessageLatencyMicroseconds, "endpoint", result.Record.Topic, "topic", result.Record.Topic).Update(te.Seconds())
k.opts.Meter.Histogram(semconv.PublishMessageDurationSeconds, "endpoint", result.Record.Topic, "topic", result.Record.Topic).Update(te.Seconds())
k.opts.Meter.Counter(semconv.PublishMessageInflight, "endpoint", result.Record.Topic, "topic", result.Record.Topic).Dec()
b.opts.Meter.Summary(semconv.PublishMessageLatencyMicroseconds, "endpoint", result.Record.Topic, "topic", result.Record.Topic).Update(te.Seconds())
b.opts.Meter.Histogram(semconv.PublishMessageDurationSeconds, "endpoint", result.Record.Topic, "topic", result.Record.Topic).Update(te.Seconds())
b.opts.Meter.Counter(semconv.PublishMessageInflight, "endpoint", result.Record.Topic, "topic", result.Record.Topic).Dec()
if result.Err != nil {
k.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", result.Record.Topic, "topic", result.Record.Topic, "status", "failure").Inc()
b.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", result.Record.Topic, "topic", result.Record.Topic, "status", "failure").Inc()
errs = append(errs, result.Err.Error())
} else {
k.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", result.Record.Topic, "topic", result.Record.Topic, "status", "success").Inc()
b.opts.Meter.Counter(semconv.PublishMessageTotal, "endpoint", result.Record.Topic, "topic", result.Record.Topic, "status", "success").Inc()
}
}
if len(errs) > 0 {
return fmt.Errorf("publish error: %s", strings.Join(errs, "\n"))
}
}
return nil
}
@@ -350,38 +444,71 @@ func (k *Broker) TopicExists(ctx context.Context, topic string) error {
return nil
}
func (k *Broker) BatchSubscribe(ctx context.Context, topic string, handler broker.BatchHandler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
return nil, nil
func (b *Broker) Subscribe(ctx context.Context, topic string, handler interface{}, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
return b.funcSubscribe(ctx, topic, handler, opts...)
}
func (k *Broker) Subscribe(ctx context.Context, topic string, handler broker.Handler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
func (b *Broker) fnSubscribe(ctx context.Context, topic string, handler interface{}, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
if err := broker.IsValidHandler(handler); err != nil {
return nil, err
}
options := broker.NewSubscribeOptions(opts...)
switch handler.(type) {
default:
return nil, broker.ErrInvalidHandler
case func(broker.Message) error:
break
case func([]broker.Message) error:
break
}
if options.Group == "" {
uid, err := uuid.NewRandom()
uid, err := id.New()
if err != nil {
return nil, err
}
options.Group = uid.String()
options.Group = uid
}
commitInterval := DefaultCommitInterval
if k.opts.Context != nil {
if v, ok := k.opts.Context.Value(commitIntervalKey{}).(time.Duration); ok && v > 0 {
if b.opts.Context != nil {
if v, ok := b.opts.Context.Value(commitIntervalKey{}).(time.Duration); ok && v > 0 {
commitInterval = v
}
}
var messagePool bool
var fatalOnError bool
if b.opts.Context != nil {
if v, ok := b.opts.Context.Value(fatalOnErrorKey{}).(bool); ok && v {
fatalOnError = v
}
if v, ok := b.opts.Context.Value(subscribeMessagePoolKey{}).(bool); ok && v {
messagePool = v
}
}
if options.Context != nil {
if v, ok := options.Context.Value(fatalOnErrorKey{}).(bool); ok && v {
fatalOnError = v
}
}
sub := &Subscriber{
topic: topic,
opts: options,
handler: handler,
kopts: k.opts,
kopts: b.opts,
consumers: make(map[tp]*consumer),
done: make(chan struct{}),
fatalOnError: fatalOnError,
connected: b.connected,
messagePool: messagePool,
}
kopts := append(k.kopts,
kopts := append(b.kopts,
kgo.ConsumerGroup(options.Group),
kgo.ConsumeTopics(topic),
kgo.ConsumeResetOffset(kgo.NewOffset().AtStart()),
@@ -389,7 +516,9 @@ func (k *Broker) Subscribe(ctx context.Context, topic string, handler broker.Han
kgo.AutoCommitInterval(commitInterval),
kgo.OnPartitionsAssigned(sub.assigned),
kgo.OnPartitionsRevoked(sub.revoked),
kgo.StopProducerOnDataLossDetected(),
kgo.OnPartitionsLost(sub.lost),
kgo.AutoCommitCallback(sub.autocommit),
kgo.AutoCommitMarks(),
)
@@ -399,7 +528,7 @@ func (k *Broker) Subscribe(ctx context.Context, topic string, handler broker.Han
}
}
c, htracer, err := k.connect(ctx, kopts...)
c, htracer, err := b.connect(ctx, kopts...)
if err != nil {
return nil, err
}
@@ -421,9 +550,10 @@ func (k *Broker) Subscribe(ctx context.Context, topic string, handler broker.Han
go sub.poll(ctx)
k.Lock()
k.subs = append(k.subs, sub)
k.Unlock()
b.mu.Lock()
b.subs = append(b.subs, sub)
b.mu.Unlock()
return sub, nil
}
@@ -450,7 +580,7 @@ func NewBroker(opts ...broker.Option) *Broker {
kgo.BlockRebalanceOnPoll(),
kgo.Balancers(kgo.CooperativeStickyBalancer()),
kgo.FetchIsolationLevel(kgo.ReadUncommitted()),
kgo.UnknownTopicRetries(0),
kgo.UnknownTopicRetries(1),
}
if options.Context != nil {
@@ -460,6 +590,7 @@ func NewBroker(opts ...broker.Option) *Broker {
}
return &Broker{
connected: &atomic.Uint32{},
opts: options,
kopts: kopts,
}

View File

@@ -2,39 +2,140 @@ package kgo_test
import (
"context"
"fmt"
"os"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/twmb/franz-go/pkg/kfake"
kg "github.com/twmb/franz-go/pkg/kgo"
kgo "go.unistack.org/micro-broker-kgo/v3"
"go.unistack.org/micro/v3/broker"
"go.unistack.org/micro/v3/logger"
"go.unistack.org/micro/v3/metadata"
kgo "go.unistack.org/micro-broker-kgo/v4"
"go.unistack.org/micro/v4/broker"
"go.unistack.org/micro/v4/codec"
"go.unistack.org/micro/v4/logger"
"go.unistack.org/micro/v4/logger/slog"
"go.unistack.org/micro/v4/metadata"
)
var (
msgcnt = int64(12000000)
msgcnt = int64(1200)
group = "38"
prefill = false
loglevel = logger.InfoLevel
prefill = true
loglevel = logger.ErrorLevel
cluster *kfake.Cluster
)
var bm = &broker.Message{
Header: map[string]string{"hkey": "hval", metadata.HeaderTopic: "test"},
Body: []byte(`"body"`),
func TestMain(m *testing.M) {
cluster = kfake.MustCluster(
kfake.AllowAutoTopicCreation(),
)
defer cluster.Close()
m.Run()
}
func TestFail(t *testing.T) {
logger.DefaultLogger = slog.NewLogger()
if err := logger.DefaultLogger.Init(logger.WithLevel(loglevel)); err != nil {
t.Fatal(err)
}
ctx := context.Background()
b := kgo.NewBroker(
broker.ContentType("application/octet-stream"),
broker.Codec("application/octet-stream", codec.NewCodec()),
broker.Addrs(cluster.ListenAddrs()...),
kgo.CommitInterval(5*time.Second),
kgo.Options(
kg.ClientID("test"),
kg.FetchMaxBytes(10*1024*1024),
kg.AllowAutoTopicCreation(),
),
)
t.Logf("broker init")
if err := b.Init(); err != nil {
t.Fatal(err)
}
t.Logf("broker connect")
if err := b.Connect(ctx); err != nil {
t.Fatal(err)
}
defer func() {
t.Logf("broker disconnect")
if err := b.Disconnect(ctx); err != nil {
t.Fatal(err)
}
}()
t.Logf("broker health %v", b.Health())
msgs := make([]broker.Message, 0, msgcnt)
for i := int64(0); i < msgcnt; i++ {
m, err := b.NewMessage(ctx, metadata.Pairs("hkey", "hval"), []byte(`test`))
if err != nil {
t.Fatal(err)
}
msgs = append(msgs, m)
}
go func() {
for _, msg := range msgs {
// t.Logf("broker publish")
if err := b.Publish(ctx, "test.fail", msg); err != nil {
t.Fatal(err)
}
}
}()
// t.Skip()
idx := int64(0)
fn := func(msg broker.Message) error {
atomic.AddInt64(&idx, 1)
time.Sleep(100 * time.Millisecond)
// t.Logf("ack")
return msg.Ack()
}
sub, err := b.Subscribe(ctx, "test.fail", fn,
broker.SubscribeAutoAck(true),
broker.SubscribeGroup(group),
broker.SubscribeBodyOnly(true))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := sub.Unsubscribe(ctx); err != nil {
t.Fatal(err)
}
}()
for {
t.Logf("health check")
if !b.Health() {
t.Logf("health works")
break
}
t.Logf("health sleep")
time.Sleep(100 * time.Millisecond)
if err := b.Disconnect(ctx); err != nil {
t.Fatal(err)
}
}
}
func TestConnect(t *testing.T) {
var addrs []string
ctx := context.TODO()
b := kgo.NewBroker(
broker.Addrs(addrs...),
broker.ContentType("application/octet-stream"),
broker.Codec("application/octet-stream", codec.NewCodec()),
broker.Addrs(cluster.ListenAddrs()...),
kgo.CommitInterval(5*time.Second),
kgo.Options(kg.ClientID("test"), kg.FetchMaxBytes(10*1024*1024)),
kgo.Options(
kg.ClientID("test"),
kg.FetchMaxBytes(10*1024*1024),
kg.AllowAutoTopicCreation(),
),
)
if err := b.Init(); err != nil {
t.Fatal(err)
@@ -46,27 +147,23 @@ func TestConnect(t *testing.T) {
}
func TestPubSub(t *testing.T) {
if tr := os.Getenv("INTEGRATION_TESTS"); len(tr) > 0 {
t.Skip()
}
if err := logger.DefaultLogger.Init(logger.WithLevel(loglevel)); err != nil {
t.Fatal(err)
}
ctx := context.Background()
var addrs []string
if addr := os.Getenv("BROKER_ADDRS"); len(addr) == 0 {
addrs = []string{"127.0.0.1:29091", "127.0.0.2:29092", "127.0.0.3:29093"}
} else {
addrs = strings.Split(addr, ",")
}
b := kgo.NewBroker(
broker.Addrs(addrs...),
broker.ContentType("application/octet-stream"),
broker.Codec("application/octet-stream", codec.NewCodec()),
broker.Addrs(cluster.ListenAddrs()...),
kgo.CommitInterval(5*time.Second),
kgo.Options(kg.ClientID("test"), kg.FetchMaxBytes(10*1024*1024)),
kgo.Options(
kg.ClientID("test"),
kg.FetchMaxBytes(10*1024*1024),
kg.AllowAutoTopicCreation(),
),
)
if err := b.Init(); err != nil {
t.Fatal(err)
}
@@ -81,25 +178,26 @@ func TestPubSub(t *testing.T) {
}
}()
if prefill {
msgs := make([]*broker.Message, 0, msgcnt)
msgs := make([]broker.Message, 0, msgcnt)
for i := int64(0); i < msgcnt; i++ {
msgs = append(msgs, bm)
m, _ := b.NewMessage(ctx, metadata.Pairs("hkey", "hval"), []byte(`test`))
msgs = append(msgs, m)
}
if err := b.BatchPublish(ctx, msgs); err != nil {
if err := b.Publish(ctx, "test.pubsub", msgs...); err != nil {
t.Fatal(err)
}
// t.Skip()
}
done := make(chan bool, 1)
idx := int64(0)
fn := func(msg broker.Event) error {
fn := func(msg broker.Message) error {
atomic.AddInt64(&idx, 1)
// time.Sleep(200 * time.Millisecond)
return msg.Ack()
}
sub, err := b.Subscribe(ctx, "test", fn,
sub, err := b.Subscribe(ctx, "test.pubsub", fn,
broker.SubscribeAutoAck(true),
broker.SubscribeGroup(group),
broker.SubscribeBodyOnly(true))
@@ -124,7 +222,7 @@ func TestPubSub(t *testing.T) {
if prc := atomic.LoadInt64(&idx); prc == msgcnt {
close(done)
} else {
fmt.Printf("processed %v\n", prc)
t.Logf("processed %v of %v\n", prc, msgcnt)
}
case <-ticker.C:
close(done)

View File

@@ -4,7 +4,7 @@ import (
"context"
"github.com/twmb/franz-go/pkg/kgo"
"go.unistack.org/micro/v3/logger"
"go.unistack.org/micro/v4/logger"
)
type mlogger struct {

View File

@@ -6,7 +6,7 @@ import (
"time"
"github.com/twmb/franz-go/pkg/kgo"
"go.unistack.org/micro/v3/meter"
"go.unistack.org/micro/v4/meter"
)
type hookMeter struct {
@@ -61,7 +61,7 @@ const (
labelTopic = "topic"
)
func (m *hookMeter) OnGroupManageError(err error) {
func (m *hookMeter) OnGroupManageError(_ error) {
m.meter.Counter(metricBrokerGroupErrors).Inc()
}

View File

@@ -5,8 +5,7 @@ import (
"time"
"github.com/twmb/franz-go/pkg/kgo"
"go.unistack.org/micro/v3/broker"
"go.unistack.org/micro/v3/client"
"go.unistack.org/micro/v4/broker"
)
var (
@@ -28,16 +27,11 @@ func SubscribeContext(ctx context.Context) broker.SubscribeOption {
return broker.SetSubscribeOption(subscribeContextKey{}, ctx)
}
type publishKey struct{}
type messageKey struct{}
// PublishKey set the kafka message key (broker option)
func PublishKey(key []byte) broker.PublishOption {
return broker.SetPublishOption(publishKey{}, key)
}
// ClientPublishKey set the kafka message key (client option)
func ClientPublishKey(key []byte) client.PublishOption {
return client.SetPublishOption(publishKey{}, key)
// MessageKey set the kafka message key (broker option)
func MessageKey(key []byte) broker.MessageOption {
return broker.SetMessageOption(messageKey{}, key)
}
type optionsKey struct{}
@@ -72,6 +66,12 @@ func SubscribeOptions(opts ...kgo.Opt) broker.SubscribeOption {
}
}
type fatalOnErrorKey struct{}
func FatalOnError(b bool) broker.Option {
return broker.SetOption(fatalOnErrorKey{}, b)
}
type clientIDKey struct{}
func ClientID(id string) broker.Option {
@@ -98,14 +98,21 @@ func SubscribeMaxInFlight(n int) broker.SubscribeOption {
return broker.SetSubscribeOption(subscribeMaxInflightKey{}, n)
}
type publishPromiseKey struct{}
// PublishPromise set the kafka promise func for Produce
func PublishPromise(fn func(*kgo.Record, error)) broker.PublishOption {
return broker.SetPublishOption(publishPromiseKey{}, fn)
// SubscribeMaxInFlight max queued messages
func SubscribeFatalOnError(b bool) broker.SubscribeOption {
return broker.SetSubscribeOption(fatalOnErrorKey{}, b)
}
// ClientPublishKey set the kafka message key (client option)
func ClientPublishPromise(fn func(*kgo.Record, error)) client.PublishOption {
return client.SetPublishOption(publishPromiseKey{}, fn)
type messagePromiseKey struct{}
// MessagePromise set the kafka promise func for Produce
func MessagePromise(fn func(*kgo.Record, error)) broker.MessageOption {
return broker.SetMessageOption(messagePromiseKey{}, fn)
}
type subscribeMessagePoolKey struct{}
// SubscribeMessagePool optionaly enabled/disable message pool
func SubscribeMessagePool(b bool) broker.SubscribeOption {
return broker.SetSubscribeOption(subscribeMessagePoolKey{}, b)
}

View File

@@ -5,15 +5,17 @@ import (
"fmt"
"strconv"
"sync"
"sync/atomic"
"time"
"github.com/twmb/franz-go/pkg/kadm"
"github.com/twmb/franz-go/pkg/kgo"
"go.unistack.org/micro/v3/broker"
"go.unistack.org/micro/v3/logger"
"go.unistack.org/micro/v3/metadata"
"go.unistack.org/micro/v3/semconv"
"go.unistack.org/micro/v3/tracer"
"github.com/twmb/franz-go/pkg/kmsg"
"go.unistack.org/micro/v4/broker"
"go.unistack.org/micro/v4/logger"
"go.unistack.org/micro/v4/metadata"
"go.unistack.org/micro/v4/semconv"
"go.unistack.org/micro/v4/tracer"
)
type tp struct {
@@ -22,29 +24,34 @@ type tp struct {
}
type consumer struct {
c *kgo.Client
topic string
partition int32
c *kgo.Client
htracer *hookTracer
opts broker.SubscribeOptions
kopts broker.Options
handler broker.Handler
quit chan struct{}
done chan struct{}
recs chan kgo.FetchTopicPartition
kopts broker.Options
partition int32
opts broker.SubscribeOptions
handler interface{}
connected *atomic.Uint32
messagePool bool
}
type Subscriber struct {
c *kgo.Client
topic string
htracer *hookTracer
opts broker.SubscribeOptions
kopts broker.Options
handler broker.Handler
closed bool
done chan struct{}
consumers map[tp]*consumer
sync.RWMutex
c *kgo.Client
htracer *hookTracer
topic string
messagePool bool
handler interface{}
done chan struct{}
kopts broker.Options
opts broker.SubscribeOptions
connected *atomic.Uint32
mu sync.RWMutex
closed bool
fatalOnError bool
}
func (s *Subscriber) Client() *kgo.Client {
@@ -63,10 +70,7 @@ func (s *Subscriber) Unsubscribe(ctx context.Context) error {
if s.closed {
return nil
}
select {
// case <-ctx.Done():
// return ctx.Err()
default:
s.c.PauseFetchTopics(s.topic)
s.c.CloseAllowingRebalance()
kc := make(map[string][]int32)
@@ -77,7 +81,7 @@ func (s *Subscriber) Unsubscribe(ctx context.Context) error {
close(s.done)
s.closed = true
s.c.ResumeFetchTopics(s.topic)
}
return nil
}
@@ -113,11 +117,11 @@ func (s *Subscriber) poll(ctx context.Context) {
continue
}
s.Lock()
s.mu.Lock()
for p, l := range lmap {
s.kopts.Meter.Counter(semconv.BrokerGroupLag, "topic", s.topic, "group", s.opts.Group, "partition", strconv.Itoa(int(p)), "lag", strconv.Itoa(int(l.Lag)))
s.kopts.Meter.Counter(semconv.BrokerGroupLag, "topic", s.topic, "group", s.opts.Group, "partition", strconv.Itoa(int(p))).Set(uint64(l.Lag))
}
s.Unlock()
s.mu.Unlock()
}
}
@@ -141,8 +145,13 @@ func (s *Subscriber) poll(ctx context.Context) {
})
fetches.EachPartition(func(p kgo.FetchTopicPartition) {
tp := tp{p.Topic, p.Partition}
s.consumers[tp].recs <- p
tps := tp{p.Topic, p.Partition}
s.mu.Lock()
c := s.consumers[tps]
s.mu.Unlock()
if c != nil {
c.recs <- p
}
})
s.c.AllowRebalance()
}
@@ -155,9 +164,16 @@ func (s *Subscriber) killConsumers(ctx context.Context, lost map[string][]int32)
for topic, partitions := range lost {
for _, partition := range partitions {
tp := tp{topic, partition}
pc := s.consumers[tp]
delete(s.consumers, tp)
tps := tp{topic, partition}
s.mu.Lock()
pc, ok := s.consumers[tps]
s.mu.Unlock()
if !ok {
continue
}
s.mu.Lock()
delete(s.consumers, tps)
s.mu.Unlock()
close(pc.quit)
if s.kopts.Logger.V(logger.DebugLevel) {
s.kopts.Logger.Debug(ctx, fmt.Sprintf("[kgo] waiting for work to finish topic %s partition %d", topic, partition))
@@ -168,11 +184,21 @@ func (s *Subscriber) killConsumers(ctx context.Context, lost map[string][]int32)
}
}
func (s *Subscriber) autocommit(_ *kgo.Client, _ *kmsg.OffsetCommitRequest, _ *kmsg.OffsetCommitResponse, err error) {
if err != nil {
// s.connected.Store(0)
if s.fatalOnError {
s.kopts.Logger.Fatal(context.TODO(), "kgo.AutoCommitCallback error", err)
}
}
}
func (s *Subscriber) lost(ctx context.Context, _ *kgo.Client, lost map[string][]int32) {
if s.kopts.Logger.V(logger.DebugLevel) {
s.kopts.Logger.Debug(ctx, fmt.Sprintf("[kgo] lost %#+v", lost))
if s.kopts.Logger.V(logger.ErrorLevel) {
s.kopts.Logger.Error(ctx, fmt.Sprintf("[kgo] lost %#+v", lost))
}
s.killConsumers(ctx, lost)
// s.connected.Store(0)
}
func (s *Subscriber) revoked(ctx context.Context, c *kgo.Client, revoked map[string][]int32) {
@@ -182,6 +208,7 @@ func (s *Subscriber) revoked(ctx context.Context, c *kgo.Client, revoked map[str
s.killConsumers(ctx, revoked)
if err := c.CommitMarkedOffsets(ctx); err != nil {
s.kopts.Logger.Error(ctx, "[kgo] revoked CommitMarkedOffsets error", err)
// s.connected.Store(0)
}
}
@@ -197,28 +224,29 @@ func (s *Subscriber) assigned(_ context.Context, c *kgo.Client, assigned map[str
done: make(chan struct{}),
recs: make(chan kgo.FetchTopicPartition, 100),
handler: s.handler,
messagePool: s.messagePool,
kopts: s.kopts,
opts: s.opts,
connected: s.connected,
}
s.Lock()
s.mu.Lock()
s.consumers[tp{topic, partition}] = pc
s.Unlock()
s.mu.Unlock()
go pc.consume()
}
}
}
func (pc *consumer) consume() {
var err error
defer close(pc.done)
if pc.kopts.Logger.V(logger.DebugLevel) {
pc.kopts.Logger.Debug(pc.kopts.Context, fmt.Sprintf("starting, topic %s partition %d", pc.topic, pc.partition))
defer pc.kopts.Logger.Debug(pc.kopts.Context, fmt.Sprintf("killing, topic %s partition %d", pc.topic, pc.partition))
}
eh := pc.kopts.ErrorHandler
if pc.opts.ErrorHandler != nil {
eh = pc.opts.ErrorHandler
}
var pm *kgoMessage
for {
select {
@@ -229,97 +257,64 @@ func (pc *consumer) consume() {
ctx, sp := pc.htracer.WithProcessSpan(record)
ts := time.Now()
pc.kopts.Meter.Counter(semconv.SubscribeMessageInflight, "endpoint", record.Topic, "topic", record.Topic).Inc()
p := eventPool.Get().(*event)
p.msg.Header = nil
p.msg.Body = nil
p.topic = record.Topic
p.err = nil
p.ack = false
p.msg.Header = metadata.New(len(record.Headers))
p.ctx = ctx
if pc.messagePool {
pm = messagePool.Get().(*kgoMessage)
} else {
pm = &kgoMessage{}
}
pm.body = record.Value
pm.topic = record.Topic
pm.ack = false
pm.hdr = metadata.New(len(record.Headers))
pm.ctx = ctx
for _, hdr := range record.Headers {
p.msg.Header.Set(hdr.Key, string(hdr.Value))
pm.hdr.Set(hdr.Key, string(hdr.Value))
}
if pc.kopts.Codec.String() == "noop" {
p.msg.Body = record.Value
} else if pc.opts.BodyOnly {
p.msg.Body = record.Value
} else {
sp.AddEvent("codec unmarshal start")
err := pc.kopts.Codec.Unmarshal(record.Value, p.msg)
sp.AddEvent("codec unmarshal stop")
pm.hdr.Set("Micro-Offset", strconv.FormatInt(record.Offset, 10))
pm.hdr.Set("Micro-Partition", strconv.FormatInt(int64(record.Partition), 10))
pm.hdr.Set("Micro-Topic", record.Topic)
pm.hdr.Set("Micro-Key", string(record.Key))
pm.hdr.Set("Micro-Timestamp", strconv.FormatInt(record.Timestamp.Unix(), 10))
switch h := pc.handler.(type) {
case func(broker.Message) error:
err = h(pm)
case func([]broker.Message) error:
err = h([]broker.Message{pm})
}
pc.kopts.Meter.Counter(semconv.SubscribeMessageInflight, "endpoint", record.Topic, "topic", record.Topic).Dec()
if err != nil {
if sp != nil {
sp.SetStatus(tracer.SpanStatusError, err.Error())
}
pc.kopts.Meter.Counter(semconv.SubscribeMessageTotal, "endpoint", record.Topic, "topic", record.Topic, "status", "failure").Inc()
p.err = err
p.msg.Body = record.Value
if eh != nil {
_ = eh(p)
pc.kopts.Meter.Counter(semconv.SubscribeMessageInflight, "endpoint", record.Topic, "topic", record.Topic).Dec()
if p.ack {
} else if pc.opts.AutoAck {
pm.ack = true
}
te := time.Since(ts)
pc.kopts.Meter.Summary(semconv.SubscribeMessageLatencyMicroseconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
pc.kopts.Meter.Histogram(semconv.SubscribeMessageDurationSeconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
ack := pm.ack
if pc.messagePool {
messagePool.Put(p)
}
if ack {
pc.c.MarkCommitRecords(record)
} else {
eventPool.Put(p)
pc.kopts.Logger.Fatal(pc.kopts.Context, "[kgo] ErrLostMessage wtf?")
if sp != nil {
sp.Finish()
}
// pc.connected.Store(0)
pc.kopts.Logger.Fatal(pc.kopts.Context, "[kgo] message not commited")
return
}
eventPool.Put(p)
te := time.Since(ts)
pc.kopts.Meter.Summary(semconv.SubscribeMessageLatencyMicroseconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
pc.kopts.Meter.Histogram(semconv.SubscribeMessageDurationSeconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
continue
} else {
pc.kopts.Logger.Error(pc.kopts.Context, "[kgo]: unmarshal error", err)
}
te := time.Since(ts)
pc.kopts.Meter.Counter(semconv.SubscribeMessageInflight, "endpoint", record.Topic, "topic", record.Topic).Dec()
pc.kopts.Meter.Summary(semconv.SubscribeMessageLatencyMicroseconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
pc.kopts.Meter.Histogram(semconv.SubscribeMessageDurationSeconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
eventPool.Put(p)
pc.kopts.Logger.Fatal(pc.kopts.Context, "[kgo] Unmarshal err not handled wtf?")
if sp != nil {
sp.Finish()
return
}
}
sp.AddEvent("handler start")
err := pc.handler(p)
sp.AddEvent("handler stop")
if err == nil {
pc.kopts.Meter.Counter(semconv.SubscribeMessageTotal, "endpoint", record.Topic, "topic", record.Topic, "status", "success").Inc()
} else {
sp.SetStatus(tracer.SpanStatusError, err.Error())
pc.kopts.Meter.Counter(semconv.SubscribeMessageTotal, "endpoint", record.Topic, "topic", record.Topic, "status", "failure").Inc()
}
pc.kopts.Meter.Counter(semconv.SubscribeMessageInflight, "endpoint", record.Topic, "topic", record.Topic).Dec()
if err == nil && pc.opts.AutoAck {
p.ack = true
} else if err != nil {
p.err = err
if eh != nil {
sp.AddEvent("error handler start")
_ = eh(p)
sp.AddEvent("error handler stop")
} else {
if pc.kopts.Logger.V(logger.ErrorLevel) {
pc.kopts.Logger.Error(pc.kopts.Context, "[kgo]: subscriber error", err)
}
}
}
te := time.Since(ts)
pc.kopts.Meter.Summary(semconv.SubscribeMessageLatencyMicroseconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
pc.kopts.Meter.Histogram(semconv.SubscribeMessageDurationSeconds, "endpoint", record.Topic, "topic", record.Topic).Update(te.Seconds())
if p.ack {
eventPool.Put(p)
pc.c.MarkCommitRecords(record)
} else {
eventPool.Put(p)
pc.kopts.Logger.Fatal(pc.kopts.Context, "[kgo] ErrLostMessage wtf?")
sp.SetStatus(tracer.SpanStatusError, "ErrLostMessage")
sp.Finish()
return
}
sp.Finish()
}
}
}
}

View File

@@ -6,14 +6,14 @@ import (
"github.com/twmb/franz-go/pkg/kgo"
semconv "go.opentelemetry.io/otel/semconv/v1.18.0"
"go.unistack.org/micro/v3/metadata"
"go.unistack.org/micro/v3/tracer"
"go.unistack.org/micro/v4/metadata"
"go.unistack.org/micro/v4/tracer"
)
type hookTracer struct {
tracer tracer.Tracer
clientID string
group string
tracer tracer.Tracer
}
var messagingSystem = semconv.MessagingSystemKey.String("kafka")
@@ -32,6 +32,9 @@ var (
// the record's context, so it can be ended in the OnProduceRecordUnbuffered
// hook.
func (m *hookTracer) OnProduceRecordBuffered(r *kgo.Record) {
if !m.tracer.Enabled() {
return
}
// Set up span options.
attrs := []interface{}{
messagingSystem,
@@ -68,7 +71,7 @@ func (m *hookTracer) OnProduceRecordBuffered(r *kgo.Record) {
r.Context, _ = m.tracer.Start(r.Context, "sdk.broker", opts...)
}
setHeaders(r, omd)
setHeaders(r, omd, metadata.HeaderContentType)
}
// OnProduceRecordUnbuffered continues and ends the "publish" span for an
@@ -77,6 +80,9 @@ func (m *hookTracer) OnProduceRecordBuffered(r *kgo.Record) {
// It sets attributes with values unset when producing and records any error
// that occurred during the publish operation.
func (m *hookTracer) OnProduceRecordUnbuffered(r *kgo.Record, err error) {
if !m.tracer.Enabled() {
return
}
if span, ok := tracer.SpanFromContext(r.Context); ok {
span.AddLabels(
semconv.MessagingKafkaDestinationPartition(int(r.Partition)),
@@ -96,6 +102,9 @@ func (m *hookTracer) OnProduceRecordUnbuffered(r *kgo.Record, err error) {
// OnFetchRecordUnbuffered hook and can be used in downstream consumer
// processing.
func (m *hookTracer) OnFetchRecordBuffered(r *kgo.Record) {
if !m.tracer.Enabled() {
return
}
// Set up the span options.
attrs := []interface{}{
messagingSystem,
@@ -135,12 +144,15 @@ func (m *hookTracer) OnFetchRecordBuffered(r *kgo.Record) {
r.Context, _ = m.tracer.Start(r.Context, "sdk.broker", opts...)
}
setHeaders(r, omd)
setHeaders(r, omd, metadata.HeaderContentType)
}
// OnFetchRecordUnbuffered continues and ends the "receive" span for an
// unbuffered record.
func (m *hookTracer) OnFetchRecordUnbuffered(r *kgo.Record, _ bool) {
if !m.tracer.Enabled() {
return
}
span, _ := tracer.SpanFromContext(r.Context)
span.Finish()
}
@@ -155,6 +167,13 @@ func (m *hookTracer) OnFetchRecordUnbuffered(r *kgo.Record, _ bool) {
// not a record which has been created for producing, so call this at the start of each
// iteration of your processing for the record.
func (m *hookTracer) WithProcessSpan(r *kgo.Record) (context.Context, tracer.Span) {
if r.Context == nil {
r.Context = context.Background()
}
if !m.tracer.Enabled() {
return r.Context, nil
}
// Set up the span options.
attrs := []interface{}{
messagingSystem,