[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hubble/relay: fix uptime value in ServerStatus implementation #23966

Merged
merged 1 commit into from
Feb 24, 2023

Conversation

rolinh
Copy link
Member
@rolinh rolinh commented Feb 23, 2023

Hubble Relay implements Hubble's observer.ServerStatus endpoint from a cluster-wide perspective. It forwards ServerStatus requests to Hubble peers in the cluster and aggregates the responses. However, while the total number of seen flows can be accumulated, the uptime value cannot. Therefore, the intent for Relay's implementation of ServerStatus regarding the uptime value was to return the uptime of the oldest running Hubble server in the cluster. However, the implementation is wrong and Relay reports the newest uptime from all the nodes in the cluster.

In both the Hubble CLI and Hubble UI, the flow rate is reported as the number of flows per second, e.g. when hubble status, which is computed as the number of seen flows divided by the uptime. As you can imagine, the flows/s rate would be very wrong when a Hubble server is restarted as the accumulated seen flows in the cluster would be a large number while the uptime, because of the bug mentioned above, would be a very small number resulting in wrong reporting of the flows rate.

This patch fixes this bug by correctly reporting the oldest uptime from all connected Hubble peers instead of the newest one.

Fixes: cilium/hubble#911

Hubble Relay: fix reported uptime

Hubble Relay implements Hubble's `observer.ServerStatus` endpoint from
a cluster-wide perspective. It forwards `ServerStatus` requests to
Hubble peers in the cluster and aggregates the responses. However, while
the total number of seen flows can be accumulated, the uptime value
cannot. Therefore, the intent for Relay's implementation of
`ServerStatus` regarding the uptime value was to return the uptime of
the oldest running Hubble server in the cluster. However, the
implementation is wrong and Relay reports the newest uptime from all the
nodes in the cluster.

In both the Hubble CLI and Hubble UI, the flow rate is reported as the
number of flows per second, e.g. when `hubble status`, which is computed
as the number of seen flows divided by the uptime. As you can imagine,
the flows/s rate would be very wrong when a Hubble server is restarted
as the accumulated seen flows in the cluster would be a large number
while the uptime, because of the bug mentioned above, would be a very
small number resulting in wrong reporting of the flows rate.

This patch fixes this bug by correctly reporting the oldest uptime from
all connected Hubble peers instead of the newest one.

Signed-off-by: Robin Hahling <robin.hahling@gw-computing.net>
@rolinh rolinh added release-note/bug This PR fixes an issue in a previous release of Cilium. sig/hubble Impacts hubble server or relay labels Feb 23, 2023
@rolinh rolinh requested a review from a team as a code owner February 23, 2023 09:07
@rolinh rolinh requested a review from chancez February 23, 2023 09:07
@rolinh
Copy link
Member Author
rolinh commented Feb 23, 2023

The bug fix is covered by a unit test, Travis is ✔️ and reviews are in. I'll mark the PR as ready to merge.

@rolinh rolinh added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Feb 23, 2023
@sayboras sayboras merged commit 8401242 into master Feb 24, 2023
@sayboras sayboras deleted the pr/rolinh/fix-relay-status-uptime branch February 24, 2023 02:16
@sayboras sayboras mentioned this pull request Feb 28, 2023
10 tasks
@julianwiedmann julianwiedmann added backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. and removed backport-pending/1.13 labels Apr 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/bug This PR fixes an issue in a previous release of Cilium. sig/hubble Impacts hubble server or relay
Projects
No open projects
Status: Released
Development

Successfully merging this pull request may close these issues.

Hubble UI shows incorrect amount of flows/s
4 participants