8000 Merge 0.14.0 to master by alex-zaitsev · Pull Request #683 · Altinity/clickhouse-operator · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
223 commits
Select commit Hold shift + click to select a range
50dd260
0.14.0
sunsingerus Jan 5, 2021
a44f9bc
dev: propagate context to cache syncer
sunsingerus Jan 7, 2021
7cbfd9b
env: rebuld new version
sunsingerus Jan 13, 2021
57efa1d
dev: make connection context-enabled
sunsingerus Jan 13, 2021
baaf9ed
dev: introduce context utils
sunsingerus Jan 13, 2021
bc0ddb3
dev: rety should be contextable
sunsingerus Jan 13, 2021
fbdabba
dev: introduce reconcile context
sunsingerus Jan 13, 2021
640edcc
dev: separate controller types
sunsingerus Jan 14, 2021
7617e33
dev: make onjects queue to conform with interfaces
sunsingerus Jan 14, 2021
891a9b4
dev: action plan should understand deletion timestamp
sunsingerus Jan 14, 2021
abbd777
dev: enqueue only changes with actions to be done
sunsingerus Jan 14, 2021
0fed0ea
dev: use handle instead of uid
sunsingerus Jan 14, 2021
3bf9d53
dev: introduce cancel+priority+unique-able queue manager and cancelat…
sunsingerus Jan 15, 2021
10b3636
libs
sunsingerus Jan 21, 2021
60f6f2b
dev: vendors
sunsingerus Feb 22, 2021
43a7022
env: merge vegrant
sunsingerus Feb 22, 2021
fca44fa
dev: release
sunsingerus Feb 22, 2021
d9765d8
new libs
sunsingerus Feb 22, 2021
164f620
docs
sunsingerus Feb 22, 2021
063b33b
metrics exporter
sunsingerus Feb 22, 2021
aafb395
tests
sunsingerus Feb 22, 2021
56c277e
dev: main operator command
sunsingerus Feb 22, 2021
b1898a6
grafana dashboard
sunsingerus Feb 22, 2021
106025d
env: deploy grafana
sunsingerus Feb 22, 2021
ef42871
env: SIGKILL/SIGSTOP can not be trapped.
sunsingerus Feb 22, 2021
75688ba
env: deploy prometheus
sunsingerus Feb 22, 2021
1c46fda
env: deploy zk
sunsingerus Feb 22, 2021
77fa642
env: deploy dev env
sunsingerus Feb 22, 2021
7491391
env: deploy operator
sunsingerus Feb 22, 2021
b00806b
env: setup Ordinary database engine
sunsingerus Feb 22, 2021
3aa6d02
dev: introduce announcer
sunsingerus Feb 22, 2021
5789190
dev: intriduce user-friendly dumper
sunsingerus Feb 22, 2021
9801b46
dev: intriduce stack unwinder
sunsingerus Feb 22, 2021
19a635b
dev; do reperatable fingerprinting
sunsingerus Feb 22, 2021
55805ff
dev: introduce repeatable fingerprint
sunsingerus Feb 22, 2021
c627e92
dev: enhanse map functions
sunsingerus Feb 22, 202 8000 1
d591473
dev: log retries by parameter function
sunsingerus Feb 22, 2021
afa8dd9
dev: minor utils
sunsingerus Feb 22, 2021
cfe8b05
dev: update looger in chop
sunsingerus Feb 22, 2021
295a177
dev: introduce StatefulSet status
sunsingerus Feb 22, 2021
82e0ed7
dev: introduce timeout
sunsingerus Feb 22, 2021
d16463b
dev: extract query
sunsingerus Feb 22, 2021
dc236d9
dev: clickhouse logger
sunsingerus Feb 22, 2021
c126349
dev: minor
sunsingerus Feb 22, 2021
7dd3646
dev: announcer log
sunsingerus Feb 22, 2021
2549277
controller logger
sunsingerus Feb 22, 2021
4f8935e
dev: merge logger
sunsingerus Feb 22, 2021
f1b4a78
dev: merge logger
sunsingerus Feb 22, 2021
ea3d57d
dev: rename events
sunsingerus Feb 22, 2021
372dc53
dev: intorduce StatefulSet and Pod getters
sunsingerus Feb 22, 2021
0f21a6b
dev: merge logger into labeler
sunsingerus Feb 22, 2021
a1ca873
dev: add ready label and logger
sunsingerus Feb 22, 2021
a6e1481
dev: merge poller
sunsingerus Feb 22, 2021
1662477
dev: merge logger
sunsingerus Feb 22, 2021
779482c
minor
sunsingerus Feb 22, 2021
7eb13bd
dev: const
sunsingerus Feb 22, 2021
157cfca
dev: cofngi generator
sunsingerus Feb 22, 2021
5445116
dev: labels and logs merger
sunsingerus Feb 22, 2021
fb2c7b7
dev: merge labeler
sunsingerus Feb 22, 2021
b04618f
minor
sunsingerus Feb 22, 2021
f4ded3a
dev: normalizer
sunsingerus Feb 22, 2021
5e9775a
dev: schemer logger
sunsingerus Feb 22, 2021
141fb19
merge worker
sunsingerus Feb 22, 2021
d97e6c2
Merge pull request #658 from Altinity/dev-vladislav
sunsingerus Feb 22, 2021
df79dae
dev: introduce registry
sunsingerus Feb 26, 2021
6c7119c
dev: introduce reconcile context
sunsingerus Feb 26, 2021
7faa0e7
dev: introduce discoverer
sunsingerus Feb 26, 2021
62cd867
dev: fix subtract behavior
sunsingerus Feb 26, 2021
1d04e48
dev: export some functions from labeler
sunsingerus Feb 26, 2021
7a7b5a3
dev: export Create functions from creator
sunsingerus Feb 26, 2021
30de73f
dev: utililze reconcile context with discoverer
sunsingerus Feb 26, 2021
24f7a10
Merge pull request #660 from Altinity/dev-vladislav
sunsingerus Feb 26, 2021
ff95de4
dev: delete explicitly
sunsingerus Feb 26, 2021
6d1d9c7
Merge remote-tracking branch 'altinity/dev-vladislav' into 0.14.0
sunsingerus Feb 26, 2021
d53d8cc
dev: simplify and fix registry subtraction
sunsingerus Feb 27, 2021
3271bae
dev: clarify unsused objects deletion
sunsingerus Feb 27, 2021
e54e5c5
dev: restart reconcile cycle upon adding finalizer
sunsingerus Mar 3, 2021
4ee3c82
dev: report errors on config maps update
sunsingerus Mar 3, 2021 8000
a8c7d3e
dev: extract tables migration into separate function
sunsingerus Mar 3, 2021
ff0e7ee
dev: introduce context in more functions
sunsingerus Mar 3, 2021
2614f37
dev: minor
sunsingerus Mar 3, 2021
1a19a34
dev: modify pvc reconsiler
sunsingerus Mar 3, 2021
634d0dc
dev: actionPlan should know about finalizer modifications
sunsingerus Mar 3, 2021
60ac905
dev: dump action plan
sunsingerus Mar 3, 2021
e1d1168
dev: introduce reclaimPolicy labeler
sunsingerus Mar 4, 2021
8b6f3e0
dev: introduce PVC preparer
sunsingerus Mar 4, 2021
1bb81e6
dev: extract updaters as separate entities
sunsingerus Mar 4, 2021
7dd5a7e
dev: walk over reconcilers
sunsingerus Mar 4, 2021
f55c2af
dev: minor ap
sunsingerus Mar 4, 2021
fce1313
dev: introduce string functions for PVCReclaimPolicy
sunsingerus Mar 4, 2021
9911a6d
dev: use PVC stringer funstions
sunsingerus Mar 4, 2021
747c0fb
dev: register PVC as reconciled
sunsingerus Mar 4, 2021
d5ec69c
dev: introduce PVC reclaim policy getter
sunsingerus Mar 4, 2021
41b863a
dev: delete deletable pvcs
sunsingerus Mar 4, 2021
f82fede
dev: introduce apply resource func
sunsingerus Mar 5, 2021
919067c
dev: add cur and new StatefulSets
sunsingerus Mar 5, 2021
e6280c3
dev: populate desired sts
sunsingerus Mar 5, 2021
c72f725
dev: simplify resources reconciler
sunsingerus Mar 5, 2021
a09d792
dev: clarify action
sunsingerus Mar 5, 2021
315a328
test: upgrade version
sunsingerus Mar 5, 2021
6aae300
env: setup code generator
sunsingerus Mar 12, 2021
4494c90
env: generated code
sunsingerus Mar 12, 2021
7d0ef02
dev: include client code
sunsingerus Mar 12, 2021
46e8135
dev: has function has to understand zero value
sunsingerus Mar 12, 2021
dea68f3
dev: controller debug log
sunsingerus Mar 12, 2021
f38d7d7
dev: switch to pointers
sunsingerus Mar 12, 2021
356c6b6
dev: create in case not exist
sunsingerus Mar 12, 2021
49dfc3a
dev: switch to pointers
sunsingerus Mar 12, 2021
31d340d
dev: simplify merger
sunsingerus Mar 12, 2021
18a4f74
dev: simplify template merger
sunsingerus Mar 12, 2021
ba1eae8
dev: simplify zk merger
sunsingerus Mar 12, 2021
6771f1c
dev: pointers
sunsingerus Mar 12, 2021
401b318
dev: rewrite settings
sunsingerus Mar 12, 2021
b40e0e8
dev: adpot updated settings
sunsingerus Mar 12, 2021
5713e1e
dev: use creator functions, not objerts
sunsingerus Mar 12, 2021
89a21d8
dev: refactor normalizer
sunsingerus Mar 12, 2021
f0659d6
dev: clarify xml builder
sunsingerus Mar 12, 2021
9e357ea
dev: todo
sunsingerus Mar 12, 2021
f4cefaf
dev: extract getter/setter
sunsingerus Mar 12, 2021
c44162e
dev: ensure firelds for migration
sunsingerus Mar 12, 2021
9317a5b
tests: fix version
sunsingerus Mar 12, 2021
65e754c
dev: yaml labels
sunsingerus Mar 12, 2021
90e8494
dev: tag
sunsingerus Mar 12, 2021
85e34f9
dev: explanation
sunsingerus Mar 18, 2021
4e31791
dev: add k8s client to kube machinery
sunsingerus Mar 18, 2021
24cb54d
dev: add k8s client to chop
sunsingerus Mar 18, 2021
cf41f92
dev: pass clickhouse lcient to chop
sunsingerus Mar 18, 2021
1b52ec0
dev: add k8s client to metrics exporter
sunsingerus Mar 18, 2021
aa184a0
dev: introduce secret passwords in cofiguration
sunsingerus Mar 18, 2021
1b6a0a9
dev: support secret handling
sunsingerus Mar 18, 2021
1b1dc0e
env: secret in prod config
sunsingerus Mar 18, 2021
8594082
env: add secret to dev config
sunsingerus Mar 18, 2021
0e04e6f
env: add secret file
sunsingerus Mar 18, 2021
4bfd655
env: regenerate deployments
sunsingerus Mar 18, 2021
eeb3e2b
dev: revisit deleter
sunsingerus Mar 18, 2021
24ca0d4
dev: add more clear error handling in creator
sunsingerus Mar 18, 2021
ca9d6f4
dev: create tables on host only in case no errors met
sunsingerus Mar 18, 2021
feccc6f
dev: drop dns on statefulset deletion
sunsingerus Mar 18, 2021
25b468b
dev: add object meta for volume claim template
sunsingerus Mar 18, 2021
e9daac9
dev: update copy functions
sunsingerus Mar 18, 2021
4b8e246
dev: merge annotations and lables into PVC
sunsingerus Mar 18, 2021
632e49c
dev: register ObjectMeta with Labels and Annotations
sunsingerus Mar 18, 2021
9d5998f
test: offset
sunsingerus Mar 18, 2021
4ca4f02
test: PVC annotation
sunsingerus Mar 18, 2021
d24383e
env: community-operators cmp script
sunsingerus Mar 19, 2021
275b56e
dev: comment
sunsingerus Mar 19, 2021
acb64c0
dev: chop functions clarification
sunsingerus Mar 19, 2021
e7ed727
increased kube libs version
preved911 Mar 24, 2021
45e36b2
dev: specify empty fields
sunsingerus Mar 25, 2021
d428b84
env: chi crd validation
sunsingerus Mar 25, 2021
2193004
env: chit crd validation
sunsingerus Mar 25, 2021
bcebd7d
env: chop config crd validation
sunsingerus Mar 25, 2021
e2a74d3
env: rebuild all install manifests
sunsingerus Mar 25, 2021
0fbd609
dev: omit empty fields
sunsingerus Mar 26, 2021
5680a39
dev: normalizer
sunsingerus Mar 26, 2021
da43217
dev: handle inccorect pvc update
sunsingerus Mar 26, 2021
e2cf3e3
docs: add PVC labels annotations example
sunsingerus Mar 26, 2021
1b2ca4b
docs: max example
sunsingerus Mar 26, 2021
133fab5
dev: naming
sunsingerus Mar 26, 2021
9e24a78
dev: do not inherit from zero values
sunsingerus Mar 26, 2021
a2641cb
dev: clarify settings
sunsingerus Mar 26, 2021
d372baf
dev: omit empoty settings
sunsingerus Mar 26, 2021 8000
9dedaf7
dev: allow to skip templates in initialization
sunsingerus Mar 26, 2021
330aa60
dev: allow to omit DistributedDDL
sunsingerus Mar 26, 2021
bcbb6e4
dev: extract indexes as separate entities
sunsingerus Mar 26, 2021
e98f618
dev: clarify naming
sunsingerus Mar 26, 2021
f63dc79
dev: report normalized on leevel 1
sunsingerus Mar 26, 2021
dedf4b4
Merge pull request #672 from preved911/dev/increase_kubernetes_libs_v…
sunsingerus Apr 1, 2021
1872454
dev: adopt
sunsingerus Apr 1, 2021
bd51419
Merge branch '0.14.0' of github.com:Altinity/clickhouse-operator into…
Slach Apr 1, 2021
017654d
update controller.go after pull request https://github.com/Altinity/c…
Slach Apr 1, 2021
d1eb251
dev: format and resolve conflicts
sunsingerus Apr 1, 2021
96dc289
dev: wait for timeout before dropping metric which is not consumed by…
sunsingerus Apr 1, 2021
fb8ab40
docs: intstruction on how to make a Pull Request
sunsingerus Apr 2, 2021
714c0a5
docs: ensure list of docs is up-to-date
sunsingerus Apr 2, 2021
583af58
restrict clickhouse-operator role, after analysis ./tests/test.py log…
Slach Apr 4, 2021
7ac5886
Merge pull request #675 from Altinity/improve_rbac
sunsingerus Apr 5, 2021
35676da
Sanity check that data is not lost when operator is upgraded
alex-zaitsev Apr 6, 2021
a596407
dev: push action and error to the head of the list
sunsingerus Apr 8, 2021
c5fd434
dev: add uuid
sunsingerus Apr 8, 2021
4674318
push task id into status object
sunsingerus Apr 8, 2021
46608f1
dev: fill status
sunsingerus Apr 8, 2021
c9a5805
dev: intorduce task id into spec on top-level
sunsingerus Apr 8, 2021
e08f282
dev: normalize task id properly
sunsingerus Apr 8, 2021
da4ccfe
dev: move delete normalization with development
sunsingerus Apr 8, 2021
b5aa8f5
dev: intorduce task is to manifests components
sunsingerus Apr 8, 2021
5d6574b
env: regenerate manifests
sunsingerus Apr 8, 2021
ed77777
allow filtering labels
aksdb Apr 12, 2021
b3850e7
Merge pull request #676 from Staffbase/support_label_exclusions
sunsingerus Apr 12, 2021
b8f0263
Test for mixed replica configuration
alex-zaitsev Apr 14, 2021
a60317b
Minor test improvement
alex-zaitsev Apr 14, 2021
a436960
Readiness probe
alex-zaitsev Apr 19, 2021
a022337
Bump stable version
alex-zaitsev Apr 19, 2021
7007eab
dev: met4rics exporter should skip stopped clusters with auto-discovery
sunsingerus Apr 19, 2021
a7fc96f
Merge remote-tracking branch 'altinity/0.14.0' into 0.14.0
sunsingerus Apr 19, 2021
4f40670
dev: exporter
sunsingerus Apr 19, 2021
cad49b9
replace TooMuch -> TooMany
Slach Apr 21, 2021
d403ebc
Merge pull request #679 from Altinity/improve_prometheus_alerts
alex-zaitsev Apr 21, 2021
8e88c3c
add ClickHouseDetachedParts draft variant
Slach Apr 21, 2021
6b1815d
Merge branch '0.14.0' of github.com:Altinity/clickhouse-operator into…
Slach Apr 21, 2021
ff869fc
test ClickHouseDetachedParts passed on 0.14.0
Slach Apr 22, 2021
e8778b0
return audit2rbac to Vagrantfile
Slach Apr 22, 2021
92e3b74
Merge pull request #680 from Altinity/add_detached_parts_monitoring
alex-zaitsev Apr 22, 2021
b2278d3
Test for buffer table
alex-zaitsev Apr 23, 2021
650bf08
dev: introduce scope labels option
sunsingerus Apr 23, 2021
190c8a3
dev: config
sunsingerus Apr 23, 2021
c717a19
env: manifests
sunsingerus Apr 23, 2021
fcbbcb4
dev: CRD add label field
sunsingerus Apr 23, 2021
27f6b9f
env: manifests
sunsingerus Apr 23, 2021
2d7e559
dev: introduce ignoreOnPropagationLabels
sunsingerus Apr 23, 2021
21b0460
conf: ignoreOnPropagationLabels
sunsingerus Apr 23, 2021
962eca6
crd: ignoreOnPropagationLabels
sunsingerus Apr 23, 2021
c3325d7
env: manifests
sunsingerus Apr 23, 2021
0e4d96b
dev: ensure string-bool value
sunsingerus Apr 23, 2021
038acb8
env: manifests
sunsingerus Apr 23, 2021
54aa51a
dev: implement excludeFromPropagationLabels
sunsingerus Apr 23, 2021
2221aec
crd: deploy/dev/clickhouse-operator-install-yaml-template-01-section-…
sunsingerus Apr 23, 2021
772f5f3
add config option excludeFromPropagationLabels
sunsingerus Apr 23, 2021
9839b64
manifests
sunsingerus Apr 23, 2021
32b7b83
add `Detached Parts` instead of `Inactive Parts` to Grafana dashboard
Slach Apr 24, 2021
7dcd102
add Legend for reason
Slach Apr 25, 2021
24db804
add Legend for reason
Slach Apr 25, 2021
97aedc8
Merge pull request #682 from Altinity/add_detached_parts_monitoring
Slach Apr 26, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
103 changes: 86 additions & 17 deletions Vagrantfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,26 @@ def total_cpus
Etc.nprocessors
end

def get_provider
provider='virtualbox'
for arg in ARGV
if ['hyperv','docker'].include? arg
provider=arg
end
end
return provider
end


Vagrant.configure(2) do |config|
config.vm.box = "generic/ubuntu2004"
config.vm.box_check_update = false
config.vm.synced_folder ".", "/vagrant"

if get_provider == "hyperv"
config.vm.synced_folder ".", "/vagrant", type: "smb", smb_username: ENV['USERNAME'], smb_password: ENV['PASSWORD'], mount_options: ["domain="+ENV['USERDOMAIN'], "user="+ENV['USERNAME'], "vers=3.0"," mfsymlinks"]
else
config.vm.synced_folder ".", "/vagrant"
end

if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
Expand All @@ -20,6 +35,7 @@ Vagrant.configure(2) do |config|
config.timezone.value = "UTC"
end


config.vm.provider "virtualbox" do |vb|
vb.gui = false
vb.cpus = total_cpus
Expand All @@ -32,6 +48,19 @@ Vagrant.configure(2) do |config|
vb.customize ["guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 10000]
end

config.vm.provider "hyperv" do |hv|
# hv.gui = false
# hv.default_nic_type = "virtio"
hv.cpus = total_cpus
hv.maxmemory = "6144"
hv.memory = "2048"
hv.enable_virtualization_extensions = true
hv.linked_clone = true
hv.vm_integration_services = {
time_synchronization: true,
}
end

config.vm.define :clickhouse_operator do |clickhouse_operator|
clickhouse_operator.vm.network "private_network", ip: "172.16.2.99", nic_type: "virtio"
# port forwarding works only when pair with kubectl port-forward
Expand All @@ -52,24 +81,40 @@ Vagrant.configure(2) do |config|
config.vm.provision "shell", inline: <<-SHELL
set -xeuo pipefail
export DEBIAN_FRONTEND=noninteractive
# make linux fast again
if [[ "0" == $(grep "mitigations" /etc/default/grub | wc -l) ]]; then
echo 'GRUB_CMDLINE_LINUX="noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off"' >> /etc/default/grub
echo 'GRUB_CMDLINE_LINUX_DEFAULT="quiet splash noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off"' >> /etc/default/grub
grub-mkconfig
fi
systemctl enable systemd-timesyncd
systemctl start systemd-timesyncd

apt-get update
apt-get install --no-install-recommends -y apt-transport-https ca-certificates software-properties-common curl
apt-get install --no-install-recommends -y htop ethtool mc curl wget jq socat git ntp
apt-get install --no-install-recommends -y htop ethtool mc curl wget jq socat git

# yq
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CC86BB64
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys CC86BB64
add-apt-repository ppa:rmescandon/yq
apt-get install --no-install-recommends -y yq

# clickhouse
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E0C56BD4
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv E0C56BD4
add-apt-repository "deb http://repo.clickhouse.tech/deb/stable/ main/"
apt-get install --no-install-recommends -y clickhouse-client

# golang
export GOLANG_VERSION=1.16
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F6BC817356A3D45E
add-apt-repository ppa:longsleep/golang-backports
apt-get install --no-install-recommends -y golang-${GOLANG_VERSION}-go
ln -nvsf /usr/lib/go-${GOLANG_VERSION}/bin/go /bin/go
ln -nvsf /usr/lib/go-${GOLANG_VERSION}/bin/gofmt /bin/gofmt

# docker
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8D81803C0EBFCD88
add-apt-repository "deb https://download.docker.com/linux/ubuntu focal test"
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb https://download.docker.com/linux/ubuntu $(lsb_release -cs) test"
apt-get install --no-install-recommends -y docker-ce pigz

# docker compose
Expand All @@ -88,9 +133,13 @@ Vagrant.configure(2) do |config|
sha256sum -c /usr/local/bin/k9s.sha256
tar --verbose -zxvf /usr/local/bin/k9s_${K9S_VERSION}_Linux_x86_64.tar.gz -C /usr/local/bin k9s

# audit2rbac
AUDIT2RBAC_VERSION=0.8.0
curl -sL https://github.com/liggitt/audit2rbac/releases/download/v${AUDIT2RBAC_VERSION}/audit2rbac-linux-amd64.tar.gz | tar -zxvf - -C /usr/local/bin

# minikube
MINIKUBE_VERSION=1.17.1
MINIKUBE_VERSION=1.18.1
# MINIKUBE_VERSION=1.19.0
wget -c --progress=bar:force:noscroll -O /usr/local/bin/minikube https://github.com/kubernetes/minikube/releases/download/v${MINIKUBE_VERSION}/minikube-linux-amd64
chmod +x /usr/local/bin/minikube
# required for k8s 1.18+
Expand All @@ -100,10 +149,10 @@ Vagrant.configure(2) do |config|
# export VALIDATE_YAML=false # only for 1.14
# K8S_VERSION=${K8S_VERSION:-1.15.12}
# K8S_VERSION=${K8S_VERSION:-1.16.15}
# K8S_VERSION=${K8S_VERSION:-1.17.14}
# K8S_VERSION=${K8S_VERSION:-1.18.12}
# K8S_VERSION=${K8S_VERSION:-1.19.7}
K8S_VERSION=${K8S_VERSION:-1.20.2}
# K8S_VERSION=${K8S_VERSION:-1.17.17}
# K8S_VERSION=${K8S_VERSION:-1.18.18}
# K8S_VERSION=${K8S_VERSION:-1.19.10}
K8S_VERSION=${K8S_VERSION:-1.20.6}
export VALIDATE_YAML=true

killall kubectl || true
Expand All @@ -114,6 +163,16 @@ Vagrant.configure(2) do |config|
mkdir -p /home/vagrant/.minikube
ln -svf /home/vagrant/.minikube /root/.minikube

mkdir -p /root/.minikube/files/etc/ssl/certs

cat <<EOF >/root/.minikube/files/etc/ssl/certs/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF

mkdir -p /home/vagrant/.kube
ln -svf /home/vagrant/.kube /root/.kube

Expand All @@ -123,14 +182,16 @@ Vagrant.configure(2) do |config|
sudo -H -u vagrant minikube config set memory 5G
sudo -H -u vagrant minikube config set driver docker
sudo -H -u vagrant minikube config set kubernetes-version ${K8S_VERSION}
sudo -H -u vagrant minikube start
sudo -H -u vagrant minikube start --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml --extra-config=apiserver.audit-log-path=-
sudo -H -u vagrant minikube addons enable ingress
sudo -H -u vagrant minikube addons enable ingress-dns
sudo -H -u vagrant minikube addons enable metrics-server

# minikube delete
# rm -rf /tmp/juju*
# minikube config set vm-driver none
# minikube config set kubernetes-version ${K8S_VERSION}
# minikube start --vm=true
# minikube start
# minikube addons enable ingress
# minikube addons enable ingress-dns
# minikube addons enable metrics-server
Expand All @@ -147,13 +208,14 @@ Vagrant.configure(2) do |config|
source $HOME/.bashrc
export KREW_ROOT=/home/vagrant/.krew
kubectl krew install tap
kubectl krew install debug
kubectl krew install sniff
kubectl krew install flame
kubectl krew install minio
# look to https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container
# kubectl krew install debug

cd /vagrant/

git_branch=$(git rev-parse --abbrev-ref HEAD)
git_branch=$(git rev-parse --abbrev-ref HEAD /vagrant/)
export OPERATOR_RELEASE=$(cat release)
export BRANCH=${BRANCH:-$git_branch}
export OPERATOR_NAMESPACE=${OPERATOR_NAMESPACE:-kube-system}
Expand Down Expand Up @@ -204,9 +266,16 @@ Vagrant.configure(2) do |config|

pip3 install -r /vagrant/tests/requirements.txt

python3 /vagrant/tests/test_metrics_alerts.py
python3 /vagrant/tests/test.py -->
python3 /vagrant/tests/test_examples.py
python3 /vagrant/tests/test_metrics_exporter.py
python3 /vagrant/tests/test_metrics_alerts.py

# audit2rbac
kubectl logs kube-apiserver-minikube -n kube-system | grep audit.k8s.io/v1 > /tmp/audit2rbac.log
audit2rbac -f /tmp/audit2rbac.log --serviceaccount kube-system:clickhouse-operator > /tmp/audit2rbac.yaml
# cp -fv /tmp/audit2rbac.yaml /vagrant/deploy/dev/clickhouse-operator-install-yaml-template-02-section-rbac-restricted.yaml

SHELL

end
4 changes: 2 additions & 2 deletions cmd/metrics_exporter/app/metrics_exporter.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,10 @@ func Run() {
log.Infof("Starting metrics exporter. Version:%s GitSHA:%s BuiltAt:%s\n", version.Version, version.GitSHA, version.BuiltAt)

// Initialize k8s API clients
_, chopClient := chop.GetClientset(kubeConfigFile, masterURL)
kubeClient, chopClient := chop.GetClientset(kubeConfigFile, masterURL)

// Create operator instance
chop := chop.GetCHOp(chopClient, chopConfigFile)
chop := chop.GetCHOp(kubeClient, chopClient, chopConfigFile)
chop.SetupLog()
log.Info(chop.Config().String(true))

Expand Down
2 changes: 1 addition & 1 deletion cmd/operator/app/clickhouse_operator.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ func Run() {
kubeClient, chopClient := chop.GetClientset(kubeConfigFile, masterURL)

// Create operator instance
chop := chop.GetCHOp(chopClient, chopConfigFile)
chop := chop.GetCHOp(kubeClient, chopClient, chopConfigFile)
chop.SetupLog()
log.V(1).A().Info("Log options parsed")
log.Info(chop.Config().String(true))
Expand Down
35 changes: 34 additions & 1 deletion config/config-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,13 @@ chConfigNetworksHostRegexpTemplate: "(chi-{chi}-[^.]+\\d+-\\d+|clickhouse\\-{chi
# located in `chUsersConfigsPath` folder
chUsername: clickhouse_operator
chPassword: clickhouse_operator_password

# Location of k8s Secret with username and password to be used by operator to connect to ClickHouse instances
# Can be used instead of explicitly specified username and password
chCredentialsSecretNamespace: "dev"
chCredentialsSecretName: "chop-clickhouse-credentials"

# Port where to connect to ClickHouse instances to
chPort: 8123

################################################
Expand All @@ -104,7 +111,7 @@ chPort: 8123

logtostderr: "true"
alsologtostderr: "false"
v: "2"
v: "3"
stderrthreshold: ""
vmodule: ""
log_backtrace_at: ""
Expand All @@ -119,3 +126,29 @@ log_backtrace_at: ""
reconcileThreadsNumber: 1
reconcileWaitExclude: false
reconcileWaitInclude: false

################################################
##
## Labels management parameters
##
################################################

# When propagating labels from the chi's `metadata.labels` section to child objects' `metadata.labels`,
# exclude labels from the following list:
#excludeFromPropagationLabels:
# - "labelA"
# - "labelB"

# Whether to append *Scope* labels to StatefulSet and Pod.
# Full list of available *scope* labels check in labeler.go
# LabelShardScopeIndex
# LabelReplicaScopeIndex
# LabelCHIScopeIndex
# LabelCHIScopeCycleSize
# LabelCHIScopeCycleIndex
# LabelCHIScopeCycleOffset
# LabelClusterScopeIndex
# LabelClusterScopeCycleSize
# LabelClusterScopeCycleIndex
# LabelClusterScopeCycleOffset
appendScopeLabels: "yes"
36 changes: 35 additions & 1 deletion config/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,14 +86,22 @@ chConfigNetworksHostRegexpTemplate: "(chi-{chi}-[^.]+\\d+-\\d+|clickhouse\\-{chi
##
################################################

# ClickHouse credentials (username, password and port) to be used by operator to connect to ClickHouse instances for:
# ClickHouse credentials (username, password and port) to be used by operator to connect to ClickHouse instances
# for:
# 1. Metrics requests
# 2. Schema maintenance
# 3. DROP DNS CACHE
# User with such credentials can be specified in additional ClickHouse .xml config files,
# located in `chUsersConfigsPath` folder
chUsername: clickhouse_operator
chPassword: clickhouse_operator_password

# Location of k8s Secret with username and password to be used by operator to connect to ClickHouse instances
# Can be used instead of explicitly specified username and password
chCredentialsSecretNamespace: ""
chCredentialsSecretName: ""

# Port where to connect to ClickHouse instances to
chPort: 8123

################################################
Expand All @@ -119,3 +127,29 @@ log_backtrace_at: ""
reconcileThreadsNumber: 10
reconcileWaitExclude: false
reconcileWaitInclude: false

################################################
##
## Labels management parameters
##
################################################

# When propagating labels from the chi's `metadata.labels` section to child objects' `metadata.labels`,
# exclude labels from the following list:
#excludeFromPropagationLabels:
# - "labelA"
# - "labelB"

# Whether to append *Scope* labels to StatefulSet and Pod.
# Full list of available *scope* labels check in labeler.go
# LabelShardScopeIndex
# LabelReplicaScopeIndex
# LabelCHIScopeIndex
# LabelCHIScopeCycleSize
# LabelCHIScopeCycleIndex
# LabelCHIScopeCycleOffset
# LabelClusterScopeIndex
# LabelClusterScopeCycleSize
# LabelClusterScopeCycleIndex
# LabelClusterScopeCycleOffset
appendScopeLabels: "no"
9 changes: 9 additions & 0 deletions config/secret.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
apiVersion: v1
kind: Secret
metadata:
namespace: dev
name: chop-clickhouse-credentials
type: Opaque
stringData:
username: chuser
password: chpassword
Loading
0