1101-1150 of 10000 results (94ms)
2023-11-20 ยง
10:56 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management records for ganeti103[5-8] - T349925 - volans@cumin1001" [production]
10:55 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management records for ganeti103[5-8] - T349925 - volans@cumin1001" [production]
10:52 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
10:50 <klausman@cumin1001> START - Cookbook sre.puppet.migrate-role for role: etcd::v3::ml_etcd [production]
10:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 100%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53620 and previous config saved to /var/cache/conftool/dbconfig/20231120-102327-arnaudb.json [production]
10:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 100%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53619 and previous config saved to /var/cache/conftool/dbconfig/20231120-102303-arnaudb.json [production]
10:22 <oblivian@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
10:22 <oblivian@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
10:13 <klausman@cumin1001> END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for ml-serve1008.eqiad.wmnet: Renew puppet certificate - klausman@cumin1001 [production]
10:13 <klausman@cumin1001> START - Cookbook sre.puppet.renew-cert for ml-serve1008.eqiad.wmnet: Renew puppet certificate - klausman@cumin1001 [production]
10:12 <klausman@cumin1001> END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for ml-serve1008.eqiad.wmnet: Renew puppet certificate - klausman@cumin1001 [production]
10:12 <klausman@cumin1001> START - Cookbook sre.puppet.renew-cert for ml-serve1008.eqiad.wmnet: Renew puppet certificate - klausman@cumin1001 [production]
10:08 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 90%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53618 and previous config saved to /var/cache/conftool/dbconfig/20231120-100823-arnaudb.json [production]
10:07 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 90%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53617 and previous config saved to /var/cache/conftool/dbconfig/20231120-100758-arnaudb.json [production]
10:05 <arnaudb@cumin1001> START - Cookbook sre.mysql.clone of db1143.eqiad.wmnet onto db1243.eqiad.wmnet [production]
10:02 <arnaudb@cumin1001> dbctl commit (dc=all): 'T344036 add db1243', diff saved to https://phabricator.wikimedia.org/P53616 and previous config saved to /var/cache/conftool/dbconfig/20231120-100212-arnaudb.json [production]
10:00 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: provisionning db1243.eqiad.wmnet - T344036 [production]
10:00 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: provisionning db1243.eqiad.wmnet - T344036 [production]
10:00 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: provisionning db1243.eqiad.wmnet - T344036 [production]
10:00 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: provisionning db1243.eqiad.wmnet - T344036 [production]
09:53 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 75%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53615 and previous config saved to /var/cache/conftool/dbconfig/20231120-095318-arnaudb.json [production]
09:52 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 75%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53614 and previous config saved to /var/cache/conftool/dbconfig/20231120-095253-arnaudb.json [production]
09:50 <Emperor> restart swift_dispersion_stats on thanos-fe1001 [production]
09:41 <godog> add 50G to prometheus/k8s in codfw [production]
09:39 <godog> add 50G to prometheus/services in eqiad [production]
09:38 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 60%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53613 and previous config saved to /var/cache/conftool/dbconfig/20231120-093813-arnaudb.json [production]
09:37 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 60%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53612 and previous config saved to /var/cache/conftool/dbconfig/20231120-093748-arnaudb.json [production]
09:34 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner2004.codfw.wmnet with OS bullseye [production]
09:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 45%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53611 and previous config saved to /var/cache/conftool/dbconfig/20231120-092308-arnaudb.json [production]
09:22 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 45%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53610 and previous config saved to /var/cache/conftool/dbconfig/20231120-092243-arnaudb.json [production]
09:18 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner2004.codfw.wmnet with reason: host reimage [production]
09:15 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner2004.codfw.wmnet with reason: host reimage [production]
09:08 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 30%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53609 and previous config saved to /var/cache/conftool/dbconfig/20231120-090803-arnaudb.json [production]
09:07 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 30%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53608 and previous config saved to /var/cache/conftool/dbconfig/20231120-090738-arnaudb.json [production]
09:00 <jelto@cumin1001> START - Cookbook sre.hosts.reimage for host gitlab-runner2004.codfw.wmnet with OS bullseye [production]
08:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1210 (re)pooling @ 100%: Testing 10.4.32', diff saved to https://phabricator.wikimedia.org/P53607 and previous config saved to /var/cache/conftool/dbconfig/20231120-085636-root.json [production]
08:54 <XioNoX> Refresh client certificate for central logging on pfw's - T351110 [production]
08:52 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 15%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53606 and previous config saved to /var/cache/conftool/dbconfig/20231120-085258-arnaudb.json [production]
08:52 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 15%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53605 and previous config saved to /var/cache/conftool/dbconfig/20231120-085233-arnaudb.json [production]
08:37 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 10%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53603 and previous config saved to /var/cache/conftool/dbconfig/20231120-083753-arnaudb.json [production]
08:37 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 10%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53602 and previous config saved to /var/cache/conftool/dbconfig/20231120-083729-arnaudb.json [production]
08:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1210 (re)pooling @ 50%: Testing 10.4.32', diff saved to https://phabricator.wikimedia.org/P53601 and previous config saved to /var/cache/conftool/dbconfig/20231120-082625-root.json [production]
08:22 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1242 (re)pooling @ 5%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53600 and previous config saved to /var/cache/conftool/dbconfig/20231120-082248-arnaudb.json [production]
08:22 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1241 (re)pooling @ 5%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53599 and previous config saved to /var/cache/conftool/dbconfig/20231120-082224-arnaudb.json [production]
08:18 <kartik@deploy2002> Finished scap: Backport for [[gerrit:973170|testwiki: Enable the Unified Content Translation Dashboard (T337915)]] (duration: 11m 49s) [production]
08:13 <kartik@deploy2002> kartik: Continuing with sync [production]
08:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1210 (re)pooling @ 25%: Testing 10.4.32', diff saved to https://phabricator.wikimedia.org/P53598 and previous config saved to /var/cache/conftool/dbconfig/20231120-081120-root.json [production]
08:10 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
08:09 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: sync [production]
08:09 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/changeprop: sync [production]