401-450 of 10000 results (79ms)
2024-04-24 §
09:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2145 (T352010)', diff saved to https://phabricator.wikimedia.org/P61137 and previous config saved to /var/cache/conftool/dbconfig/20240424-094004-ladsgroup.json [production]
09:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1248 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61136 and previous config saved to /var/cache/conftool/dbconfig/20240424-093859-arnaudb.json [production]
09:33 <taavi@deploy1002> taavi: Continuing with sync [production]
09:32 <taavi@deploy1002> taavi: Backport for [[gerrit:1023812|logos: Update cawiki 750k logo tagline (T363057)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:29 <taavi@deploy1002> Started scap: Backport for [[gerrit:1023812|logos: Update cawiki 750k logo tagline (T363057)]] [production]
09:29 <claime> 80% of external traffix to mw-on-k8s - T362323 [production]
09:28 <arnaudb@cumin1002> START - Cookbook sre.hosts.reimage for host db1247.eqiad.wmnet with OS bookworm [production]
09:25 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1247', diff saved to https://phabricator.wikimedia.org/P61135 and previous config saved to /var/cache/conftool/dbconfig/20240424-092540-arnaudb.json [production]
09:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P61134 and previous config saved to /var/cache/conftool/dbconfig/20240424-092457-ladsgroup.json [production]
09:24 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1247.eqiad.wmnet with reason: T362746 [production]
09:24 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1247.eqiad.wmnet with reason: T362746 [production]
09:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1248 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61133 and previous config saved to /var/cache/conftool/dbconfig/20240424-092353-arnaudb.json [production]
09:14 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1248.eqiad.wmnet with OS bookworm [production]
09:09 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P61132 and previous config saved to /var/cache/conftool/dbconfig/20240424-090950-ladsgroup.json [production]
09:08 <elukey> run 'kill `pgrep -u dbad2021`' on all stat nodes to unblock puppet [production]
08:55 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1248.eqiad.wmnet with reason: host reimage [production]
08:54 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
08:54 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2145 (T352010)', diff saved to https://phabricator.wikimedia.org/P61131 and previous config saved to /var/cache/conftool/dbconfig/20240424-085442-ladsgroup.json [production]
08:54 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
08:54 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
08:54 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
08:53 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
08:53 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
08:53 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
08:53 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
08:52 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1248.eqiad.wmnet with reason: host reimage [production]
08:52 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-eqiad [production]
08:51 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-eqiad [production]
08:47 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-codfw [production]
08:46 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-codfw [production]
08:39 <arnaudb@cumin1002> START - Cookbook sre.hosts.reimage for host db1248.eqiad.wmnet with OS bookworm [production]
08:37 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1248', diff saved to https://phabricator.wikimedia.org/P61130 and previous config saved to /var/cache/conftool/dbconfig/20240424-083736-arnaudb.json [production]
08:36 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1248.eqiad.wmnet with reason: T362746 [production]
08:36 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1248.eqiad.wmnet with reason: T362746 [production]
08:08 <filippo@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
08:08 <filippo@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
06:17 <moritzm> installing glibc security updates [production]
04:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2152 (T352010)', diff saved to https://phabricator.wikimedia.org/P61129 and previous config saved to /var/cache/conftool/dbconfig/20240424-045230-ladsgroup.json [production]
04:52 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance [production]
04:52 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance [production]
00:14 <zabe@deploy1002> Finished scap: Backport for [[gerrit:1023550|Set timezones for new wikis (T360310 T360303 T363263 T363256 T363250 T363243 T363270)]], [[gerrit:1023528|Update interwiki cache]] (duration: 13m 56s) [production]
00:04 <zabe@deploy1002> zabe: Continuing with sync [production]
00:03 <zabe@deploy1002> zabe: Backport for [[gerrit:1023550|Set timezones for new wikis (T360310 T360303 T363263 T363256 T363250 T363243 T363270)]], [[gerrit:1023528|Update interwiki cache]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
00:01 <zabe@deploy1002> Started scap: Backport for [[gerrit:1023550|Set timezones for new wikis (T360310 T360303 T363263 T363256 T363250 T363243 T363270)]], [[gerrit:1023528|Update interwiki cache]] [production]
2024-04-23 §
23:58 <eileen> config revision changed from 75af3eb6 to dc53becd [production]
23:57 <zabe> zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=kawikisource --cluster=all 2>&1 | tee /tmp/kawikisource.UpdateSearchIndexConfig.log # T363085 [production]
23:56 <zabe@deploy1002> Finished scap: Creating kawikisource (T363085) (duration: 14m 40s) [production]
23:45 <zabe@deploy1002> zabe: Continuing with sync [production]
23:44 <zabe@deploy1002> zabe: Creating kawikisource (T363085) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
23:42 <zabe@deploy1002> Started scap: Creating kawikisource (T363085) [production]