8301-8350 of 10000 results (113ms)
2024-04-20 §
00:39 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T352010)', diff saved to https://phabricator.wikimedia.org/P61032 and previous config saved to /var/cache/conftool/dbconfig/20240420-003927-ladsgroup.json [production]
00:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P61031 and previous config saved to /var/cache/conftool/dbconfig/20240420-002420-ladsgroup.json [production]
00:09 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P61030 and previous config saved to /var/cache/conftool/dbconfig/20240420-000912-ladsgroup.json [production]
2024-04-19 §
23:54 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T352010)', diff saved to https://phabricator.wikimedia.org/P61029 and previous config saved to /var/cache/conftool/dbconfig/20240419-235405-ladsgroup.json [production]
22:30 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
22:30 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
21:03 <taavi> taavi@mwmaint1002 ~ $ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki Kou.i5h 'Renamed user 8356771833137' # T362942 [production]
21:02 <taavi> taavi@mwmaint1002 ~ $ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=eowiki --logwiki=metawiki 'Gzsimonfbi' 'Renamed user 2409354752759' # T362941 [production]
20:22 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
20:21 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
20:12 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
20:12 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
19:56 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T362508, journal in uncertain state) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards [production]
19:51 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
19:49 <jforrester@deploy1002> Finished deploy [integration/docroot@c090350]: I1c1c2564d5e78483c766f77ae4c4c74b14578493 trivial CI fix (duration: 00m 06s) [production]
19:49 <jforrester@deploy1002> Started deploy [integration/docroot@c090350]: I1c1c2564d5e78483c766f77ae4c4c74b14578493 trivial CI fix [production]
19:41 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
19:15 <ryankemper> [WDQS] T363004 Restarted wdqs2012 to clear out its in-application-memory ban lists (it had pybal's twisted user agent banned) [production]
18:50 <ryankemper> [WDQS] T363004 Restarted wdqs2010 and wdqs2024 to clear out their in-application-memory ban lists [production]
18:34 <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T362508, journal in uncertain state) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards [production]
18:33 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:33 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding more reverse v6 INCLUDES into dns for magru transport links - cmooney@cumin1002" [production]
18:32 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding more reverse v6 INCLUDES into dns for magru transport links - cmooney@cumin1002" [production]
18:24 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
18:08 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6 with reason: disabling oob link on mr1-ulsfo to stop the SSH attempts long enough to get a homer run in [production]
18:07 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 0:20:00 on mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6 with reason: disabling oob link on mr1-ulsfo to stop the SSH attempts long enough to get a homer run in [production]
17:58 <sukhe> sudo cookbook -d sre.dns.netbox "test" [production]
17:49 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
17:48 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
16:02 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
16:01 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
16:01 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
16:01 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
16:01 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
16:01 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
16:01 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
16:00 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
15:48 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
15:44 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance [production]
15:44 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance [production]
15:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235 (T352010)', diff saved to https://phabricator.wikimedia.org/P61028 and previous config saved to /var/cache/conftool/dbconfig/20240419-154430-ladsgroup.json [production]
15:35 <cmooney@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [production]
15:35 <cmooney@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [production]
15:35 <cmooney@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [production]
15:35 <cmooney@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [production]
15:29 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P61027 and previous config saved to /var/cache/conftool/dbconfig/20240419-152922-ladsgroup.json [production]
15:14 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P61026 and previous config saved to /var/cache/conftool/dbconfig/20240419-151415-ladsgroup.json [production]
15:11 <vgutierrez> repool ncredir2001 [production]
14:59 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235 (T352010)', diff saved to https://phabricator.wikimedia.org/P61025 and previous config saved to /var/cache/conftool/dbconfig/20240419-145907-ladsgroup.json [production]
14:27 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1183 (T352010)', diff saved to https://phabricator.wikimedia.org/P61023 and previous config saved to /var/cache/conftool/dbconfig/20240419-142726-ladsgroup.json [production]