6301-6350 of 10000 results (96ms)
2023-08-25 §
05:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2140 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P51427 and previous config saved to /var/cache/conftool/dbconfig/20230825-054701-ladsgroup.json [production]
05:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2140 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P51426 and previous config saved to /var/cache/conftool/dbconfig/20230825-053156-ladsgroup.json [production]
05:28 <marostegui> failover m3-master to dbproxy1020 [production]
05:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2140 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P51425 and previous config saved to /var/cache/conftool/dbconfig/20230825-051651-ladsgroup.json [production]
05:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2140 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P51424 and previous config saved to /var/cache/conftool/dbconfig/20230825-050147-ladsgroup.json [production]
2023-08-24 §
23:10 <bblack> geodns: DE+GB mapped back to esams (were temporarily on drmrs) [production]
22:15 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes2025.codfw.wmnet with OS bullseye [production]
21:59 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host kubernetes2025.codfw.wmnet with OS bullseye [production]
21:43 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes2025.codfw.wmnet with OS bullseye [production]
21:43 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host kubernetes2025.codfw.wmnet with OS bullseye [production]
21:38 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes2025.codfw.wmnet with OS bullseye [production]
21:29 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 (duration: 00m 15s) [production]
21:29 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 [production]
21:28 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 (duration: 08m 18s) [production]
21:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T344589)', diff saved to https://phabricator.wikimedia.org/P51422 and previous config saved to /var/cache/conftool/dbconfig/20230824-212554-ladsgroup.json [production]
21:23 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host kubernetes2025.codfw.wmnet with OS bullseye [production]
21:19 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 [production]
21:18 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 (duration: 02m 17s) [production]
21:16 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 [production]
21:15 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: (no justification provided) (duration: 00m 55s) [production]
21:14 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: (no justification provided) [production]
21:14 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: (no justification provided) (duration: 00m 40s) [production]
21:14 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: (no justification provided) [production]
21:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P51421 and previous config saved to /var/cache/conftool/dbconfig/20230824-211048-ladsgroup.json [production]
21:09 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 (duration: 02m 56s) [production]
21:06 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 [production]
21:06 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 (duration: 22m 03s) [production]
21:01 <thcipriani> mwmaint1002:foreachwiki extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --create-system-user # ref. 952132 [production]
20:58 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:952132|Add option to just create the 'Global rename script' system user (T344632)]], [[gerrit:952130|watchlist: Don't assume only named users have watchlist access (T344870)]] (duration: 12m 31s) [production]
20:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P51419 and previous config saved to /var/cache/conftool/dbconfig/20230824-205541-ladsgroup.json [production]
20:52 <thcipriani@deploy1002> thcipriani and jdrewniak and krinkle: Continuing with sync [production]
20:47 <thcipriani@deploy1002> thcipriani and jdrewniak and krinkle: Backport for [[gerrit:952132|Add option to just create the 'Global rename script' system user (T344632)]], [[gerrit:952130|watchlist: Don't assume only named users have watchlist access (T344870)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment ( [production]
20:45 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:952132|Add option to just create the 'Global rename script' system user (T344632)]], [[gerrit:952130|watchlist: Don't assume only named users have watchlist access (T344870)]] [production]
20:43 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: allow list changes T343856 0.3.125 [production]
20:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T344589)', diff saved to https://phabricator.wikimedia.org/P51418 and previous config saved to /var/cache/conftool/dbconfig/20230824-204035-ladsgroup.json [production]
20:37 <bking@deploy1002> Finished deploy [wdqs/wdqs@2455ffd]: (no justification provided) (duration: 04m 41s) [production]
20:34 <inflatador> bking@deploy1002 'scap deploy new wdqs T343856' [production]
20:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1219 (T344589)', diff saved to https://phabricator.wikimedia.org/P51417 and previous config saved to /var/cache/conftool/dbconfig/20230824-203322-ladsgroup.json [production]
20:33 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
20:33 <bking@deploy1002> Started deploy [wdqs/wdqs@2455ffd]: (no justification provided) [production]
20:33 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
20:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance [production]
20:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance [production]
20:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1207 (T344589)', diff saved to https://phabricator.wikimedia.org/P51416 and previous config saved to /var/cache/conftool/dbconfig/20230824-202836-ladsgroup.json [production]
20:21 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:949593|Remove unused RESTBase-related VisualEditor config settings (T341618)]] (duration: 09m 58s) [production]
20:15 <thcipriani@deploy1002> thcipriani and matmarex: Continuing with sync [production]
20:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P51415 and previous config saved to /var/cache/conftool/dbconfig/20230824-201329-ladsgroup.json [production]
20:12 <thcipriani@deploy1002> thcipriani and matmarex: Backport for [[gerrit:949593|Remove unused RESTBase-related VisualEditor config settings (T341618)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:11 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:949593|Remove unused RESTBase-related VisualEditor config settings (T341618)]] [production]
20:03 <effie> enabling puppet on thanos-fe* hosts [production]