4401-4450 of 10000 results (84ms)
2022-12-06 ยง
12:10 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public [production]
11:22 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry [production]
11:20 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry [production]
10:59 <kostajh> UTC morning deploys done [production]
10:56 <moritzm> installing freetype security updates [production]
10:48 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864910|NewImpact: Show "999+" when we could not count edits/thanks (T324286)]] (duration: 31m 25s) [production]
10:36 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864910|NewImpact: Show "999+" when we could not count edits/thanks (T324286)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
10:16 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864910|NewImpact: Show "999+" when we could not count edits/thanks (T324286)]] [production]
09:37 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864908|User impact: Do not show impact module if user has no mainspace edits (T324285)]] (duration: 28m 05s) [production]
09:11 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864908|User impact: Do not show impact module if user has no mainspace edits (T324285)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
09:09 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864908|User impact: Do not show impact module if user has no mainspace edits (T324285)]] [production]
06:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
06:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
06:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201 (T323907)', diff saved to https://phabricator.wikimedia.org/P42426 and previous config saved to /var/cache/conftool/dbconfig/20221206-064402-ladsgroup.json [production]
06:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P42425 and previous config saved to /var/cache/conftool/dbconfig/20221206-062856-ladsgroup.json [production]
06:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P42424 and previous config saved to /var/cache/conftool/dbconfig/20221206-061349-ladsgroup.json [production]
05:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201 (T323907)', diff saved to https://phabricator.wikimedia.org/P42423 and previous config saved to /var/cache/conftool/dbconfig/20221206-055843-ladsgroup.json [production]
05:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180 (T323907)', diff saved to https://phabricator.wikimedia.org/P42422 and previous config saved to /var/cache/conftool/dbconfig/20221206-054030-ladsgroup.json [production]
05:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1201 (T323907)', diff saved to https://phabricator.wikimedia.org/P42421 and previous config saved to /var/cache/conftool/dbconfig/20221206-053911-ladsgroup.json [production]
05:39 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
05:38 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
05:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1187 (T323907)', diff saved to https://phabricator.wikimedia.org/P42420 and previous config saved to /var/cache/conftool/dbconfig/20221206-053850-ladsgroup.json [production]
05:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P42419 and previous config saved to /var/cache/conftool/dbconfig/20221206-052523-ladsgroup.json [production]
05:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P42418 and previous config saved to /var/cache/conftool/dbconfig/20221206-052343-ladsgroup.json [production]
05:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P42417 and previous config saved to /var/cache/conftool/dbconfig/20221206-051016-ladsgroup.json [production]
05:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P42416 and previous config saved to /var/cache/conftool/dbconfig/20221206-050837-ladsgroup.json [production]
04:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180 (T323907)', diff saved to https://phabricator.wikimedia.org/P42415 and previous config saved to /var/cache/conftool/dbconfig/20221206-045510-ladsgroup.json [production]
04:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1187 (T323907)', diff saved to https://phabricator.wikimedia.org/P42414 and previous config saved to /var/cache/conftool/dbconfig/20221206-045330-ladsgroup.json [production]
04:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1187 (T323907)', diff saved to https://phabricator.wikimedia.org/P42413 and previous config saved to /var/cache/conftool/dbconfig/20221206-043348-ladsgroup.json [production]
04:33 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
04:33 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
04:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T323907)', diff saved to https://phabricator.wikimedia.org/P42412 and previous config saved to /var/cache/conftool/dbconfig/20221206-043326-ladsgroup.json [production]
04:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2180 (T323907)', diff saved to https://phabricator.wikimedia.org/P42411 and previous config saved to /var/cache/conftool/dbconfig/20221206-042850-ladsgroup.json [production]
04:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance [production]
04:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance [production]
04:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T323907)', diff saved to https://phabricator.wikimedia.org/P42410 and previous config saved to /var/cache/conftool/dbconfig/20221206-042828-ladsgroup.json [production]
04:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P42409 and previous config saved to /var/cache/conftool/dbconfig/20221206-041820-ladsgroup.json [production]
04:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P42408 and previous config saved to /var/cache/conftool/dbconfig/20221206-041322-ladsgroup.json [production]
04:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P42407 and previous config saved to /var/cache/conftool/dbconfig/20221206-040313-ladsgroup.json [production]
04:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 [production]
03:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P42406 and previous config saved to /var/cache/conftool/dbconfig/20221206-035815-ladsgroup.json [production]
03:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T323907)', diff saved to https://phabricator.wikimedia.org/P42405 and previous config saved to /var/cache/conftool/dbconfig/20221206-034806-ladsgroup.json [production]
03:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T323907)', diff saved to https://phabricator.wikimedia.org/P42404 and previous config saved to /var/cache/conftool/dbconfig/20221206-034309-ladsgroup.json [production]
03:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1180 (T323907)', diff saved to https://phabricator.wikimedia.org/P42403 and previous config saved to /var/cache/conftool/dbconfig/20221206-032818-ladsgroup.json [production]
03:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
03:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
03:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T323907)', diff saved to https://phabricator.wikimedia.org/P42402 and previous config saved to /var/cache/conftool/dbconfig/20221206-032756-ladsgroup.json [production]
03:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P42401 and previous config saved to /var/cache/conftool/dbconfig/20221206-031250-ladsgroup.json [production]
02:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2171:3316 (T323907)', diff saved to https://phabricator.wikimedia.org/P42400 and previous config saved to /var/cache/conftool/dbconfig/20221206-025831-ladsgroup.json [production]
02:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance [production]