5901-5950 of 10000 results (94ms)
2022-12-06 ยง
14:15 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1004.eqiad.wmnet [production]
14:13 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864919|NewImpact: Adjust hasMainspaceEditsCache check (T324285)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
14:11 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864919|NewImpact: Adjust hasMainspaceEditsCache check (T324285)]] [production]
14:04 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1004.eqiad.wmnet [production]
13:18 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
13:17 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
13:17 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
13:16 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
13:04 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
13:03 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
13:02 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wdqs-all [production]
13:00 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864912|Revert "resourceloader: Modern ES6 code should be forced to target mobile" (T323542)]] (duration: 07m 57s) [production]
12:54 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
12:54 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
12:54 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864912|Revert "resourceloader: Modern ES6 code should be forced to target mobile" (T323542)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
12:52 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wdqs-all [production]
12:52 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864912|Revert "resourceloader: Modern ES6 code should be forced to target mobile" (T323542)]] [production]
12:49 <moritzm> installing glibc security updates on buster [production]
12:43 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public [production]
12:29 <jnuche@deploy1002> Pruned MediaWiki: 1.40.0-wmf.10 (duration: 02m 09s) [production]
12:27 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public [production]
12:27 <jmm@cumin2002> END (FAIL) - Cookbook sre.wdqs.restart-nginx (exit_code=1) rolling restart_daemons on A:wcqs-public [production]
12:27 <jnuche@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 (duration: 05m 52s) [production]
12:21 <jnuche@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 [production]
12:14 <jnuche@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 [production]
12:10 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public [production]
11:22 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry [production]
11:20 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry [production]
10:59 <kostajh> UTC morning deploys done [production]
10:56 <moritzm> installing freetype security updates [production]
10:48 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864910|NewImpact: Show "999+" when we could not count edits/thanks (T324286)]] (duration: 31m 25s) [production]
10:36 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864910|NewImpact: Show "999+" when we could not count edits/thanks (T324286)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
10:16 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864910|NewImpact: Show "999+" when we could not count edits/thanks (T324286)]] [production]
09:37 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864908|User impact: Do not show impact module if user has no mainspace edits (T324285)]] (duration: 28m 05s) [production]
09:11 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864908|User impact: Do not show impact module if user has no mainspace edits (T324285)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
09:09 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864908|User impact: Do not show impact module if user has no mainspace edits (T324285)]] [production]
06:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
06:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
06:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201 (T323907)', diff saved to https://phabricator.wikimedia.org/P42426 and previous config saved to /var/cache/conftool/dbconfig/20221206-064402-ladsgroup.json [production]
06:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P42425 and previous config saved to /var/cache/conftool/dbconfig/20221206-062856-ladsgroup.json [production]
06:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P42424 and previous config saved to /var/cache/conftool/dbconfig/20221206-061349-ladsgroup.json [production]
05:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1201 (T323907)', diff saved to https://phabricator.wikimedia.org/P42423 and previous config saved to /var/cache/conftool/dbconfig/20221206-055843-ladsgroup.json [production]
05:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180 (T323907)', diff saved to https://phabricator.wikimedia.org/P42422 and previous config saved to /var/cache/conftool/dbconfig/20221206-054030-ladsgroup.json [production]
05:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1201 (T323907)', diff saved to https://phabricator.wikimedia.org/P42421 and previous config saved to /var/cache/conftool/dbconfig/20221206-053911-ladsgroup.json [production]
05:39 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
05:38 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
05:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1187 (T323907)', diff saved to https://phabricator.wikimedia.org/P42420 and previous config saved to /var/cache/conftool/dbconfig/20221206-053850-ladsgroup.json [production]
05:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P42419 and previous config saved to /var/cache/conftool/dbconfig/20221206-052523-ladsgroup.json [production]
05:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P42418 and previous config saved to /var/cache/conftool/dbconfig/20221206-052343-ladsgroup.json [production]
05:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P42417 and previous config saved to /var/cache/conftool/dbconfig/20221206-051016-ladsgroup.json [production]