2351-2400 of 10000 results (39ms)
2021-03-09 §
10:07 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be[2020-2027].codfw.wmnet [production]
10:00 <moritzm> installing busybox security updates [production]
09:54 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1001.eqiad.wmnet [production]
09:52 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host puppetboard1001.eqiad.wmnet [production]
09:50 <marostegui> Reboot db2073 for kernel upgrade (stretch) [production]
09:50 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2001.codfw.wmnet [production]
09:49 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:49 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:46 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host puppetboard2001.codfw.wmnet [production]
09:44 <marostegui> Reboot db2072 for kernel upgrade (stretch) [production]
09:44 <marostegui> Reboot db20712for kernel upgrade (stretch) [production]
09:44 <marostegui> Reboot db2071 for kernel upgrade (stretch) [production]
09:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1112.eqiad.wmnet with reason: REIMAGE [production]
09:40 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1112.eqiad.wmnet with reason: REIMAGE [production]
09:39 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1076.eqiad.wmnet with reason: REIMAGE [production]
09:36 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1076.eqiad.wmnet with reason: REIMAGE [production]
09:14 <elukey> drain + reimage analytics1076 and an-worker1112 to Buster [production]
09:01 <moritzm> installing Linux 4.9.258 updates on Stretch hosts [production]
08:59 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[2017-2019].codfw.wmnet [production]
08:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1093.eqiad.wmnet with reason: REIMAGE [production]
08:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1092.eqiad.wmnet with reason: REIMAGE [production]
08:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1093.eqiad.wmnet with reason: REIMAGE [production]
08:50 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1092.eqiad.wmnet with reason: REIMAGE [production]
08:46 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be[2017-2019].codfw.wmnet [production]
08:46 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
08:46 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
08:33 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
08:27 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
08:26 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be2016.codfw.wmnet [production]
08:12 <marostegui> Stop mysql on clouddb1015:3314, 3316 [production]
07:59 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be2016.codfw.wmnet [production]
07:50 <dcausse> restarted blazegraph on wdqs1004 and depooled it to catchup lag [production]
07:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1095.eqiad.wmnet with reason: REIMAGE [production]
07:38 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1094.eqiad.wmnet with reason: REIMAGE [production]
07:38 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1095.eqiad.wmnet with reason: REIMAGE [production]
07:36 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1094.eqiad.wmnet with reason: REIMAGE [production]
07:24 <godog> swift eqiad-prod: add weight to ms-be106[0-3] - T268435 [production]
07:03 <elukey@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:01 <elukey> drain + reimage an-worker109[4,5] to Buster [production]
06:58 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
06:30 <_joe_> restarting gerrit on gerrit1001, using 48 GB of heap [production]
06:19 <marostegui> Deploy schema change on s6 codfw (there will be lag on codfw) T276150 T276156 [production]
05:37 <marostegui> Stop mysql on clouddb1014:3312, 3317 to transfer its data to cloudb1021 [production]
05:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1175 for table check T276742', diff saved to https://phabricator.wikimedia.org/P14675 and previous config saved to /var/cache/conftool/dbconfig/20210309-051646-marostegui.json [production]
00:58 <Krinkle> krinkle@mwmaint1002 Ran invalidateUserSesssions.php for one user [production]
00:12 <urbanecm@deploy1002> Synchronized wmf-config/config/incubatorwiki.yaml: 0d260eda2d62ae053310ee978201b1a031522d59: Enable modern Vector on incubator (T275479; 2/2) (duration: 00m 57s) [production]
00:11 <urbanecm@deploy1002> Synchronized dblists/desktop-improvements.dblist: 0d260eda2d62ae053310ee978201b1a031522d59: Enable modern Vector on incubator (T275479; 1/2) (duration: 01m 01s) [production]
00:09 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: ce82e0cd6015b362f9acc8e90d300cf88738cf98: Logo updates (T273085) (duration: 00m 58s) [production]
00:08 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/: ce82e0cd6015b362f9acc8e90d300cf88738cf98: Logo updates (T273085) (duration: 00m 58s) [production]
2021-03-08 §
22:36 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: REIMAGE [production]