3151-3200 of 10000 results (54ms)
2021-03-09 ยง
10:49 <hnowlan@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:45 <hnowlan@cumin1001> START - Cookbook sre.dns.netbox [production]
10:43 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[2020-2027].codfw.wmnet [production]
10:36 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host prometheus2003.codfw.wmnet [production]
10:31 <moritzm> upgrading perf on stretch hosts [production]
10:30 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host graphite2003.codfw.wmnet [production]
10:23 <moritzm> installing gdisk security updates [production]
10:15 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host graphite2003.codfw.wmnet [production]
10:14 <moritzm> installing libbsd security updates [production]
10:07 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be[2020-2027].codfw.wmnet [production]
10:00 <moritzm> installing busybox security updates [production]
09:54 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1001.eqiad.wmnet [production]
09:52 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host puppetboard1001.eqiad.wmnet [production]
09:50 <marostegui> Reboot db2073 for kernel upgrade (stretch) [production]
09:50 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2001.codfw.wmnet [production]
09:49 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:49 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:46 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host puppetboard2001.codfw.wmnet [production]
09:44 <marostegui> Reboot db2072 for kernel upgrade (stretch) [production]
09:44 <marostegui> Reboot db20712for kernel upgrade (stretch) [production]
09:44 <marostegui> Reboot db2071 for kernel upgrade (stretch) [production]
09:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1112.eqiad.wmnet with reason: REIMAGE [production]
09:40 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1112.eqiad.wmnet with reason: REIMAGE [production]
09:39 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1076.eqiad.wmnet with reason: REIMAGE [production]
09:36 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1076.eqiad.wmnet with reason: REIMAGE [production]
09:14 <elukey> drain + reimage analytics1076 and an-worker1112 to Buster [production]
09:01 <moritzm> installing Linux 4.9.258 updates on Stretch hosts [production]
08:59 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[2017-2019].codfw.wmnet [production]
08:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1093.eqiad.wmnet with reason: REIMAGE [production]
08:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1092.eqiad.wmnet with reason: REIMAGE [production]
08:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1093.eqiad.wmnet with reason: REIMAGE [production]
08:50 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1092.eqiad.wmnet with reason: REIMAGE [production]
08:46 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be[2017-2019].codfw.wmnet [production]
08:46 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
08:46 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
08:33 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
08:27 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
08:26 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be2016.codfw.wmnet [production]
08:12 <marostegui> Stop mysql on clouddb1015:3314, 3316 [production]
07:59 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be2016.codfw.wmnet [production]
07:50 <dcausse> restarted blazegraph on wdqs1004 and depooled it to catchup lag [production]
07:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1095.eqiad.wmnet with reason: REIMAGE [production]
07:38 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1094.eqiad.wmnet with reason: REIMAGE [production]
07:38 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1095.eqiad.wmnet with reason: REIMAGE [production]
07:36 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1094.eqiad.wmnet with reason: REIMAGE [production]
07:24 <godog> swift eqiad-prod: add weight to ms-be106[0-3] - T268435 [production]
07:03 <elukey@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:01 <elukey> drain + reimage an-worker109[4,5] to Buster [production]
06:58 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
06:30 <_joe_> restarting gerrit on gerrit1001, using 48 GB of heap [production]