1551-1600 of 10000 results (32ms)
2021-03-09 §
09:40 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1112.eqiad.wmnet with reason: REIMAGE [production]
09:39 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1076.eqiad.wmnet with reason: REIMAGE [production]
09:36 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1076.eqiad.wmnet with reason: REIMAGE [production]
09:14 <elukey> drain + reimage analytics1076 and an-worker1112 to Buster [production]
09:01 <moritzm> installing Linux 4.9.258 updates on Stretch hosts [production]
08:59 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[2017-2019].codfw.wmnet [production]
08:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1093.eqiad.wmnet with reason: REIMAGE [production]
08:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1092.eqiad.wmnet with reason: REIMAGE [production]
08:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1093.eqiad.wmnet with reason: REIMAGE [production]
08:50 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1092.eqiad.wmnet with reason: REIMAGE [production]
08:46 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be[2017-2019].codfw.wmnet [production]
08:46 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
08:46 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
08:33 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
08:27 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
08:26 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be2016.codfw.wmnet [production]
08:12 <marostegui> Stop mysql on clouddb1015:3314, 3316 [production]
07:59 <filippo@cumin1001> START - Cookbook sre.hosts.decommission for hosts ms-be2016.codfw.wmnet [production]
07:50 <dcausse> restarted blazegraph on wdqs1004 and depooled it to catchup lag [production]
07:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1095.eqiad.wmnet with reason: REIMAGE [production]
07:38 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1094.eqiad.wmnet with reason: REIMAGE [production]
07:38 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1095.eqiad.wmnet with reason: REIMAGE [production]
07:36 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1094.eqiad.wmnet with reason: REIMAGE [production]
07:24 <godog> swift eqiad-prod: add weight to ms-be106[0-3] - T268435 [production]
07:03 <elukey@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:01 <elukey> drain + reimage an-worker109[4,5] to Buster [production]
06:58 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
06:30 <_joe_> restarting gerrit on gerrit1001, using 48 GB of heap [production]
06:19 <marostegui> Deploy schema change on s6 codfw (there will be lag on codfw) T276150 T276156 [production]
05:37 <marostegui> Stop mysql on clouddb1014:3312, 3317 to transfer its data to cloudb1021 [production]
05:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1175 for table check T276742', diff saved to https://phabricator.wikimedia.org/P14675 and previous config saved to /var/cache/conftool/dbconfig/20210309-051646-marostegui.json [production]
00:58 <Krinkle> krinkle@mwmaint1002 Ran invalidateUserSesssions.php for one user [production]
00:12 <urbanecm@deploy1002> Synchronized wmf-config/config/incubatorwiki.yaml: 0d260eda2d62ae053310ee978201b1a031522d59: Enable modern Vector on incubator (T275479; 2/2) (duration: 00m 57s) [production]
00:11 <urbanecm@deploy1002> Synchronized dblists/desktop-improvements.dblist: 0d260eda2d62ae053310ee978201b1a031522d59: Enable modern Vector on incubator (T275479; 1/2) (duration: 01m 01s) [production]
00:09 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: ce82e0cd6015b362f9acc8e90d300cf88738cf98: Logo updates (T273085) (duration: 00m 58s) [production]
00:08 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/: ce82e0cd6015b362f9acc8e90d300cf88738cf98: Logo updates (T273085) (duration: 00m 58s) [production]
2021-03-08 §
22:36 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: REIMAGE [production]
22:34 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: REIMAGE [production]
21:42 <mholloway-shell@deploy1002> Synchronized wmf-config/CommonSettings.php: WikimediaEvents: Create data QA group/right on testwiki (T276515) (duration: 00m 57s) [production]
21:18 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate Editing schemas to Event Platform on all wikis - T267343, T267353 (duration: 00m 58s) [production]
21:04 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate Editing schemas to Event Platform on testwiki, take 2 - T267343, T267353 (duration: 00m 58s) [production]
20:53 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1227e2ad8d14e5f0a10a1050e0fadbe0d3c3e238: idwiki: Growth features: Add mentorlist (T259024) (duration: 00m 58s) [production]
20:44 <legoktm> legoktm@registry1004:~$ sudo systemctl reset-failed # to fix icinga warning [production]
20:43 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: REIMAGE [production]
20:41 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: REIMAGE [production]
20:38 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5ce7b4602d2b109adfb86bef6795a4d07a1208b9: Set wgGEHelpPanelAskMentor to true by default (T275908) (duration: 01m 07s) [production]
20:32 <bblack> miscweb[12]002 - re-enabled puppet and deployed new cert [production]
20:23 <bblack> miscweb[12]002 - disabling puppet to remake cergen cert... [production]
19:55 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate Editing schemas to Event Platform on testwiki - T267343, T267353 (duration: 00m 57s) [production]
19:47 <dduvall@deploy1002> Synchronized php-1.36.0-wmf.33/maintenance/: maintenance: aa6f291: 4893ddb: fa97162: 380c448: DB_NONE offline maintenance improvements (duration: 00m 58s) [production]