851-900 of 10000 results (27ms)
2021-07-15 ยง
13:03 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on db1165.eqiad.wmnet with reason: Rebooting for T273281 [production]
13:02 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1165 (s6 sanitarium master) for kernel upgrade T273281 [production]
13:02 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1165 (s6 sanitarium master) for kernel upgrade T273281 [production]
12:55 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T273281 [production]
12:55 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T273281 [production]
12:54 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw142[3-5].eqiad.wmnet [production]
12:54 <mutante> mw1423, mw1424, mw1425 - pooled as new API servers [production]
12:54 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw142[3-5].eqiad.wmnet [production]
12:53 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw142[3-5].eqiad.wmnet [production]
12:51 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw142[3-5].eqiad.wmnet [production]
12:41 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2142.codfw.wmnet with reason: Rebooting db1103 (x1 primary) for kernel upgrade T273281 [production]
12:41 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on db2142.codfw.wmnet with reason: Rebooting db1103 (x1 primary) for kernel upgrade T273281 [production]
12:40 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1103.eqiad.wmnet with reason: Rebooting for T273281 [production]
12:40 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on db1103.eqiad.wmnet with reason: Rebooting for T273281 [production]
12:40 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db[1102-1103,1120,1137].eqiad.wmnet,dbstore1005.eqiad.wmnet with reason: Rebooting db1103 (x1 primary) for kernel upgrade T273281 [production]
12:40 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on db[1102-1103,1120,1137].eqiad.wmnet,dbstore1005.eqiad.wmnet with reason: Rebooting db1103 (x1 primary) for kernel upgrade T273281 [production]
12:35 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1103.eqiad.wmnet with reason: Rebooting db1103 for kernel upgrade T273281 [production]
12:35 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db1103.eqiad.wmnet with reason: Rebooting db1103 for kernel upgrade T273281 [production]
12:34 <mutante> mw1423, mw1424, mw1425 - scap pull [production]
12:31 <dcausse@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . [production]
12:09 <mutante> mw1423,mw1424,mw1425 - rebooting [production]
11:48 <moritzm> restarting restbase1028-1030 to pick up libuv security update [production]
11:47 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw[1423-1425].eqiad.wmnet with reason: new host [production]
11:47 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw[1423-1425].eqiad.wmnet with reason: new host [production]
11:47 <mutante> mw1423, mw1424, mw1425 - initial puppet run, new API appservers going into production [production]
11:43 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:704527|Make idwiki use protect mode of flaggedrevs (T268317)]] (duration: 01m 07s) [production]
11:40 <moritzm> restarting Etherpad to pick up libuv security update [production]
11:37 <moritzm> restarting Turnilo to pick up libuv security update [production]
11:34 <moritzm> installing libuv1 security updates [production]
11:09 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 10 hosts [production]
11:09 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 10 hosts [production]
11:05 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on thanos-fe2001.codfw.wmnet with reason: Extending downtime post-reimage [production]
11:05 <volans@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on thanos-fe2001.codfw.wmnet with reason: Extending downtime post-reimage [production]
10:56 <volans> commented out cron-spam entries on thanos-fe2001, puppet is disabled, thanos-store.service fails to start - T285835 [production]
10:41 <godog> move wikibase.queryService.ui.app to wikibase.queryService.ui.index.app - T272128 [production]
10:34 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'sync'. [production]
10:34 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/admin 'sync'. [production]
10:33 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'sync'. [production]
10:32 <jayme@deploy1002> helmfile [codfw] START helmfile.d/admin 'sync'. [production]
10:32 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
10:31 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
10:26 <jmm@cumin2002> END (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Muehlenhoff out of all services on: 10 hosts [production]
10:26 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 10 hosts [production]
10:06 <dcausse@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . [production]
10:02 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:02 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [production]
10:02 <effie> disableing puppet on maps* for 704394 [production]
09:38 <dcausse@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . [production]
09:34 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2001.codfw.wmnet with reason: REIMAGE [production]
09:32 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2001.codfw.wmnet with reason: REIMAGE [production]