|
2021-02-22
ยง
|
| 22:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1410.eqiad.wmnet with reason: REIMAGE |
[production] |
| 22:42 |
<krinkle@deploy1001> |
Synchronized w/fatal-error.php: df694d695 (duration: 00m 56s) |
[production] |
| 22:42 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1412.eqiad.wmnet with reason: REIMAGE |
[production] |
| 22:41 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1410.eqiad.wmnet with reason: REIMAGE |
[production] |
| 22:31 |
<ppchelko@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 22:31 |
<ppchelko@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 22:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1279.eqiad.wmnet |
[production] |
| 22:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1312.eqiad.wmnet |
[production] |
| 22:16 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1314.eqiad.wmnet |
[production] |
| 21:46 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1314.eqiad.wmnet |
[production] |
| 21:46 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1279.eqiad.wmnet |
[production] |
| 21:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1312.eqiad.wmnet |
[production] |
| 21:00 |
<Amir1> |
end of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T273463 T271985 T273468) |
[production] |
| 20:59 |
<sbassett> |
Deployed security patch for T274883 |
[production] |
| 20:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1279.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:58 |
<mutante> |
fixed puppet run on deploy-1002 by adding empty array of wikimedia-sites to hiera |
[devtools] |
| 20:57 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1279.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:50 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1312.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:48 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1312.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:46 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1314.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:44 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1314.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:40 |
<bstorm> |
repooled tools-sgeexec-0918.tools.eqiad.wmflabs |
[tools] |
| 20:39 |
<Amir1> |
start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T273463 T271985 T273468) |
[production] |
| 20:29 |
<mutante> |
mw1279 (canary) - reimaging to buster |
[production] |
| 20:29 |
<mutante> |
mw1279 (canary) - reimaging to stretch |
[production] |
| 20:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1349.eqiad.wmnet |
[production] |
| 20:25 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1349.eqiad.wmnet |
[production] |
| 20:19 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1316.eqiad.wmnet |
[production] |
| 20:16 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1316.eqiad.wmnet |
[production] |
| 20:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1315.eqiad.wmnet |
[production] |
| 20:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1315.eqiad.wmnet |
[production] |
| 20:06 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1349.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:04 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1349.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:01 |
<mutante> |
deploy-1002 is broken because mediawiki::sites is not in Hiera (yet) |
[devtools] |
| 19:36 |
<urbanecm@deploy1001> |
Synchronized wmf-config/config/rowiki.yaml: fc7b071b98b2c14d45259212bd6bea858e3f5aa7: Enable GrowthExperiments on rowiki (T275130; 3/3) (duration: 00m 55s) |
[production] |
| 19:35 |
<urbanecm@deploy1001> |
Synchronized dblists/growthexperiments.dblist: fc7b071b98b2c14d45259212bd6bea858e3f5aa7: Enable GrowthExperiments on rowiki (T275130; 2/3) (duration: 00m 55s) |
[production] |
| 19:33 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: fc7b071b98b2c14d45259212bd6bea858e3f5aa7: Enable GrowthExperiments on rowiki (T275130; 1/3) (duration: 00m 55s) |
[production] |
| 19:27 |
<ottomata> |
restart oozie on an-coord1001 to pick up new spark share lib without hadoop jars - T274384 |
[analytics] |
| 19:25 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1315.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:23 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1315.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:22 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1316.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:19 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1316.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:09 |
<bstorm> |
hard rebooted tools-sgeexec-0918 from openstack T275411 |
[tools] |
| 19:08 |
<urbanecm@deploy1001> |
Synchronized dblists/growthexperiments.dblist: 902b6854b5d56fde9fbf5d2c779282049bf7288a: Enable GrowthExperiments on thwiki (T274646) (duration: 00m 54s) |
[production] |
| 19:07 |
<bstorm> |
shutting down tools-sgeexec-0918 with the VM's command line (not libvirt directly yet) T275411 |
[tools] |
| 19:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 902b6854b5d56fde9fbf5d2c779282049bf7288a: Enable GrowthExperiments on thwiki (T274646) (duration: 00m 56s) |
[production] |
| 19:05 |
<bstorm> |
shutting down tools-sgeexec-0918 (with openstack to see what happens) T275411 |
[tools] |
| 19:03 |
<bstorm> |
depooled tools-sgeexec-0918 T275411 |
[tools] |
| 18:56 |
<bstorm> |
deleted job 1962508 from the grid to clear it up T275301 |
[tools] |
| 17:18 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@c5c4b2d] (dev-cluster): remove graphoid (duration: 03m 09s) |
[production] |