|
2020-03-13
§
|
| 22:21 |
<bstorm_> |
downtimed labstore1006 for upgrades T224583 |
[production] |
| 20:02 |
<mutante> |
stat1005 - ip link set en01 down ; ip link set en01 up (T247561) |
[production] |
| 19:30 |
<bstorm_> |
rebooting labstore1007 for upgrade to buster T224583 |
[production] |
| 18:51 |
<shdubsh> |
test increase fs.inotify.max_user_watches on prometheus2004 |
[production] |
| 17:58 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 17:21 |
<mutante> |
removed squid from install1002/install2002 (formerly webproxy.(eqiad|codfw).wmnet until 2 days ago, replaced by install1003/install2003) T224576 |
[production] |
| 17:20 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) |
[production] |
| 17:09 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 17:08 |
<elukey@cumin1001> |
START - Cookbook sre.kafka.roll-restart-mirror-maker |
[production] |
| 17:00 |
<krinkle@deploy1001> |
Synchronized dblists/: If4d17082f, Iadba5b01b, Ibe16d5f09 (duration: 01m 07s) |
[production] |
| 16:58 |
<krinkle@deploy1001> |
Synchronized wmf-config/config/: Ibe16d5f09 (duration: 01m 10s) |
[production] |
| 16:51 |
<bstorm_> |
rebooting labstore1007 for stretch upgrade T224583 |
[production] |
| 16:37 |
<krinkle@deploy1001> |
Synchronized wmf-config/config/: If4d17082f, Iadba5b01b (duration: 01m 11s) |
[production] |
| 16:18 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:15 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:04 |
<bstorm_> |
rebooting labstore1007 for first cycle of upgrades T224583 |
[production] |
| 16:02 |
<elukey> |
powercycle kafka-jumbo1006 after switch port changed - T247561 |
[production] |
| 15:28 |
<_joe_> |
switch envoy logging to debug on mw2231 |
[production] |
| 14:57 |
<cdanis> |
T247586 ✔️ cdanis@grafana1002.eqiad.wmnet ~ 🕥☕ sudo systemctl restart apache2.service |
[production] |
| 12:48 |
<Urbanecm> |
Password reset for SUL User:FuduBot (T247601) |
[production] |
| 12:16 |
<akosiaris@deploy1001> |
Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 01m 16s) |
[production] |
| 10:26 |
<moritzm> |
installing python-werkzeug security updates |
[production] |
| 10:09 |
<vgutierrez> |
upload trafficserver 8.0.6-1wm3 to apt.wm.o (buster) - T245616 |
[production] |
| 09:55 |
<_joe_> |
running puppet across appservers to switch to http for eventgate-analytics T247484 |
[production] |
| 09:17 |
<moritzm> |
installing perl updates from Stretch point release |
[production] |
| 06:16 |
<vgutierrez> |
triggering OCSP response updates in eqiad,codfw and ulsfo - T247584 |
[production] |
| 06:12 |
<vgutierrez> |
triggering OCSP response updates in eqsin - T247584 |
[production] |
| 06:05 |
<vgutierrez> |
triggering OCSP response updates in esams - T247584 |
[production] |
| 00:20 |
<shdubsh> |
reload prometheus@ops on prometheus1003 |
[production] |
| 00:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw215[8-9].codfw.wmnet |
[production] |
| 00:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw216[0-9].codfw.wmnet |
[production] |
| 00:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw217[1-2].codfw.wmnet |
[production] |
| 00:04 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:04 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
|
2020-03-12
§
|
| 23:58 |
<shdubsh> |
reload prometheus@ops on prometheus1004 |
[production] |
| 23:42 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw217[1-2].codfw.wmnet |
[production] |
| 23:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw216[0-9].codfw.wmnet |
[production] |
| 23:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw215[89].codfw.wmnet |
[production] |
| 23:26 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw215[89].codfw.wmnet |
[production] |
| 23:25 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2178.codfw.wmnet |
[production] |
| 23:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw216[0-6].codfw.wmnet |
[production] |
| 22:45 |
<krinkle@deploy1001> |
Synchronized multiversion/: I403a9890a9 (duration: 01m 07s) |
[production] |
| 22:44 |
<krinkle@deploy1001> |
Synchronized dblists/: I403a9890a9 (duration: 01m 09s) |
[production] |
| 22:41 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@906bd1e]: deploying refinery together with refinery-source v0.0.118 (duration: 12m 20s) |
[production] |
| 22:28 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@906bd1e]: deploying refinery together with refinery-source v0.0.118 |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 22:09 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:07 |
<bstorm_> |
moving all nfs traffic off labstore1007 and to labstore1006 for upgrades T224583 |
[production] |
| 22:06 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |