|
2020-03-13
§
|
| 16:15 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:04 |
<bstorm_> |
rebooting labstore1007 for first cycle of upgrades T224583 |
[production] |
| 16:02 |
<elukey> |
powercycle kafka-jumbo1006 after switch port changed - T247561 |
[production] |
| 15:28 |
<_joe_> |
switch envoy logging to debug on mw2231 |
[production] |
| 14:57 |
<cdanis> |
T247586 ✔️ cdanis@grafana1002.eqiad.wmnet ~ 🕥☕ sudo systemctl restart apache2.service |
[production] |
| 12:48 |
<Urbanecm> |
Password reset for SUL User:FuduBot (T247601) |
[production] |
| 12:16 |
<akosiaris@deploy1001> |
Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 01m 16s) |
[production] |
| 10:26 |
<moritzm> |
installing python-werkzeug security updates |
[production] |
| 10:09 |
<vgutierrez> |
upload trafficserver 8.0.6-1wm3 to apt.wm.o (buster) - T245616 |
[production] |
| 09:55 |
<_joe_> |
running puppet across appservers to switch to http for eventgate-analytics T247484 |
[production] |
| 09:17 |
<moritzm> |
installing perl updates from Stretch point release |
[production] |
| 06:16 |
<vgutierrez> |
triggering OCSP response updates in eqiad,codfw and ulsfo - T247584 |
[production] |
| 06:12 |
<vgutierrez> |
triggering OCSP response updates in eqsin - T247584 |
[production] |
| 06:05 |
<vgutierrez> |
triggering OCSP response updates in esams - T247584 |
[production] |
| 00:20 |
<shdubsh> |
reload prometheus@ops on prometheus1003 |
[production] |
| 00:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw215[8-9].codfw.wmnet |
[production] |
| 00:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw216[0-9].codfw.wmnet |
[production] |
| 00:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw217[1-2].codfw.wmnet |
[production] |
| 00:04 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:04 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
|
2020-03-12
§
|
| 23:58 |
<shdubsh> |
reload prometheus@ops on prometheus1004 |
[production] |
| 23:42 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw217[1-2].codfw.wmnet |
[production] |
| 23:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw216[0-9].codfw.wmnet |
[production] |
| 23:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw215[89].codfw.wmnet |
[production] |
| 23:26 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw215[89].codfw.wmnet |
[production] |
| 23:25 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2178.codfw.wmnet |
[production] |
| 23:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw216[0-6].codfw.wmnet |
[production] |
| 22:45 |
<krinkle@deploy1001> |
Synchronized multiversion/: I403a9890a9 (duration: 01m 07s) |
[production] |
| 22:44 |
<krinkle@deploy1001> |
Synchronized dblists/: I403a9890a9 (duration: 01m 09s) |
[production] |
| 22:41 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@906bd1e]: deploying refinery together with refinery-source v0.0.118 (duration: 12m 20s) |
[production] |
| 22:28 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@906bd1e]: deploying refinery together with refinery-source v0.0.118 |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 22:09 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:07 |
<bstorm_> |
moving all nfs traffic off labstore1007 and to labstore1006 for upgrades T224583 |
[production] |
| 22:06 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:05 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 22:02 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:02 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 21:47 |
<mutante> |
doc1001 - had to manually run "/usr/local/sbin/build-envoy-config -c /etc/envoy/" to get envoy tls_terminator_443 listener into the config or envoy would not listen on 443 (T210411) |
[production] |
| 21:19 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 21:19 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 21:06 |
<foks> |
remove one file for legal compliance |
[production] |
| 20:49 |
<ottomata> |
kafka-jumbo1006 - stopping kafka and powercycling - T247561 |
[production] |
| 20:15 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "all wikis to 1.35.0-wmf.23" |
[production] |
| 20:11 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.23 |
[production] |
| 20:10 |
<mutante> |
revoking puppet cert for doc.discovery.wmnet, re-creating with doc.wikimedia.org as SAN |
[production] |
| 20:09 |
<eileen> |
civicrm revision changed from a301076871 to a1b2cbeac1, config revision is 37232d8460 |
[production] |
| 19:46 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Set term store to WRITE_BOTH for all of Wikidata", take II (duration: 01m 06s) |
[production] |
| 19:45 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Set term store to WRITE_BOTH for all of Wikidata" (duration: 01m 08s) |
[production] |