3101-3150 of 10000 results (74ms)
2020-03-13 §
17:21 <mutante> removed squid from install1002/install2002 (formerly webproxy.(eqiad|codfw).wmnet until 2 days ago, replaced by install1003/install2003) T224576 [production]
17:20 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) [production]
17:09 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
17:08 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker [production]
17:00 <krinkle@deploy1001> Synchronized dblists/: If4d17082f, Iadba5b01b, Ibe16d5f09 (duration: 01m 07s) [production]
16:58 <krinkle@deploy1001> Synchronized wmf-config/config/: Ibe16d5f09 (duration: 01m 10s) [production]
16:51 <bstorm_> rebooting labstore1007 for stretch upgrade T224583 [production]
16:37 <krinkle@deploy1001> Synchronized wmf-config/config/: If4d17082f, Iadba5b01b (duration: 01m 11s) [production]
16:18 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:15 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:04 <bstorm_> rebooting labstore1007 for first cycle of upgrades T224583 [production]
16:02 <elukey> powercycle kafka-jumbo1006 after switch port changed - T247561 [production]
15:28 <_joe_> switch envoy logging to debug on mw2231 [production]
14:57 <cdanis> T247586 ✔️ cdanis@grafana1002.eqiad.wmnet ~ 🕥☕ sudo systemctl restart apache2.service [production]
12:48 <Urbanecm> Password reset for SUL User:FuduBot (T247601) [production]
12:16 <akosiaris@deploy1001> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 01m 16s) [production]
10:26 <moritzm> installing python-werkzeug security updates [production]
10:09 <vgutierrez> upload trafficserver 8.0.6-1wm3 to apt.wm.o (buster) - T245616 [production]
09:55 <_joe_> running puppet across appservers to switch to http for eventgate-analytics T247484 [production]
09:17 <moritzm> installing perl updates from Stretch point release [production]
06:16 <vgutierrez> triggering OCSP response updates in eqiad,codfw and ulsfo - T247584 [production]
06:12 <vgutierrez> triggering OCSP response updates in eqsin - T247584 [production]
06:05 <vgutierrez> triggering OCSP response updates in esams - T247584 [production]
00:20 <shdubsh> reload prometheus@ops on prometheus1003 [production]
00:08 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw215[8-9].codfw.wmnet [production]
00:08 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw216[0-9].codfw.wmnet [production]
00:08 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw217[1-2].codfw.wmnet [production]
00:04 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
00:04 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-03-12 §
23:58 <shdubsh> reload prometheus@ops on prometheus1004 [production]
23:42 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw217[1-2].codfw.wmnet [production]
23:41 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw216[0-9].codfw.wmnet [production]
23:40 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw215[89].codfw.wmnet [production]
23:26 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw215[89].codfw.wmnet [production]
23:25 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2178.codfw.wmnet [production]
23:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw216[0-6].codfw.wmnet [production]
22:45 <krinkle@deploy1001> Synchronized multiversion/: I403a9890a9 (duration: 01m 07s) [production]
22:44 <krinkle@deploy1001> Synchronized dblists/: I403a9890a9 (duration: 01m 09s) [production]
22:41 <mforns@deploy1001> Finished deploy [analytics/refinery@906bd1e]: deploying refinery together with refinery-source v0.0.118 (duration: 12m 20s) [production]
22:28 <mforns@deploy1001> Started deploy [analytics/refinery@906bd1e]: deploying refinery together with refinery-source v0.0.118 [production]
22:15 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
22:15 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
22:09 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
22:07 <bstorm_> moving all nfs traffic off labstore1007 and to labstore1006 for upgrades T224583 [production]
22:06 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
22:05 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
22:02 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
22:02 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
21:47 <mutante> doc1001 - had to manually run "/usr/local/sbin/build-envoy-config -c /etc/envoy/" to get envoy tls_terminator_443 listener into the config or envoy would not listen on 443 (T210411) [production]
21:19 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]