2051-2100 of 10000 results (61ms)
2020-04-02 ยง
09:28 <oblivian@deploy1001> Started deploy [docker-pkg/deploy@4f86d77]: (no justification provided) [production]
08:51 <marostegui> Deploy schema change db1104 [production]
08:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1104 for schema change', diff saved to https://phabricator.wikimedia.org/P10854 and previous config saved to /var/cache/conftool/dbconfig/20200402-085057-marostegui.json [production]
08:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P10853 and previous config saved to /var/cache/conftool/dbconfig/20200402-085019-marostegui.json [production]
08:28 <gehel> repooling wdqs1006 - catched up on lag [production]
08:22 <vgutierrez> Enable inbound TLSv1.3 in upload@esams - T170567 [production]
08:21 <vgutierrez> Enable TLS Session tickets in esams - T245616 [production]
07:45 <moritzm> bounced ferm on ms-be1040 [production]
07:27 <marostegui> Deploy schema change on db1092 [production]
07:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 for schema change', diff saved to https://phabricator.wikimedia.org/P10850 and previous config saved to /var/cache/conftool/dbconfig/20200402-072730-marostegui.json [production]
07:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1101:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10849 and previous config saved to /var/cache/conftool/dbconfig/20200402-072500-marostegui.json [production]
05:49 <marostegui> Deploy schema change on db1101:3318 [production]
05:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10848 and previous config saved to /var/cache/conftool/dbconfig/20200402-054931-marostegui.json [production]
05:29 <elukey> powercycle analytics1045 (host not responsive to ssh, weird chars showed in mgmt serial console) [production]
2020-04-01 ยง
22:44 <volker-e@deploy1001> Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s) [production]
22:43 <volker-e@deploy1001> Started deploy [design/style-guide@4bfe647]: Deploy design/style-guide: [production]
22:02 <volans> forcing logrotate on netflow2001 to compress yesterday's logs [production]
21:53 <volans> force-rebooting ms-be1023, unresponsive - T249174 [production]
21:50 <volans> stopped and restarted kafkatee-webrequest.service on netflow2001, was in a restart loop [production]
19:48 <marxarelli> rollback of 1.35.0-wmf.26 from group1 (T247773). blocked by T249162 [production]
19:30 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: rollback 1.35.0-wmf.26 from group1 [production]
19:21 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.26 (duration: 01m 06s) [production]
19:20 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.26 [production]
19:18 <marxarelli> promoting group1 to 1.35.0-wmf.26 to group1 [production]
17:21 <cdanis> โœ”๏ธ cdanis@cumin1001.eqiad.wmnet ~ ๐Ÿ•โ˜• homer 'cr*eqord*' commit 'enable sampling on eqord Iac15379cc' [production]
16:54 <cdanis> โœ”๏ธ cdanis@cumin1001.eqiad.wmnet ~ ๐Ÿ•โ˜• homer 'cr*eqdfw*' commit 'enable sampling on eqdfw Iac15379cc' [production]
16:39 <vgutierrez> pool cp2027 - T248816 [production]
16:31 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:28 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:17 <ariel@deploy1001> Finished deploy [dumps/dumps@21363c1]: page range prefetch fixup (duration: 00m 09s) [production]
16:17 <ariel@deploy1001> Started deploy [dumps/dumps@21363c1]: page range prefetch fixup [production]
15:33 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
15:31 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:31 <vgutierrez@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
15:31 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:29 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:29 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:27 <vgutierrez> depool & decommission cp20[16,19,23,27] - T249125 [production]
15:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1099:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10845 and previous config saved to /var/cache/conftool/dbconfig/20200401-152258-marostegui.json [production]
15:11 <herron> performing kafka-main rolling restarts to pick up security updates [production]
14:52 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:50 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:49 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:49 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:46 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:46 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:43 <vgutierrez> depool && decommission cp[2018,2020,2022,2024-2026].codfw.wmnet - T249115 [production]
14:32 <gehel> depooling wdqs1006 to allow catching up on lag [production]
14:30 <vgutierrez> pool cp2042 - T248816 [production]
14:16 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]