851-900 of 10000 results (63ms)
2020-04-02 §
05:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10848 and previous config saved to /var/cache/conftool/dbconfig/20200402-054931-marostegui.json [production]
05:29 <elukey> powercycle analytics1045 (host not responsive to ssh, weird chars showed in mgmt serial console) [production]
2020-04-01 §
22:44 <volker-e@deploy1001> Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s) [production]
22:43 <volker-e@deploy1001> Started deploy [design/style-guide@4bfe647]: Deploy design/style-guide: [production]
22:02 <volans> forcing logrotate on netflow2001 to compress yesterday's logs [production]
21:53 <volans> force-rebooting ms-be1023, unresponsive - T249174 [production]
21:50 <volans> stopped and restarted kafkatee-webrequest.service on netflow2001, was in a restart loop [production]
19:48 <marxarelli> rollback of 1.35.0-wmf.26 from group1 (T247773). blocked by T249162 [production]
19:30 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: rollback 1.35.0-wmf.26 from group1 [production]
19:21 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.26 (duration: 01m 06s) [production]
19:20 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.26 [production]
19:18 <marxarelli> promoting group1 to 1.35.0-wmf.26 to group1 [production]
17:21 <cdanis> ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqord*' commit 'enable sampling on eqord Iac15379cc' [production]
16:54 <cdanis> ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqdfw*' commit 'enable sampling on eqdfw Iac15379cc' [production]
16:39 <vgutierrez> pool cp2027 - T248816 [production]
16:31 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:28 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:17 <ariel@deploy1001> Finished deploy [dumps/dumps@21363c1]: page range prefetch fixup (duration: 00m 09s) [production]
16:17 <ariel@deploy1001> Started deploy [dumps/dumps@21363c1]: page range prefetch fixup [production]
15:33 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
15:31 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:31 <vgutierrez@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
15:31 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:29 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:29 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:27 <vgutierrez> depool & decommission cp20[16,19,23,27] - T249125 [production]
15:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1099:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10845 and previous config saved to /var/cache/conftool/dbconfig/20200401-152258-marostegui.json [production]
15:11 <herron> performing kafka-main rolling restarts to pick up security updates [production]
14:52 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:50 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:49 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:49 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:46 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:46 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:43 <vgutierrez> depool && decommission cp[2018,2020,2022,2024-2026].codfw.wmnet - T249115 [production]
14:32 <gehel> depooling wdqs1006 to allow catching up on lag [production]
14:30 <vgutierrez> pool cp2042 - T248816 [production]
14:16 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:13 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:09 <XioNoX> remove AS-path prepending in esams [production]
13:47 <XioNoX> remove AS-path prepending in eqsin [production]
13:39 <vgutierrez> pool cp2041 - T248816 [production]
13:34 <mutante> sodium (mirror): sudo -u mirror ftpsync to get Debian mirror updated (Icinga says it's old) [production]
13:24 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:24 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:17 <marostegui> Deploy schema change on db1099:3318 [production]
13:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10843 and previous config saved to /var/cache/conftool/dbconfig/20200401-131719-marostegui.json [production]
13:13 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:19 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]