451-500 of 10000 results (55ms)
2020-04-02 ยง
13:50 <marostegui> Compress wbqc_constraints on testcommonswiki and commonswiki (empty tables) - T248967 [production]
13:44 <vgutierrez> update puppet compiler facts [production]
13:40 <marostegui> Deploy schema change on db1111 [production]
13:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1111 for schema change', diff saved to https://phabricator.wikimedia.org/P10861 and previous config saved to /var/cache/conftool/dbconfig/20200402-133956-marostegui.json [production]
13:32 <gehel> OSM data reimport on maps2004 - T249086 [production]
12:55 <mutante> mw1390 - mw1399 - pooled and active but status "staged" in netbox, fixing to 'active' [production]
12:52 <mutante> mw1297 - is pooled and serving traffic but status "staged" in netbox. set to "active" [production]
11:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 after schema change', diff saved to https://phabricator.wikimedia.org/P10858 and previous config saved to /var/cache/conftool/dbconfig/20200402-114020-marostegui.json [production]
10:56 <mutante> decom planet1001 (T248863) [production]
10:56 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
10:55 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
10:19 <marostegui> Deploy schema change on db1087, this will generate lag on s8 on wiki replicas [production]
10:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 for schema change', diff saved to https://phabricator.wikimedia.org/P10857 and previous config saved to /var/cache/conftool/dbconfig/20200402-101920-marostegui.json [production]
10:17 <elukey> set up TLS encryption for all pmacct instances on netflow* to Kafka Jumbo [production]
10:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1104 after schema change', diff saved to https://phabricator.wikimedia.org/P10856 and previous config saved to /var/cache/conftool/dbconfig/20200402-101747-marostegui.json [production]
09:47 <marostegui> Remove haproxy@10.64.37.14 from labsdb hosts - T231280 T248944 [production]
09:40 <gehel> CORRECTION: depool maps2004 for data reimport - T249086 [production]
09:40 <gehel> depool wdqs2004 for data reimport - T249086 [production]
09:33 <oblivian@deploy1001> Finished deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) (duration: 00m 18s) [production]
09:32 <oblivian@deploy1001> Started deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) [production]
09:28 <oblivian@deploy1001> Finished deploy [docker-pkg/deploy@4f86d77]: (no justification provided) (duration: 00m 09s) [production]
09:28 <oblivian@deploy1001> Started deploy [docker-pkg/deploy@4f86d77]: (no justification provided) [production]
08:51 <marostegui> Deploy schema change db1104 [production]
08:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1104 for schema change', diff saved to https://phabricator.wikimedia.org/P10854 and previous config saved to /var/cache/conftool/dbconfig/20200402-085057-marostegui.json [production]
08:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P10853 and previous config saved to /var/cache/conftool/dbconfig/20200402-085019-marostegui.json [production]
08:28 <gehel> repooling wdqs1006 - catched up on lag [production]
08:22 <vgutierrez> Enable inbound TLSv1.3 in upload@esams - T170567 [production]
08:21 <vgutierrez> Enable TLS Session tickets in esams - T245616 [production]
07:45 <moritzm> bounced ferm on ms-be1040 [production]
07:27 <marostegui> Deploy schema change on db1092 [production]
07:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 for schema change', diff saved to https://phabricator.wikimedia.org/P10850 and previous config saved to /var/cache/conftool/dbconfig/20200402-072730-marostegui.json [production]
07:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1101:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10849 and previous config saved to /var/cache/conftool/dbconfig/20200402-072500-marostegui.json [production]
05:49 <marostegui> Deploy schema change on db1101:3318 [production]
05:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10848 and previous config saved to /var/cache/conftool/dbconfig/20200402-054931-marostegui.json [production]
05:29 <elukey> powercycle analytics1045 (host not responsive to ssh, weird chars showed in mgmt serial console) [production]
2020-04-01 ยง
22:44 <volker-e@deploy1001> Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s) [production]
22:43 <volker-e@deploy1001> Started deploy [design/style-guide@4bfe647]: Deploy design/style-guide: [production]
22:02 <volans> forcing logrotate on netflow2001 to compress yesterday's logs [production]
21:53 <volans> force-rebooting ms-be1023, unresponsive - T249174 [production]
21:50 <volans> stopped and restarted kafkatee-webrequest.service on netflow2001, was in a restart loop [production]
19:48 <marxarelli> rollback of 1.35.0-wmf.26 from group1 (T247773). blocked by T249162 [production]
19:30 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: rollback 1.35.0-wmf.26 from group1 [production]
19:21 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.26 (duration: 01m 06s) [production]
19:20 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.26 [production]
19:18 <marxarelli> promoting group1 to 1.35.0-wmf.26 to group1 [production]
17:21 <cdanis> โœ”๏ธ cdanis@cumin1001.eqiad.wmnet ~ ๐Ÿ•โ˜• homer 'cr*eqord*' commit 'enable sampling on eqord Iac15379cc' [production]
16:54 <cdanis> โœ”๏ธ cdanis@cumin1001.eqiad.wmnet ~ ๐Ÿ•โ˜• homer 'cr*eqdfw*' commit 'enable sampling on eqdfw Iac15379cc' [production]
16:39 <vgutierrez> pool cp2027 - T248816 [production]
16:31 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:28 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]