1701-1750 of 10000 results (26ms)
2020-12-04 §
01:04 <ryankemper> T269406 https://grafana.wikimedia.org/d/000000305/maps-performances?viewPanel=11&orgId=1&var-cluster=maps1&from=1606827063027&to=1607043666975 shows that the normal daily dropoff in lag did not occur today, leading to the criticals. It's possible some sort of daily job has failed [production]
00:16 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
00:14 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
00:14 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
00:12 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
00:06 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
00:04 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-12-03 §
23:47 <ejegg> adjusted timings for donations queue consumer and thank you mailer [production]
23:44 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
23:39 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
22:50 <ejegg> updated standalone SmashPig IPN listener from 63dffcb11f to 3029b07004 [production]
22:49 <razzi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:49 <razzi@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:15 <shdubsh> restart elasticsearch on logstash1010 - gc issues [production]
22:15 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:46 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: rollback to wmf.18 [production]
21:39 <twentyafterfour> rolling back wmf.20 due to T269396 refs T263186 [production]
21:37 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet [production]
21:26 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
21:26 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
21:25 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:23 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
21:23 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:21 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:21 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:21 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:21 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:21 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:05 <bstorm> running maintain-dbusers harvest-replicas to populate the user accounts on new wikireplicas servers [production]
20:46 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:44 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:43 <shdubsh> kill slapd on serpens and restart it [production]
20:42 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:40 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:39 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:38 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:38 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:38 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:38 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:28 <ryankemper@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:26 <ryankemper@cumin2001> START - Cookbook sre.hosts.downtime [production]
20:12 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:10 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:08 <ryankemper> T269204 Re-imaging `wdqs2004` to upgrade it to buster: `sudo -i wmf-auto-reimage-host --conftool -p T269204 wdqs2004.codfw.wmnet` [production]
20:03 <hashar@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.20 [production]
19:58 <milimetric@deploy1001> Finished deploy [analytics/aqs/deploy@95d6432]: (no justification provided) (duration: 00m 23s) [production]
19:58 <milimetric@deploy1001> Started deploy [analytics/aqs/deploy@95d6432]: (no justification provided) [production]
19:57 <milimetric@deploy1001> Finished deploy [analytics/aqs/deploy@95d6432]: (no justification provided) (duration: 00m 19s) [production]
19:57 <shdubsh> restart logstash kafka in codfw - java updates [production]
19:57 <milimetric@deploy1001> Started deploy [analytics/aqs/deploy@95d6432]: (no justification provided) [production]