|
2020-12-04
§
|
| 15:15 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:38 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 13:38 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 13:07 |
<akosiaris> |
create apertium namespace on k8s clusters. T255672 |
[production] |
| 11:24 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:24 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:31 |
<jynus> |
setting db1133 as read-write for backup testing |
[production] |
| 10:28 |
<moritzm> |
resetting cumin-check-aliases.service on cumin* hosts |
[production] |
| 09:54 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:54 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:30 |
<moritzm> |
installing zsh security updates on stretch |
[production] |
| 09:26 |
<moritzm> |
installing mutt security updates |
[production] |
| 08:58 |
<moritzm> |
installing lxml security updates |
[production] |
| 07:09 |
<marostegui> |
Stop mysql on clouddb1016 to clone clouddb1020 T267090 |
[production] |
| 07:02 |
<marostegui> |
Increase pvs on db[1151-1155] T269324 T268742 |
[production] |
| 02:16 |
<eileen> |
civicrm revision changed from 913ccdfd2b to 5fa107d32a, config revision is ffe0a99133 |
[production] |
| 01:43 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 01:42 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:04 |
<ryankemper> |
T269406 https://grafana.wikimedia.org/d/000000305/maps-performances?viewPanel=11&orgId=1&var-cluster=maps1&from=1606827063027&to=1607043666975 shows that the normal daily dropoff in lag did not occur today, leading to the criticals. It's possible some sort of daily job has failed |
[production] |
| 00:16 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:14 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:14 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 00:12 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 00:06 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:04 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
|
2020-12-03
§
|
| 23:47 |
<ejegg> |
adjusted timings for donations queue consumer and thank you mailer |
[production] |
| 23:44 |
<ryankemper@cumin2001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 23:39 |
<ryankemper@cumin2001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 22:50 |
<ejegg> |
updated standalone SmashPig IPN listener from 63dffcb11f to 3029b07004 |
[production] |
| 22:49 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:49 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:15 |
<shdubsh> |
restart elasticsearch on logstash1010 - gc issues |
[production] |
| 22:15 |
<ryankemper@cumin2001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 21:46 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: rollback to wmf.18 |
[production] |
| 21:39 |
<twentyafterfour> |
rolling back wmf.20 due to T269396 refs T263186 |
[production] |
| 21:37 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet |
[production] |
| 21:26 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:26 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:25 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:23 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:23 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:21 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:21 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:21 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:21 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:21 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:05 |
<bstorm> |
running maintain-dbusers harvest-replicas to populate the user accounts on new wikireplicas servers |
[production] |
| 20:46 |
<ryankemper@cumin2001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |