|
2020-07-20
§
|
| 10:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 10:48 |
<moritzm> |
rebooting releases* hosts for kernel security update |
[production] |
| 10:35 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:614698| Bumping portals to master (614698)]] (duration: 00m 56s) |
[production] |
| 10:34 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:614698| Bumping portals to master (614698)]] (duration: 00m 59s) |
[production] |
| 10:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1114', diff saved to https://phabricator.wikimedia.org/P11962 and previous config saved to /var/cache/conftool/dbconfig/20200720-103058-marostegui.json |
[production] |
| 09:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1114', diff saved to https://phabricator.wikimedia.org/P11961 and previous config saved to /var/cache/conftool/dbconfig/20200720-094609-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1114', diff saved to https://phabricator.wikimedia.org/P11960 and previous config saved to /var/cache/conftool/dbconfig/20200720-093154-marostegui.json |
[production] |
| 09:25 |
<godog> |
update compiler facts |
[production] |
| 09:17 |
<jayme> |
updating envoyproxy to 1.14.4-1 on all eqiad hosts |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1114', diff saved to https://phabricator.wikimedia.org/P11959 and previous config saved to /var/cache/conftool/dbconfig/20200720-091119-marostegui.json |
[production] |
| 09:04 |
<jayme> |
updating envoyproxy to 1.14.4-1 on all codfw hosts |
[production] |
| 07:54 |
<moritzm> |
installing libopenmpt security updates |
[production] |
| 07:51 |
<jayme> |
updating envoyproxy to 1.14.4-1 on all non mw and restbase hosts |
[production] |
| 07:29 |
<marostegui> |
Move m1-master from dbproxy1012 to dbproxy1014 - T255408 |
[production] |
| 07:19 |
<marostegui> |
Drop non used reviewdb database - T255715 |
[production] |
| 06:55 |
<elukey> |
restart matomo1002's mariadb to pick up new TLS settings |
[production] |
| 06:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1114', diff saved to https://phabricator.wikimedia.org/P11958 and previous config saved to /var/cache/conftool/dbconfig/20200720-065438-marostegui.json |
[production] |
| 06:15 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.41/extensions/Score/includes/Score.php: reverting Reedy's temporary patch for hardcoding the lilypond version (duration: 00m 57s) |
[production] |
| 06:07 |
<tstarling@deploy1001> |
Finished scap: fixing missing message from previous sync-dir (duration: 29m 57s) |
[production] |
| 05:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11957 and previous config saved to /var/cache/conftool/dbconfig/20200720-055614-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11956 and previous config saved to /var/cache/conftool/dbconfig/20200720-054747-marostegui.json |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11955 and previous config saved to /var/cache/conftool/dbconfig/20200720-053816-marostegui.json |
[production] |
| 05:37 |
<tstarling@deploy1001> |
Started scap: fixing missing message from previous sync-dir |
[production] |
| 05:30 |
<tstarling@deploy1001> |
scap sync-l10n completed (1.35.0-wmf.41) (duration: 02m 44s) |
[production] |
| 05:25 |
<marostegui> |
Deploy MCR schema change on enwiki on db1119 - T238966 |
[production] |
| 05:24 |
<tstarling@deploy1001> |
Synchronized wmf-config/CommonSettings.php: disable lilypond with better error message (duration: 00m 57s) |
[production] |
| 05:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11953 and previous config saved to /var/cache/conftool/dbconfig/20200720-051846-marostegui.json |
[production] |
| 05:18 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.41/extensions/Score: better error message for disabling of Score (duration: 01m 10s) |
[production] |
|
2020-07-18
§
|
| 21:41 |
<shdubsh> |
restart logstash on logstash200[456] |
[production] |
| 21:14 |
<shdubsh> |
bounce logstash on logstash1007 |
[production] |
| 21:10 |
<shdubsh> |
bounce logstash on logstash1008 |
[production] |
| 21:06 |
<shdubsh> |
bounce logstash on logstash1009 |
[production] |
| 20:52 |
<marostegui> |
Due to db1082 crash there will be replication lag on s5 on labsdb hosts - T258336 |
[production] |
| 20:37 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 'depool db1082, it crashed', diff saved to https://phabricator.wikimedia.org/P11951 and previous config saved to /var/cache/conftool/dbconfig/20200718-203704-cdanis.json |
[production] |
| 00:13 |
<dpifke> |
Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged. |
[production] |
|
2020-07-17
§
|
| 21:16 |
<dpifke> |
Removing MongoDB packages and data from webperf1002. |
[production] |
| 17:39 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) (duration: 00m 05s) |
[production] |
| 17:38 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) |
[production] |
| 13:53 |
<akosiaris> |
powercycle kubernetes2002 |
[production] |
| 12:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P11944 and previous config saved to /var/cache/conftool/dbconfig/20200717-122400-marostegui.json |
[production] |
| 12:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11941 and previous config saved to /var/cache/conftool/dbconfig/20200717-120126-marostegui.json |
[production] |
| 11:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11940 and previous config saved to /var/cache/conftool/dbconfig/20200717-115155-marostegui.json |
[production] |
| 11:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11939 and previous config saved to /var/cache/conftool/dbconfig/20200717-113800-marostegui.json |
[production] |
| 11:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11938 and previous config saved to /var/cache/conftool/dbconfig/20200717-113050-marostegui.json |
[production] |
| 11:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1104', diff saved to https://phabricator.wikimedia.org/P11937 and previous config saved to /var/cache/conftool/dbconfig/20200717-112413-marostegui.json |
[production] |