|
2020-07-20
§
|
| 05:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11957 and previous config saved to /var/cache/conftool/dbconfig/20200720-055614-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11956 and previous config saved to /var/cache/conftool/dbconfig/20200720-054747-marostegui.json |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11955 and previous config saved to /var/cache/conftool/dbconfig/20200720-053816-marostegui.json |
[production] |
| 05:37 |
<tstarling@deploy1001> |
Started scap: fixing missing message from previous sync-dir |
[production] |
| 05:30 |
<tstarling@deploy1001> |
scap sync-l10n completed (1.35.0-wmf.41) (duration: 02m 44s) |
[production] |
| 05:25 |
<marostegui> |
Deploy MCR schema change on enwiki on db1119 - T238966 |
[production] |
| 05:24 |
<tstarling@deploy1001> |
Synchronized wmf-config/CommonSettings.php: disable lilypond with better error message (duration: 00m 57s) |
[production] |
| 05:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11953 and previous config saved to /var/cache/conftool/dbconfig/20200720-051846-marostegui.json |
[production] |
| 05:18 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.41/extensions/Score: better error message for disabling of Score (duration: 01m 10s) |
[production] |
|
2020-07-18
§
|
| 21:41 |
<shdubsh> |
restart logstash on logstash200[456] |
[production] |
| 21:14 |
<shdubsh> |
bounce logstash on logstash1007 |
[production] |
| 21:10 |
<shdubsh> |
bounce logstash on logstash1008 |
[production] |
| 21:06 |
<shdubsh> |
bounce logstash on logstash1009 |
[production] |
| 20:52 |
<marostegui> |
Due to db1082 crash there will be replication lag on s5 on labsdb hosts - T258336 |
[production] |
| 20:37 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 'depool db1082, it crashed', diff saved to https://phabricator.wikimedia.org/P11951 and previous config saved to /var/cache/conftool/dbconfig/20200718-203704-cdanis.json |
[production] |
| 00:13 |
<dpifke> |
Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged. |
[production] |
|
2020-07-17
§
|
| 21:16 |
<dpifke> |
Removing MongoDB packages and data from webperf1002. |
[production] |
| 17:39 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) (duration: 00m 05s) |
[production] |
| 17:38 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) |
[production] |
| 13:53 |
<akosiaris> |
powercycle kubernetes2002 |
[production] |
| 12:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P11944 and previous config saved to /var/cache/conftool/dbconfig/20200717-122400-marostegui.json |
[production] |
| 12:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11941 and previous config saved to /var/cache/conftool/dbconfig/20200717-120126-marostegui.json |
[production] |
| 11:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11940 and previous config saved to /var/cache/conftool/dbconfig/20200717-115155-marostegui.json |
[production] |
| 11:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11939 and previous config saved to /var/cache/conftool/dbconfig/20200717-113800-marostegui.json |
[production] |
| 11:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11938 and previous config saved to /var/cache/conftool/dbconfig/20200717-113050-marostegui.json |
[production] |
| 11:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1104', diff saved to https://phabricator.wikimedia.org/P11937 and previous config saved to /var/cache/conftool/dbconfig/20200717-112413-marostegui.json |
[production] |
| 09:15 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1280.eqiad.wmnet |
[production] |
| 09:12 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=mw1280.eqiad.wmnet |
[production] |
| 08:48 |
<moritzm> |
imported prometheus-atlas-exporter 1.0+git20191204.ffafab7-2 to buster-wikimedia T247967 |
[production] |
| 08:29 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 08:05 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:54 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11936 and previous config saved to /var/cache/conftool/dbconfig/20200717-075124-marostegui.json |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1111', diff saved to https://phabricator.wikimedia.org/P11935 and previous config saved to /var/cache/conftool/dbconfig/20200717-074335-marostegui.json |
[production] |
| 07:34 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 07:34 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 07:33 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 07:33 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 07:32 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 07:30 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 06:30 |
<XioNoX> |
rename msw1-codfw interface range |
[production] |
| 06:28 |
<XioNoX> |
rename msw1-eqiad interface range |
[production] |
| 04:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111', diff saved to https://phabricator.wikimedia.org/P11934 and previous config saved to /var/cache/conftool/dbconfig/20200717-044748-marostegui.json |
[production] |
| 04:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092', diff saved to https://phabricator.wikimedia.org/P11933 and previous config saved to /var/cache/conftool/dbconfig/20200717-044658-marostegui.json |
[production] |