|
2020-02-20
§
|
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084 after crash - T245621', diff saved to https://phabricator.wikimedia.org/P10466 and previous config saved to /var/cache/conftool/dbconfig/20200220-062445-marostegui.json |
[production] |
| 06:17 |
<marostegui> |
Repool labsdb1011 |
[production] |
| 06:12 |
<marostegui> |
Remove partitions from db1101:3318 - T239453 |
[production] |
| 06:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318 to remove revision partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10465 and previous config saved to /var/cache/conftool/dbconfig/20200220-061213-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1099:3318 this host already had the partitions removed - T239453', diff saved to https://phabricator.wikimedia.org/P10464 and previous config saved to /var/cache/conftool/dbconfig/20200220-061019-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 to remove revision partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10463 and previous config saved to /var/cache/conftool/dbconfig/20200220-060914-marostegui.json |
[production] |
| 05:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 on s8, db1099:3318 back to its original weight', diff saved to https://phabricator.wikimedia.org/P10462 and previous config saved to /var/cache/conftool/dbconfig/20200220-055943-marostegui.json |
[production] |
| 00:22 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571860|Allow non-autoconfirmed users to propose OAuth apps (T213760)]] (duration: 01m 04s) |
[production] |
| 00:16 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:573397|Enable password-reset (requireemail pref) on test WD and Commons (T245660)]] (duration: 01m 03s) |
[production] |
|
2020-02-19
§
|
| 23:39 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw138[0-3].eqiad.wmnet |
[production] |
| 23:38 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw137[4-9].eqiad.wmnet |
[production] |
| 23:36 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1363.eqiad.wmnet |
[production] |
| 23:28 |
<jforrester@deploy1001> |
Synchronized wmf-config/PoolCounterSettings.php: cirrus: Reduce CirrusSearch-MoreLike cache workers and queue back to normal (duration: 01m 03s) |
[production] |
| 23:26 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw138[0-3].eqiad.wmnet |
[production] |
| 23:26 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw137[4-9].eqiad.wmnet |
[production] |
| 23:25 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw1363.eqiad.wmnet |
[production] |
| 23:23 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: cirrus: redirect more_like from codfw back to eqiad (duration: 01m 04s) |
[production] |
| 23:13 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:10 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:10 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 23:10 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:09 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 23:09 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:57 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@c16c63a]: articletopic thresholding for ores scores and eventgate port update (duration: 00m 57s) |
[production] |
| 22:56 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@c16c63a]: articletopic thresholding for ores scores and eventgate port update |
[production] |
| 22:54 |
<robh> |
cp3050 & cp3051 returned to service via T243167 |
[production] |
| 22:49 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:49 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:42 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Set wgServer to protocol-relative for Wikitech and Test Wikitech (duration: 01m 05s) |
[production] |
| 22:37 |
<robh> |
taking cp3050 & cp3051 offline for firmware update via T243167 |
[production] |
| 22:23 |
<mutante> |
phabricator - upgrading PHP packages |
[production] |
| 22:14 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw231([0-6]).codfw.wmnet |
[production] |
| 22:12 |
<dzahn@cumin1001> |
conftool action : set/weight=15; selector: name=mw231([0-6]).codfw.wmnet |
[production] |
| 22:11 |
<rzl@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw13(6[4-9]|7[0-3]|84).eqiad.wmnet |
[production] |
| 22:10 |
<rzl@cumin1001> |
conftool action : set/weight=30; selector: name=mw13(6[4-9]|7[0-3]|84).eqiad.wmnet |
[production] |
| 22:08 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw2314.codfw.wmnet |
[production] |
| 21:58 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:58 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:54 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:52 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:48 |
<bblack> |
all authdns servers - upgrade to gdnsd-3.2.2 |
[production] |
| 21:39 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:39 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:36 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:35 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:35 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:35 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:35 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:32 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |