|
2020-02-20
§
|
| 10:12 |
<Reedy> |
created $wikidb.blobs_cluster27 on es1023 - T245720 |
[production] |
| 10:08 |
<Reedy> |
created $wikidb.blobs_cluster26 on es1020 - T245720 |
[production] |
| 10:08 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.20/extensions/WikimediaMaintenance/storage/make-all-blobs: (no justification provided) (duration: 01m 04s) |
[production] |
| 09:42 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.20/extensions/WikimediaMaintenance/storage/make-all-blobs: (no justification provided) (duration: 01m 03s) |
[production] |
| 09:27 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.20/extensions/WikimediaMaintenance/storage/make-all-blobs: (no justification provided) (duration: 01m 01s) |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084 after crash - T245621', diff saved to https://phabricator.wikimedia.org/P10467 and previous config saved to /var/cache/conftool/dbconfig/20200220-091233-marostegui.json |
[production] |
| 09:02 |
<akosiaris> |
restart etherpad-lite on etherpad1002 T244238 |
[production] |
| 09:00 |
<marostegui> |
Restart m1 database master db1135 (etherpad will not be available for around 1 minute) - T244238 |
[production] |
| 08:40 |
<jynus> |
disable puppet and stop bacula service T244238 |
[production] |
| 08:35 |
<marostegui> |
Upgrade mysql on db1135 without restart T244238 |
[production] |
| 07:47 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q15k (was Q10k) (T225057) - in case of cache issues (duration: 01m 03s) |
[production] |
| 07:46 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q15k (was Q10k) (T225057) (duration: 01m 03s) |
[production] |
| 07:26 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q10k (was Q8k) (T225057) - in case of cache issue (duration: 01m 01s) |
[production] |
| 07:25 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q10k (was Q8k) (T225057) (duration: 01m 03s) |
[production] |
| 07:17 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8000 (T225057) - in case of cache issue (duration: 01m 03s) |
[production] |
| 07:15 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8000 (T225057) (duration: 01m 03s) |
[production] |
| 07:01 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6000 (T225057) - extra sync for cache issue (duration: 01m 04s) |
[production] |
| 07:00 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6000 (T225057) (duration: 01m 06s) |
[production] |
| 06:46 |
<vgutierrez> |
test trafficserver 8.0.6-rc1 in cp30[64,65] |
[production] |
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084 after crash - T245621', diff saved to https://phabricator.wikimedia.org/P10466 and previous config saved to /var/cache/conftool/dbconfig/20200220-062445-marostegui.json |
[production] |
| 06:17 |
<marostegui> |
Repool labsdb1011 |
[production] |
| 06:12 |
<marostegui> |
Remove partitions from db1101:3318 - T239453 |
[production] |
| 06:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318 to remove revision partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10465 and previous config saved to /var/cache/conftool/dbconfig/20200220-061213-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1099:3318 this host already had the partitions removed - T239453', diff saved to https://phabricator.wikimedia.org/P10464 and previous config saved to /var/cache/conftool/dbconfig/20200220-061019-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 to remove revision partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10463 and previous config saved to /var/cache/conftool/dbconfig/20200220-060914-marostegui.json |
[production] |
| 05:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 on s8, db1099:3318 back to its original weight', diff saved to https://phabricator.wikimedia.org/P10462 and previous config saved to /var/cache/conftool/dbconfig/20200220-055943-marostegui.json |
[production] |
| 00:22 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571860|Allow non-autoconfirmed users to propose OAuth apps (T213760)]] (duration: 01m 04s) |
[production] |
| 00:16 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:573397|Enable password-reset (requireemail pref) on test WD and Commons (T245660)]] (duration: 01m 03s) |
[production] |
|
2020-02-19
§
|
| 23:39 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw138[0-3].eqiad.wmnet |
[production] |
| 23:38 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw137[4-9].eqiad.wmnet |
[production] |
| 23:36 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1363.eqiad.wmnet |
[production] |
| 23:28 |
<jforrester@deploy1001> |
Synchronized wmf-config/PoolCounterSettings.php: cirrus: Reduce CirrusSearch-MoreLike cache workers and queue back to normal (duration: 01m 03s) |
[production] |
| 23:26 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw138[0-3].eqiad.wmnet |
[production] |
| 23:26 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw137[4-9].eqiad.wmnet |
[production] |
| 23:25 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw1363.eqiad.wmnet |
[production] |
| 23:23 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: cirrus: redirect more_like from codfw back to eqiad (duration: 01m 04s) |
[production] |
| 23:13 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:10 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:10 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 23:10 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:09 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 23:09 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:57 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@c16c63a]: articletopic thresholding for ores scores and eventgate port update (duration: 00m 57s) |
[production] |
| 22:56 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@c16c63a]: articletopic thresholding for ores scores and eventgate port update |
[production] |
| 22:54 |
<robh> |
cp3050 & cp3051 returned to service via T243167 |
[production] |
| 22:49 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:49 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:42 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Set wgServer to protocol-relative for Wikitech and Test Wikitech (duration: 01m 05s) |
[production] |
| 22:37 |
<robh> |
taking cp3050 & cp3051 offline for firmware update via T243167 |
[production] |
| 22:23 |
<mutante> |
phabricator - upgrading PHP packages |
[production] |