|
2020-02-14
§
|
| 09:05 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:46 |
<moritzm> |
installing 4.19.98 kernel update on Buster systems |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 100 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10408 and previous config saved to /var/cache/conftool/dbconfig/20200214-080600-marostegui.json |
[production] |
| 06:51 |
<vgutierrez> |
updating puppet compiler facts |
[production] |
| 01:27 |
<dpifke@deploy1001> |
Finished deploy [performance/navtiming@2eec00a]: (no justification provided) (duration: 00m 05s) |
[production] |
| 01:27 |
<dpifke@deploy1001> |
Started deploy [performance/navtiming@2eec00a]: (no justification provided) |
[production] |
| 00:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T245202 cirrus: Move all move_like traffic to codfw (duration: 01m 02s) |
[production] |
| 00:51 |
<jforrester@deploy1001> |
Synchronized wmf-config/PoolCounterSettings.php: T245202 cirrus: Increase the pool counter limits a bit (duration: 01m 05s) |
[production] |
|
2020-02-13
§
|
| 22:13 |
<jeh> |
running filesystem tests on cloudvirt1024 T241884 |
[production] |
| 21:42 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:41 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:40 |
<jbond42> |
refresh facts on compilers |
[production] |
| 21:38 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:37 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:35 |
<ottomata> |
deploying production and canary releases for eventgate-logging-external (and destroying the 'logging-external' release) (safe because eventgate-logging-external is not in use) - T245203 |
[production] |
| 21:29 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:28 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:33 |
<marxarelli> |
rollback to group1 due to 500 spike (2k/min) (T233867) |
[production] |
| 20:32 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: (no justification provided) |
[production] |
| 20:30 |
<marxarelli> |
varnish 500 spike. rolling back |
[production] |
| 20:20 |
<gehel> |
restarting blazegraph + updater on wdqs2006 |
[production] |
| 20:19 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.19 |
[production] |
| 19:44 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/includes/api/ApiRollback.php: T245159 ApiRollback: Properly deal with UserIdentity (duration: 01m 04s) |
[production] |
| 19:20 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/includes/resourceloader/ResourceLoaderSkinModule.php: T245182 ResourceLoaderSkinModule: Don't hard-deprecate wgLogoHD just now (duration: 01m 03s) |
[production] |
| 19:17 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T219534 Add new MLR models for Cirrus on zh/ja/kowiki (duration: 01m 03s) |
[production] |
| 19:10 |
<moritzm> |
installing e2fsprogs security updates |
[production] |
| 18:48 |
<bblack> |
ns1.wikimedia.org - re-routing back to authdns2001 instead of dns2002 on cr[12]-codfw - T242017 |
[production] |
| 18:38 |
<bblack> |
authdns2001 - reboot - T242017 |
[production] |
| 18:36 |
<bblack> |
ns1.wikimedia.org - re-routing from authdns2001 to dns2002 on cr[12]-codfw - T242017 |
[production] |
| 18:09 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I9d0c8af3c577 (duration: 01m 06s) |
[production] |
| 18:00 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Iae1f45896 (duration: 01m 06s) |
[production] |
| 17:59 |
<volans> |
downtimed mgmt in eqiad for 1h |
[production] |
| 17:58 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Iae1f45896 (duration: 01m 08s) |
[production] |
| 17:49 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Ibfca686f681 (duration: 01m 06s) |
[production] |
| 17:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Iefff596955e (duration: 01m 08s) |
[production] |
| 17:40 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Iefff596955e (duration: 01m 06s) |
[production] |
| 17:35 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: I2e4fb0c086de0f8ac (duration: 01m 06s) |
[production] |
| 17:32 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I2e4fb0c086de0f8ac (duration: 01m 06s) |
[production] |
| 17:10 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: No-op (code style only) deploy sync (duration: 01m 07s) |
[production] |
| 17:09 |
<jforrester@deploy1001> |
sync aborted: wmf-config/CommonSettings.php No-op (code style only) deploy sync (duration: 00m 04s) |
[production] |
| 17:09 |
<jforrester@deploy1001> |
Started scap: wmf-config/CommonSettings.php No-op (code style only) deploy sync |
[production] |
| 16:32 |
<robh> |
ps1-a8-codfw.mgmt.codfw.wmnet firmware upgraded via T245164 |
[production] |
| 16:28 |
<papaul> |
rebooting elastic2043 for firmware upgrade |
[production] |
| 16:22 |
<gehel> |
canceled the restart of elastic2043 - T243715 |
[production] |
| 16:21 |
<gehel> |
restarting elastic2043 - T243715 |
[production] |
| 16:10 |
<_joe_> |
depooling/repooling mw1240 |
[production] |
| 16:02 |
<_joe_> |
pooled mw1238 again |
[production] |
| 15:59 |
<_joe_> |
depooling mw1238 for analysis |
[production] |
| 15:42 |
<vgutierrez> |
rolling restart of ats-be on esams - T170567 |
[production] |
| 15:38 |
<vgutierrez> |
disable allow_half_open on ats-tls @ cp4031 - T236458 |
[production] |