|
2020-02-13
ยง
|
| 21:37 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:35 |
<ottomata> |
deploying production and canary releases for eventgate-logging-external (and destroying the 'logging-external' release) (safe because eventgate-logging-external is not in use) - T245203 |
[production] |
| 21:29 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:28 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:33 |
<marxarelli> |
rollback to group1 due to 500 spike (2k/min) (T233867) |
[production] |
| 20:32 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: (no justification provided) |
[production] |
| 20:30 |
<marxarelli> |
varnish 500 spike. rolling back |
[production] |
| 20:20 |
<gehel> |
restarting blazegraph + updater on wdqs2006 |
[production] |
| 20:19 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.19 |
[production] |
| 19:44 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/includes/api/ApiRollback.php: T245159 ApiRollback: Properly deal with UserIdentity (duration: 01m 04s) |
[production] |
| 19:20 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/includes/resourceloader/ResourceLoaderSkinModule.php: T245182 ResourceLoaderSkinModule: Don't hard-deprecate wgLogoHD just now (duration: 01m 03s) |
[production] |
| 19:17 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T219534 Add new MLR models for Cirrus on zh/ja/kowiki (duration: 01m 03s) |
[production] |
| 19:10 |
<moritzm> |
installing e2fsprogs security updates |
[production] |
| 18:48 |
<bblack> |
ns1.wikimedia.org - re-routing back to authdns2001 instead of dns2002 on cr[12]-codfw - T242017 |
[production] |
| 18:38 |
<bblack> |
authdns2001 - reboot - T242017 |
[production] |
| 18:36 |
<bblack> |
ns1.wikimedia.org - re-routing from authdns2001 to dns2002 on cr[12]-codfw - T242017 |
[production] |
| 18:09 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I9d0c8af3c577 (duration: 01m 06s) |
[production] |
| 18:00 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Iae1f45896 (duration: 01m 06s) |
[production] |
| 17:59 |
<volans> |
downtimed mgmt in eqiad for 1h |
[production] |
| 17:58 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Iae1f45896 (duration: 01m 08s) |
[production] |
| 17:49 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Ibfca686f681 (duration: 01m 06s) |
[production] |
| 17:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Iefff596955e (duration: 01m 08s) |
[production] |
| 17:40 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Iefff596955e (duration: 01m 06s) |
[production] |
| 17:35 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: I2e4fb0c086de0f8ac (duration: 01m 06s) |
[production] |
| 17:32 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I2e4fb0c086de0f8ac (duration: 01m 06s) |
[production] |
| 17:10 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: No-op (code style only) deploy sync (duration: 01m 07s) |
[production] |
| 17:09 |
<jforrester@deploy1001> |
sync aborted: wmf-config/CommonSettings.php No-op (code style only) deploy sync (duration: 00m 04s) |
[production] |
| 17:09 |
<jforrester@deploy1001> |
Started scap: wmf-config/CommonSettings.php No-op (code style only) deploy sync |
[production] |
| 16:32 |
<robh> |
ps1-a8-codfw.mgmt.codfw.wmnet firmware upgraded via T245164 |
[production] |
| 16:28 |
<papaul> |
rebooting elastic2043 for firmware upgrade |
[production] |
| 16:22 |
<gehel> |
canceled the restart of elastic2043 - T243715 |
[production] |
| 16:21 |
<gehel> |
restarting elastic2043 - T243715 |
[production] |
| 16:10 |
<_joe_> |
depooling/repooling mw1240 |
[production] |
| 16:02 |
<_joe_> |
pooled mw1238 again |
[production] |
| 15:59 |
<_joe_> |
depooling mw1238 for analysis |
[production] |
| 15:42 |
<vgutierrez> |
rolling restart of ats-be on esams - T170567 |
[production] |
| 15:38 |
<vgutierrez> |
disable allow_half_open on ats-tls @ cp4031 - T236458 |
[production] |
| 15:27 |
<vgutierrez> |
turning on TLSv1.3 between ats-be and applayer in cp30[51-52] - T170567 |
[production] |
| 15:22 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/extensions/WikibaseMediaInfo/resources/: UBN fix: Force non-value to be undefined (duration: 01m 06s) |
[production] |
| 14:51 |
<vgutierrez> |
test TLSv1.3 between ats-be and applayer in cp3050 - T170567 |
[production] |
| 14:47 |
<XioNoX> |
re-image rpki2001 - T244585 |
[production] |
| 14:33 |
<XioNoX> |
add routinator_0.6.4_amd64.deb to buster-wikimedia apt repo |
[production] |
| 14:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10405 and previous config saved to /var/cache/conftool/dbconfig/20200213-142735-marostegui.json |
[production] |
| 14:24 |
<XioNoX> |
re-enable ping offload in esams - T244584 |
[production] |
| 13:31 |
<XioNoX> |
disable ping offload in esams - T244584 |
[production] |
| 13:24 |
<XioNoX> |
re-enable ping offload in eqiad - T244584 |
[production] |
| 13:06 |
<XioNoX> |
disable ping offload in eqiad - T244584 |
[production] |
| 13:03 |
<XioNoX> |
re-enable ping offload in codfw - T244584 |
[production] |
| 13:00 |
<vgutierrez> |
pool cp10[75,76] running buster - T242093 |
[production] |
| 12:51 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |