|
2020-02-13
ยง
|
| 18:48 |
<bblack> |
ns1.wikimedia.org - re-routing back to authdns2001 instead of dns2002 on cr[12]-codfw - T242017 |
[production] |
| 18:38 |
<bblack> |
authdns2001 - reboot - T242017 |
[production] |
| 18:36 |
<bblack> |
ns1.wikimedia.org - re-routing from authdns2001 to dns2002 on cr[12]-codfw - T242017 |
[production] |
| 18:09 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I9d0c8af3c577 (duration: 01m 06s) |
[production] |
| 18:00 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Iae1f45896 (duration: 01m 06s) |
[production] |
| 17:59 |
<volans> |
downtimed mgmt in eqiad for 1h |
[production] |
| 17:58 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Iae1f45896 (duration: 01m 08s) |
[production] |
| 17:49 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Ibfca686f681 (duration: 01m 06s) |
[production] |
| 17:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Iefff596955e (duration: 01m 08s) |
[production] |
| 17:40 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Iefff596955e (duration: 01m 06s) |
[production] |
| 17:35 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: I2e4fb0c086de0f8ac (duration: 01m 06s) |
[production] |
| 17:32 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I2e4fb0c086de0f8ac (duration: 01m 06s) |
[production] |
| 17:10 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: No-op (code style only) deploy sync (duration: 01m 07s) |
[production] |
| 17:09 |
<jforrester@deploy1001> |
sync aborted: wmf-config/CommonSettings.php No-op (code style only) deploy sync (duration: 00m 04s) |
[production] |
| 17:09 |
<jforrester@deploy1001> |
Started scap: wmf-config/CommonSettings.php No-op (code style only) deploy sync |
[production] |
| 16:32 |
<robh> |
ps1-a8-codfw.mgmt.codfw.wmnet firmware upgraded via T245164 |
[production] |
| 16:28 |
<papaul> |
rebooting elastic2043 for firmware upgrade |
[production] |
| 16:22 |
<gehel> |
canceled the restart of elastic2043 - T243715 |
[production] |
| 16:21 |
<gehel> |
restarting elastic2043 - T243715 |
[production] |
| 16:10 |
<_joe_> |
depooling/repooling mw1240 |
[production] |
| 16:02 |
<_joe_> |
pooled mw1238 again |
[production] |
| 15:59 |
<_joe_> |
depooling mw1238 for analysis |
[production] |
| 15:42 |
<vgutierrez> |
rolling restart of ats-be on esams - T170567 |
[production] |
| 15:38 |
<vgutierrez> |
disable allow_half_open on ats-tls @ cp4031 - T236458 |
[production] |
| 15:27 |
<vgutierrez> |
turning on TLSv1.3 between ats-be and applayer in cp30[51-52] - T170567 |
[production] |
| 15:22 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/extensions/WikibaseMediaInfo/resources/: UBN fix: Force non-value to be undefined (duration: 01m 06s) |
[production] |
| 14:51 |
<vgutierrez> |
test TLSv1.3 between ats-be and applayer in cp3050 - T170567 |
[production] |
| 14:47 |
<XioNoX> |
re-image rpki2001 - T244585 |
[production] |
| 14:33 |
<XioNoX> |
add routinator_0.6.4_amd64.deb to buster-wikimedia apt repo |
[production] |
| 14:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10405 and previous config saved to /var/cache/conftool/dbconfig/20200213-142735-marostegui.json |
[production] |
| 14:24 |
<XioNoX> |
re-enable ping offload in esams - T244584 |
[production] |
| 13:31 |
<XioNoX> |
disable ping offload in esams - T244584 |
[production] |
| 13:24 |
<XioNoX> |
re-enable ping offload in eqiad - T244584 |
[production] |
| 13:06 |
<XioNoX> |
disable ping offload in eqiad - T244584 |
[production] |
| 13:03 |
<XioNoX> |
re-enable ping offload in codfw - T244584 |
[production] |
| 13:00 |
<vgutierrez> |
pool cp10[75,76] running buster - T242093 |
[production] |
| 12:51 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:49 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:48 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:47 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:34 |
<Amir1> |
EU SWAT is done |
[production] |
| 12:30 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571961|Read and write more in the new term store]], take II, the cache issue (T219123 T225055) (duration: 01m 03s) |
[production] |
| 12:29 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571961|Read and write more in the new term store]] (duration: 01m 03s) |
[production] |
| 12:29 |
<vgutierrez> |
depool cp10[75,76] and reimage as buster - T242093 |
[production] |
| 12:28 |
<vgutierrez> |
pool cp10[77,78] running buster - T242093 |
[production] |
| 12:20 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571956|Revert: Triple the factor of WDQS lag to maxlag for Wikidata (T244722)]] (duration: 01m 04s) |
[production] |
| 12:18 |
<XioNoX> |
re-image ping2001 to buster - T244584 |
[production] |
| 12:16 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 1c81925: Create Test Custodians group at Beta Wikiversity (T240438) (duration: 01m 07s) |
[production] |
| 12:13 |
<XioNoX> |
disable ping offload in codfw |
[production] |