|
2019-09-18
§
|
| 13:11 |
<hashar> |
Restarting Jenkins, starting Zuul |
[production] |
| 12:56 |
<marostegui> |
Deploy schema change on the following s6 hosts: db1088, db1093, db1096, db1098, db1139, dbstore1005 - T231172 |
[production] |
| 12:52 |
<hashar> |
gracefully stopping Zuul (kill SIGUSR1) to prepare for Jenkins restart |
[production] |
| 12:40 |
<marostegui> |
Deploy schema change on s6 codfw master with replication T231172 |
[production] |
| 12:18 |
<vgutierrez> |
restarting ats-tls to avoid spreading Proxy-Connection header - T233205 |
[production] |
| 12:03 |
<marostegui> |
Stop haproxy on dbproxy1006 - T233207 |
[production] |
| 11:29 |
<mobrovac> |
bootstrap restbase2011-c -- T224553 |
[production] |
| 11:27 |
<awight> |
EU SWAT complete |
[production] |
| 11:27 |
<awight@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: SWAT: [[gerrit:537586|Enable FileImport source wiki editing (T228851)]] (duration: 00m 59s) |
[production] |
| 11:25 |
<awight@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:537586|Enable FileImport source wiki editing (T228851)]] (duration: 01m 03s) |
[production] |
| 11:14 |
<awight@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:537375|NowCommons test & test2wiki configuration (T228851)]] (duration: 01m 15s) |
[production] |
| 10:17 |
<onimisionipe> |
force relocation of shards for eqiad search(chi) cluster |
[production] |
| 10:16 |
<moritzm> |
restarting postgres on puppetdb1002/2002 after updating permissions for replication user |
[production] |
| 10:00 |
<mobrovac> |
bootstrap restbase2011-b -- T224553 |
[production] |
| 09:37 |
<godog> |
run swiftrepl eqiad -> codfw on all containers, no deletes |
[production] |
| 09:37 |
<effie> |
upgrading netmon* to PHP 7.2.22 T230024 |
[production] |
| 09:35 |
<godog> |
run swiftrepl eqiad -> codfw for transcoded containers |
[production] |
| 08:59 |
<jynus@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:57 |
<jynus@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2089:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P9125 and previous config saved to /var/cache/conftool/dbconfig/20190918-085721-marostegui.json |
[production] |
| 08:22 |
<mobrovac> |
bootstrap restbase2011-a -- T224553 |
[production] |
| 07:43 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) |
[production] |
| 07:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:17 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:43 |
<moritzm> |
reimaging restbase2011 to stretch T224553 |
[production] |
| 06:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2089:3316 for schema change', diff saved to https://phabricator.wikimedia.org/P9124 and previous config saved to /var/cache/conftool/dbconfig/20190918-060401-marostegui.json |
[production] |
| 05:58 |
<marostegui> |
Deploy schema change on db2097:3316 - T233135 |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool host after onsite checks T233184', diff saved to https://phabricator.wikimedia.org/P9123 and previous config saved to /var/cache/conftool/dbconfig/20190918-054755-marostegui.json |
[production] |
| 05:33 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2055 from config T233186 (duration: 01m 04s) |
[production] |
| 05:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2055 from config T233186 (duration: 01m 06s) |
[production] |
| 05:03 |
<marostegui> |
Start MySQL on db2127 T233184 |
[production] |
| 03:55 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.22/resources/src/mediawiki.util/: 0333729e, ccfe88241 (duration: 01m 07s) |
[production] |
|
2019-09-17
§
|
| 23:25 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.34.0-wmf.23 refs T220748 |
[production] |
| 23:20 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.22/extensions/VisualEditor/extension.json: aae62a87be3c954378b07dfb881f79a4f73c5def (duration: 01m 05s) |
[production] |
| 22:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 22:43 |
<dzahn@cumin1001> |
Updating IPMI password on 6 hosts - dzahn@cumin1001 |
[production] |
| 22:43 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 22:09 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Add comment about MinimumPasswordLengthToLogin (duration: 01m 03s) |
[production] |
| 21:45 |
<cstone> |
civicrm revision changed from 45dbfdb96f to 90db6cb5a1 |
[production] |
| 21:45 |
<tzatziki> |
removed one file for legal compliance |
[production] |
| 21:12 |
<XioNoX> |
delete AS13335 91.198.174.0/24 RPKI/ROA |
[production] |
| 21:10 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 21:10 |
<dzahn@cumin1001> |
Updating IPMI password on 1 hosts - dzahn@cumin1001 |
[production] |
| 21:10 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 21:08 |
<@> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 21:07 |
<twentyafterfour@deploy1001> |
Finished scap: testwikis to 1.34.0-wmf.23 refs T220748 (duration: 24m 55s) |
[production] |
| 21:01 |
<XioNoX> |
enable interface damping on primary eqiad-esams link (eqiad side) - T196432 |
[production] |
| 20:57 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:47 |
<dzahn@cumin1001> |
Updating IPMI password on 660 hosts - dzahn@cumin1001 |
[production] |
| 20:46 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |