|
2019-10-15
§
|
| 07:48 |
<Urbanecm> |
Password reset for Xaris333 #2 (T235441) |
[production] |
| 07:41 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:41 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:34 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 for PDU maintenance T226782', diff saved to https://phabricator.wikimedia.org/P9345 and previous config saved to /var/cache/conftool/dbconfig/20191015-071338-marostegui.json |
[production] |
| 07:10 |
<XioNoX> |
failover VRRP from cr1-eqiad to cr2-eqiad in prevision of the PDU work of - T226782 |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2086:3318 T232446', diff saved to https://phabricator.wikimedia.org/P9344 and previous config saved to /var/cache/conftool/dbconfig/20191015-064419-marostegui.json |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1070 T235464', diff saved to https://phabricator.wikimedia.org/P9343 and previous config saved to /var/cache/conftool/dbconfig/20191015-064005-marostegui.json |
[production] |
| 05:38 |
<marostegui> |
Depool labsdb1009 for PDU maintenance T226782 |
[production] |
| 05:28 |
<marostegui> |
Deploy schema change on db1098:3317 T234066 T233135 |
[production] |
| 05:28 |
<marostegui> |
Deploy schema change on db1097:3314 T233625 |
[production] |
| 05:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3314', diff saved to https://phabricator.wikimedia.org/P9342 and previous config saved to /var/cache/conftool/dbconfig/20191015-052621-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3317', diff saved to https://phabricator.wikimedia.org/P9341 and previous config saved to /var/cache/conftool/dbconfig/20191015-052220-marostegui.json |
[production] |
| 05:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2085:3318', diff saved to https://phabricator.wikimedia.org/P9340 and previous config saved to /var/cache/conftool/dbconfig/20191015-051924-marostegui.json |
[production] |
| 05:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314', diff saved to https://phabricator.wikimedia.org/P9339 and previous config saved to /var/cache/conftool/dbconfig/20191015-051400-marostegui.json |
[production] |
| 05:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P9338 and previous config saved to /var/cache/conftool/dbconfig/20191015-051236-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Promote db1100 to s5 master and remove read-only from s5 T234300', diff saved to https://phabricator.wikimedia.org/P9337 and previous config saved to /var/cache/conftool/dbconfig/20191015-050042-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Set s5 as read-only for maintenance T234300', diff saved to https://phabricator.wikimedia.org/P9336 and previous config saved to /var/cache/conftool/dbconfig/20191015-050016-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s5 failover from db1070 to db1100 - T234300 |
[production] |
| 04:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317 after schema change', diff saved to https://phabricator.wikimedia.org/P9335 and previous config saved to /var/cache/conftool/dbconfig/20191015-043403-marostegui.json |
[production] |
| 04:15 |
<marostegui> |
Start pre-switchover steps T234300 |
[production] |
|
2019-10-14
§
|
| 23:26 |
<Krinkle> |
Delete 2019-09-01––2019-09-10 arclamp trace logs from webperf1002, and decompress the rest of 2019-09 (this will trigger svg re-generation), T235425 |
[production] |
| 23:10 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 86f12b6e (duration: 00m 51s) |
[production] |
| 21:47 |
<Krinkle> |
Deleting 2019-09-01––2019-09-10 arclamp logs on webperf2002, and decompress the rest of 2019-09, T235425 |
[production] |
| 21:12 |
<Krinkle> |
Delete misc arclamp/logs and arclamp/svgs data from between 2018 and and 2019-08 on webperf1002/webperf2002, T235425 |
[production] |
| 20:41 |
<maxsem@deploy1001> |
Synchronized php-1.35.0-wmf.1/includes/: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/542963/ (duration: 00m 55s) |
[production] |
| 17:56 |
<mutante> |
webperf2002 - /srv/xenon/logs/daily# gzip 2019-09*excimer*.log (T235425) |
[production] |
| 17:21 |
<onimisionipe@deploy1001> |
Finished deploy [wdqs/wdqs@217cac5]: New blazegraph build and GUI updates (duration: 16m 45s) |
[production] |
| 17:04 |
<onimisionipe@deploy1001> |
Started deploy [wdqs/wdqs@217cac5]: New blazegraph build and GUI updates |
[production] |
| 16:07 |
<moritzm> |
imported cergen 0.2.4-1+deb10u3 to component/cergen for buster-wikimedia T235405 |
[production] |
| 16:00 |
<Urbanecm> |
Password reset for Xaris333 (T235441) |
[production] |
| 15:57 |
<moritzm> |
imported cergen 0.2.4-1+deb10u2 to component/cergen for buster-wikimedia T235405 |
[production] |
| 14:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3314 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9329 and previous config saved to /var/cache/conftool/dbconfig/20191014-142843-marostegui.json |
[production] |
| 14:28 |
<elukey> |
upload matomo 3.11 to stretch-wikimedia and upgrade matomo1001 - T234607 |
[production] |
| 14:21 |
<marostegui> |
Deploy schema change on db1116:3317 T234066 T233135 |
[production] |
| 14:13 |
<effie> |
Enable puppet on mw* servers and reload apache - T229792 |
[production] |
| 13:48 |
<moritzm> |
imported cergen 0.2.4-1+deb10u1 to component/cergen for buster-wikimedia T235405 |
[production] |
| 13:42 |
<marostegui> |
Repool labsdb1009 after PSU replacement - T233273 |
[production] |
| 13:36 |
<effie> |
Slowly enable puppet on mw* canaries |
[production] |
| 13:26 |
<moritzm> |
imported python-networkx 1.11-2~wmf1 to component/cergen for buster-wikimedia T235405 |
[production] |
| 13:21 |
<@> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 13:19 |
<@> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 13:18 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 13:18 |
<effie> |
Disable puppet on mw* to remove php72_only feature flag - T229792 |
[production] |
| 13:13 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 245b4e5: Add banwiki logo to IS.php (T234768) (duration: 00m 51s) |
[production] |
| 13:12 |
<Urbanecm> |
Run git reset --hard origin/master in /srv/mediawiki-stagging (deleted https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/542920 and https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/542919 from deployment srv, both don't actually change anything => safe to delete) (T234768) |
[production] |
| 13:10 |
<marostegui> |
Sanitize banwiki on db1124:3313 and db2094:3313 T234770 |
[production] |
| 12:44 |
<Amir1> |
Creating banwiki is banned (done) |
[production] |
| 12:40 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 03m 04s) |
[production] |