|
2019-09-10
§
|
| 10:34 |
<@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 10:32 |
<vgutierrez> |
repool cp5001 with ats-tls collecting memory usage details every hour - T232298 |
[production] |
| 09:56 |
<elukey> |
restart archiva on archiva1001 - UI not working (probably due to connections to maven central being stuck) |
[production] |
| 09:50 |
<moritzm> |
installing ghostscript security updates on jessie |
[production] |
| 09:37 |
<moritzm> |
added jbond as chanserv ops for #wikimedia-operations |
[production] |
| 08:08 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:42 |
<moritzm> |
reimaging mw2231 after hardware maintenance T231192 |
[production] |
| 07:21 |
<moritzm> |
iron.wikimedia.org is no longer a bastion host |
[production] |
| 06:57 |
<moritzm> |
upgrading snapshot* to PHP 7.2.22 T230024 |
[production] |
| 05:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1073 from config T231892 (duration: 00m 54s) |
[production] |
| 05:45 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db1073 from config T231892 (duration: 00m 55s) |
[production] |
| 05:35 |
<marostegui> |
Stop MySQL on db2047 T231852 |
[production] |
| 05:35 |
<marostegui> |
Remove db2047 from tendril and zarcillo - T231852 |
[production] |
| 05:33 |
<urandom> |
decommissioning Cassandra, restbase-dev1005-b -- T224554 |
[production] |
| 05:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1104 into API T230762', diff saved to https://phabricator.wikimedia.org/P9071 and previous config saved to /var/cache/conftool/dbconfig/20190910-051529-marostegui.json |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1109 to s8 master and remove read-only from s8 T227062', diff saved to https://phabricator.wikimedia.org/P9070 and previous config saved to /var/cache/conftool/dbconfig/20190910-050213-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s8 as read-only for maintenance T230762', diff saved to https://phabricator.wikimedia.org/P9069 and previous config saved to /var/cache/conftool/dbconfig/20190910-050046-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s8 failover from db1104 to db1109 - T227062 |
[production] |
| 04:46 |
<vgutierrez> |
depool cp5001 for memory leak debugging on ATS - T232298 |
[production] |
| 04:23 |
<marostegui> |
Start topology changes on s8, connect everything under db1109 - T230762 |
[production] |
| 04:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1109 with weight 0 and depool it from API T230762', diff saved to https://phabricator.wikimedia.org/P9068 and previous config saved to /var/cache/conftool/dbconfig/20190910-042243-marostegui.json |
[production] |
| 04:18 |
<marostegui> |
Start s8 (wikidata) pre switchover steps T230762 |
[production] |
| 00:59 |
<krinkle@deploy1001> |
Finished deploy [performance/navtiming@f2a0863]: (no justification provided) (duration: 00m 05s) |
[production] |
| 00:59 |
<krinkle@deploy1001> |
Started deploy [performance/navtiming@f2a0863]: (no justification provided) |
[production] |
| 00:57 |
<Krinkle> |
krinkle@deploy1001: Deploy performance/navtiming f2a0863b9e4774140463a79d08051814c0d21116 - T226539 |
[production] |
| 00:41 |
<urandom> |
decommissioning Cassandra, restbase-dev1005-a -- T224554 |
[production] |
|
2019-09-09
§
|
| 23:44 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.21/skins/MinervaNeue/: T232260 (duration: 00m 57s) |
[production] |
| 22:28 |
<ejegg> |
updated payments-wiki from 51d9ed79b6 to 15baf7f58b |
[production] |
| 20:50 |
<urandom> |
bootstrapping Cassandra, restbase-dev1004-b -- T224554 |
[production] |
| 19:48 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 05m 45s) |
[production] |
| 19:42 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 |
[production] |
| 19:41 |
<mdholloway> |
mobileapps deployment failed repooling canary (scb2001); retrying |
[production] |
| 19:40 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 02m 59s) |
[production] |
| 19:37 |
<XioNoX> |
fix eqsin CF tunnel missconfig |
[production] |
| 19:37 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 |
[production] |
| 17:56 |
<andrewbogott> |
disabling puppet on labpuppetmaster1001 as part of T171188 |
[production] |
| 17:55 |
<XioNoX> |
push cloudflare tunnel config to cr1-eqsin |
[production] |
| 16:50 |
<papaul> |
replacing Fan kit and power supplies on cr1-codfw |
[production] |
| 14:22 |
<urandom> |
bootstrapping Cassandra, restbase-dev1004-a -- T224554 |
[production] |
| 13:51 |
<vgutierrez> |
upgrading ats to 8.0.5-1wm6 on cp5001 - T232298 |
[production] |
| 13:39 |
<vgutierrez> |
uploaded trafficserver 8.0.5-1wm6 to apt.wikimedia.org (stretch) - T232298 |
[production] |
| 13:31 |
<moritzm> |
installing facter update from buster 10.1 point release (T222356) |
[production] |
| 13:15 |
<moritzm> |
upgrading labweb/wikitech to PHP 7.2.22 T230024 |
[production] |
| 13:02 |
<Urbanecm> |
Patch is deployed, deploy1001 should be clear |
[production] |
| 13:01 |
<moritzm> |
upgrading remaining mediawiki app servers (mw1266-mw1275) to PHP 7.2.22 T230024 |
[production] |
| 12:55 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.21/extensions/WikibaseMediaInfo/: ubn patch T231276 (duration: 00m 58s) |
[production] |
| 12:51 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.21/extensions/Wikibase: ubn patch T231276 (duration: 01m 03s) |
[production] |
| 12:48 |
<moritzm> |
upgrading remaining job runners to PHP 7.2.22 T230024 |
[production] |
| 12:44 |
<Urbanecm> |
EU SWAT wmf patch ongoing, testing with mwdebug1002 |
[production] |