|
2019-09-10
§
|
| 10:34 |
<@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 10:34 |
<@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 10:32 |
<vgutierrez> |
repool cp5001 with ats-tls collecting memory usage details every hour - T232298 |
[production] |
| 09:56 |
<elukey> |
restart archiva on archiva1001 - UI not working (probably due to connections to maven central being stuck) |
[production] |
| 09:50 |
<moritzm> |
installing ghostscript security updates on jessie |
[production] |
| 09:37 |
<moritzm> |
added jbond as chanserv ops for #wikimedia-operations |
[production] |
| 08:08 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:03 |
<hashar> |
Updated Phan jobs for T232413 | https://gerrit.wikimedia.org/r/#/c/integration/config/+/535507 phan: stop installing MediaWiki |
[releng] |
| 07:42 |
<moritzm> |
reimaging mw2231 after hardware maintenance T231192 |
[production] |
| 07:21 |
<moritzm> |
iron.wikimedia.org is no longer a bastion host |
[production] |
| 06:57 |
<moritzm> |
upgrading snapshot* to PHP 7.2.22 T230024 |
[production] |
| 05:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1073 from config T231892 (duration: 00m 54s) |
[production] |
| 05:45 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db1073 from config T231892 (duration: 00m 55s) |
[production] |
| 05:35 |
<marostegui> |
Stop MySQL on db2047 T231852 |
[production] |
| 05:35 |
<marostegui> |
Remove db2047 from tendril and zarcillo - T231852 |
[production] |
| 05:33 |
<urandom> |
decommissioning Cassandra, restbase-dev1005-b -- T224554 |
[production] |
| 05:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1104 into API T230762', diff saved to https://phabricator.wikimedia.org/P9071 and previous config saved to /var/cache/conftool/dbconfig/20190910-051529-marostegui.json |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1109 to s8 master and remove read-only from s8 T227062', diff saved to https://phabricator.wikimedia.org/P9070 and previous config saved to /var/cache/conftool/dbconfig/20190910-050213-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s8 as read-only for maintenance T230762', diff saved to https://phabricator.wikimedia.org/P9069 and previous config saved to /var/cache/conftool/dbconfig/20190910-050046-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s8 failover from db1104 to db1109 - T227062 |
[production] |
| 04:46 |
<vgutierrez> |
depool cp5001 for memory leak debugging on ATS - T232298 |
[production] |
| 04:23 |
<marostegui> |
Start topology changes on s8, connect everything under db1109 - T230762 |
[production] |
| 04:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1109 with weight 0 and depool it from API T230762', diff saved to https://phabricator.wikimedia.org/P9068 and previous config saved to /var/cache/conftool/dbconfig/20190910-042243-marostegui.json |
[production] |
| 04:18 |
<marostegui> |
Start s8 (wikidata) pre switchover steps T230762 |
[production] |
| 01:12 |
<bd808> |
Added tools-elastic-01.tools.eqiad.wmflabs to elasticsearch.servers list |
[tools.stashbot] |
| 01:12 |
<bd808> |
Changed LDAP server setting to point to ldap-ro.eqiad.wikimedia.org |
[tools.stashbot] |
| 01:11 |
<bd808> |
Updated to 3b5f6d0 (Write severe errors to a file on disk) |
[tools.stashbot] |
| 01:01 |
<andrewbogott> |
removed 'Postgresql::Master' roles from both VMs as it was causing a puppet failure |
[wikibrain] |
| 00:59 |
<krinkle@deploy1001> |
Finished deploy [performance/navtiming@f2a0863]: (no justification provided) (duration: 00m 05s) |
[production] |
| 00:59 |
<krinkle@deploy1001> |
Started deploy [performance/navtiming@f2a0863]: (no justification provided) |
[production] |
| 00:57 |
<Krinkle> |
krinkle@deploy1001: Deploy performance/navtiming f2a0863b9e4774140463a79d08051814c0d21116 - T226539 |
[production] |
| 00:41 |
<urandom> |
decommissioning Cassandra, restbase-dev1005-a -- T224554 |
[production] |
|
2019-09-09
§
|
| 23:44 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.21/skins/MinervaNeue/: T232260 (duration: 00m 57s) |
[production] |
| 22:44 |
<bstorm_> |
uncordoned tools-worker-1030 and tools-worker-1038 |
[tools] |
| 22:28 |
<ejegg> |
updated payments-wiki from 51d9ed79b6 to 15baf7f58b |
[production] |
| 22:08 |
<halfak> |
deleting ores-worker-03 because it didn't work |
[ores] |
| 22:03 |
<jeh> |
schedule icingia downtime on nova-fullstack VM check during puppet maintenance |
[openstack] |
| 22:00 |
<JeanFred> |
Deploy latest from Git master: 109be22 (T231727) |
[tools.heritage] |
| 21:50 |
<halfak> |
configured ores-worker-03 with role::labs::ores::worker |
[ores] |
| 21:42 |
<JeanFred> |
Deploy latest from Git master: 0a4fbae, dff30f0 (T232394, T138668) |
[tools.heritage] |
| 20:50 |
<urandom> |
bootstrapping Cassandra, restbase-dev1004-b -- T224554 |
[production] |
| 19:48 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 05m 45s) |
[production] |
| 19:42 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 |
[production] |
| 19:41 |
<mdholloway> |
mobileapps deployment failed repooling canary (scb2001); retrying |
[production] |
| 19:40 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 02m 59s) |
[production] |
| 19:37 |
<XioNoX> |
fix eqsin CF tunnel missconfig |
[production] |
| 19:37 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 |
[production] |
| 18:23 |
<Krenair> |
Started mariadb service on cloudinfra-db02 - was not running for some reason |
[cloudinfra] |
| 17:56 |
<andrewbogott> |
disabling puppet on labpuppetmaster1001 as part of T171188 |
[production] |