|
2019-09-10
§
|
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s8 as read-only for maintenance T230762', diff saved to https://phabricator.wikimedia.org/P9069 and previous config saved to /var/cache/conftool/dbconfig/20190910-050046-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s8 failover from db1104 to db1109 - T227062 |
[production] |
| 04:46 |
<vgutierrez> |
depool cp5001 for memory leak debugging on ATS - T232298 |
[production] |
| 04:23 |
<marostegui> |
Start topology changes on s8, connect everything under db1109 - T230762 |
[production] |
| 04:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1109 with weight 0 and depool it from API T230762', diff saved to https://phabricator.wikimedia.org/P9068 and previous config saved to /var/cache/conftool/dbconfig/20190910-042243-marostegui.json |
[production] |
| 04:18 |
<marostegui> |
Start s8 (wikidata) pre switchover steps T230762 |
[production] |
| 01:12 |
<bd808> |
Added tools-elastic-01.tools.eqiad.wmflabs to elasticsearch.servers list |
[tools.stashbot] |
| 01:12 |
<bd808> |
Changed LDAP server setting to point to ldap-ro.eqiad.wikimedia.org |
[tools.stashbot] |
| 01:11 |
<bd808> |
Updated to 3b5f6d0 (Write severe errors to a file on disk) |
[tools.stashbot] |
| 01:01 |
<andrewbogott> |
removed 'Postgresql::Master' roles from both VMs as it was causing a puppet failure |
[wikibrain] |
| 00:59 |
<krinkle@deploy1001> |
Finished deploy [performance/navtiming@f2a0863]: (no justification provided) (duration: 00m 05s) |
[production] |
| 00:59 |
<krinkle@deploy1001> |
Started deploy [performance/navtiming@f2a0863]: (no justification provided) |
[production] |
| 00:57 |
<Krinkle> |
krinkle@deploy1001: Deploy performance/navtiming f2a0863b9e4774140463a79d08051814c0d21116 - T226539 |
[production] |
| 00:41 |
<urandom> |
decommissioning Cassandra, restbase-dev1005-a -- T224554 |
[production] |
|
2019-09-09
§
|
| 23:44 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.21/skins/MinervaNeue/: T232260 (duration: 00m 57s) |
[production] |
| 22:44 |
<bstorm_> |
uncordoned tools-worker-1030 and tools-worker-1038 |
[tools] |
| 22:28 |
<ejegg> |
updated payments-wiki from 51d9ed79b6 to 15baf7f58b |
[production] |
| 22:08 |
<halfak> |
deleting ores-worker-03 because it didn't work |
[ores] |
| 22:03 |
<jeh> |
schedule icingia downtime on nova-fullstack VM check during puppet maintenance |
[openstack] |
| 22:00 |
<JeanFred> |
Deploy latest from Git master: 109be22 (T231727) |
[tools.heritage] |
| 21:50 |
<halfak> |
configured ores-worker-03 with role::labs::ores::worker |
[ores] |
| 21:42 |
<JeanFred> |
Deploy latest from Git master: 0a4fbae, dff30f0 (T232394, T138668) |
[tools.heritage] |
| 20:50 |
<urandom> |
bootstrapping Cassandra, restbase-dev1004-b -- T224554 |
[production] |
| 19:48 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 05m 45s) |
[production] |
| 19:42 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 |
[production] |
| 19:41 |
<mdholloway> |
mobileapps deployment failed repooling canary (scb2001); retrying |
[production] |
| 19:40 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 02m 59s) |
[production] |
| 19:37 |
<XioNoX> |
fix eqsin CF tunnel missconfig |
[production] |
| 19:37 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 |
[production] |
| 18:23 |
<Krenair> |
Started mariadb service on cloudinfra-db02 - was not running for some reason |
[cloudinfra] |
| 17:56 |
<andrewbogott> |
disabling puppet on labpuppetmaster1001 as part of T171188 |
[production] |
| 17:55 |
<XioNoX> |
push cloudflare tunnel config to cr1-eqsin |
[production] |
| 16:50 |
<papaul> |
replacing Fan kit and power supplies on cr1-codfw |
[production] |
| 15:05 |
<hashar> |
Reloading Zuul for https://gerrit.wikimedia.org/r/535198 Shibboleth disable selenium tests |
[releng] |
| 14:22 |
<urandom> |
bootstrapping Cassandra, restbase-dev1004-a -- T224554 |
[production] |
| 13:51 |
<vgutierrez> |
upgrading ats to 8.0.5-1wm6 on cp5001 - T232298 |
[production] |
| 13:39 |
<vgutierrez> |
uploaded trafficserver 8.0.5-1wm6 to apt.wikimedia.org (stretch) - T232298 |
[production] |
| 13:31 |
<moritzm> |
installing facter update from buster 10.1 point release (T222356) |
[production] |
| 13:15 |
<moritzm> |
upgrading labweb/wikitech to PHP 7.2.22 T230024 |
[production] |
| 13:02 |
<Urbanecm> |
Patch is deployed, deploy1001 should be clear |
[production] |
| 13:01 |
<moritzm> |
upgrading remaining mediawiki app servers (mw1266-mw1275) to PHP 7.2.22 T230024 |
[production] |
| 12:55 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.21/extensions/WikibaseMediaInfo/: ubn patch T231276 (duration: 00m 58s) |
[production] |
| 12:51 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.21/extensions/Wikibase: ubn patch T231276 (duration: 01m 03s) |
[production] |
| 12:48 |
<moritzm> |
upgrading remaining job runners to PHP 7.2.22 T230024 |
[production] |
| 12:44 |
<Urbanecm> |
EU SWAT wmf patch ongoing, testing with mwdebug1002 |
[production] |
| 12:41 |
<ema> |
lvs1015 (primary): restart pybal to add service restbase-ssl T210411 |
[production] |
| 12:36 |
<ema> |
lvs2003 (primary): restart pybal to add service restbase-ssl T210411 |
[production] |
| 12:32 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=restbase-ssl,dc=eqiad |
[production] |
| 12:30 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=restbase-ssl,dc=codfw |
[production] |
| 12:29 |
<elukey> |
restart archiva again to debug download artifact issue |
[production] |