|
2017-10-04
§
|
| 11:49 |
<moritzm> |
upgrading job runners mw1161-mw1167 to HHVM 3.18.5 |
[production] |
| 11:34 |
<moritzm> |
installing ghostscript security updates |
[production] |
| 11:13 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight (duration: 00m 47s) |
[production] |
| 10:33 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1087 weight (duration: 00m 51s) |
[production] |
| 10:29 |
<jynus> |
upgrade and reboot labsdb1011 |
[production] |
| 10:22 |
<jynus> |
draining labsdb1011 connections |
[production] |
| 10:18 |
<akosiaris> |
T177374, fully depool wtp1001-wtp1024 |
[production] |
| 10:18 |
<akosiaris@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet |
[production] |
| 10:03 |
<moritzm> |
install libidn security updates |
[production] |
| 10:01 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1087 with low weight (duration: 00m 50s) |
[production] |
| 09:54 |
<_joe_> |
restarting hhvm on canaries (both appservers and canaries) T107128 |
[production] |
| 09:47 |
<moritzm> |
upgrading API app servers mw1189-mw1208 to HHVM 3.18.5 |
[production] |
| 09:45 |
<elukey> |
rolling restart of aqs nodes to pick up the new logstash lvs config |
[production] |
| 09:39 |
<hoo> |
Killed remaining dumpRdf runners on snapshot1007 after two crashed due to the db1087 depool. Auto-restart will do the rest. |
[production] |
| 09:33 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s) |
[production] |
| 09:32 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s) |
[production] |
| 09:25 |
<moritzm> |
installing ocaml security updates on trusty |
[production] |
| 09:16 |
<marostegui> |
Reboot db1087 to pick up new kernel |
[production] |
| 09:09 |
<_joe_> |
restarting hhvm on mwdebug*, T107128 |
[production] |
| 09:07 |
<marostegui> |
Optimize templatelinks and pagelinks tables on db1097 (s4) - T174509 |
[production] |
| 09:06 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1097 - T174509 (duration: 00m 50s) |
[production] |
| 08:41 |
<akosiaris@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet |
[production] |
| 08:40 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 (duration: 10m 05s) |
[production] |
| 08:36 |
<moritzm> |
upgrading app servers mw1180-mw1188, mw1209-mw1220 to HHVM 3.18.5 |
[production] |
| 08:30 |
<mobrovac@tin> |
Started deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 |
[production] |
| 08:28 |
<marostegui> |
Stop MySQL on db2010 as it will be decommissioned - T175685 |
[production] |
| 07:30 |
<jynus> |
stopping s5 replication on dbstore1002 and converting wikidatawiki.wb_terms into TokuDB |
[production] |
| 07:11 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s) |
[production] |
| 07:10 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s) |
[production] |
| 06:53 |
<marostegui> |
Stop MySQL on db1087 to clone db1104 from it - T172679 |
[production] |
| 06:39 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s) |
[production] |
| 06:30 |
<marostegui> |
Optimize templatelinks and pagelinks tables on db1066 - T174509 |
[production] |
| 06:27 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2035 and db2056 - T174509 (duration: 00m 50s) |
[production] |
| 06:21 |
<elukey> |
restart varnish backend on cp3043 |
[production] |
| 06:19 |
<marostegui> |
Optimize pagelinks and templatelinks on dbstore2002 s1 - T174509 |
[production] |
| 06:16 |
<elukey> |
restart varnish backend on cp3041 |
[production] |
| 06:13 |
<marostegui> |
Optimize pagelinks and templatelinks tables on s7 codfw master (db2029) this will generate lag - T174509 |
[production] |
| 05:58 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2034, db2055 db2062 - T174509 (duration: 00m 51s) |
[production] |
| 03:09 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Oct 4 03:09:34 UTC 2017 (duration 7m 15s) |
[production] |
| 03:02 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 20s) |
[production] |
| 02:25 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 44s) |
[production] |
| 00:33 |
<ejegg> |
re-enabled fundraising queue consumers |
[production] |
| 00:29 |
<ejegg> |
re-enabled CiviCRM de-dupe jobs |
[production] |
| 00:24 |
<ejegg> |
re-enabled civicrm cron and ingenico jobs |
[production] |
| 00:20 |
<ejegg> |
re-enabled fundraising stats and export jobs |
[production] |
| 00:16 |
<ejegg> |
re-enabled omnimail import jobs |
[production] |
| 00:10 |
<ejegg> |
re-enabled fundraising audit jobs |
[production] |
| 00:07 |
<ejegg> |
re-enabled pending queue consumer |
[production] |