|
2017-10-06
§
|
| 15:30 |
<moritzm> |
repooling mw2145 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it |
[production] |
| 15:25 |
<moritzm> |
repooling mw2203 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it |
[production] |
| 12:54 |
<moritzm> |
repooling mw2248-mw2250 (job runners). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool |
[production] |
| 11:46 |
<bblack> |
cp1* (eqiad caches) - upgrade to nginx-1.13.5-1+wmf1~jessie1 to match all the other sites |
[production] |
| 11:34 |
<moritzm> |
repooling mw2244/mw2245 (image scalers). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool |
[production] |
| 11:21 |
<moritzm> |
installing gdk-pixbuf security updates |
[production] |
| 11:10 |
<elukey> |
rolling restart of all the druid daemons on druid100[1-6] to pick up new logging changes |
[production] |
| 11:03 |
<moritzm> |
installing git security updates on trusty (Debian already fixed) |
[production] |
| 10:51 |
<moritzm> |
upgrading HHVM on script runners to HHVM 3.18.5 |
[production] |
| 10:47 |
<moritzm> |
upgrade remaining video scalers in codfw to HHVM 3.18.5 |
[production] |
| 10:43 |
<elukey> |
restart replication (mysql+ eventlogging_sync) on dbstore1002 after mysql restart |
[production] |
| 10:17 |
<moritzm> |
upgrade remaining job runners in codfw to HHVM 3.18.5 |
[production] |
| 09:53 |
<elukey> |
stop replication (eventlogging_sync + mysql) on dbstore1002 as prep step for mysql restart |
[production] |
| 09:17 |
<hashar> |
Restarted the CI Jenkins for plugin upgrades |
[production] |
| 09:01 |
<moritzm> |
upgrade remaining app servers in codfw to HHVM 3.18.5 |
[production] |
| 08:40 |
<moritzm> |
upgrade remaining image scalers in codfw to HHVM 3.18.5 |
[production] |
| 08:09 |
<ema> |
upgrade pybal to 1.14.0 on eqiad primary LVSs |
[production] |
| 07:48 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1076 original weight - T174054 (duration: 00m 48s) |
[production] |
| 07:23 |
<ema> |
upgrade pybal to 1.14.0 on eqiad secondary LVSs |
[production] |
| 07:18 |
<moritzm> |
upgrade remaining API servers in codfw to HHVM 3.18.5 |
[production] |
| 07:11 |
<legoktm> |
live hacking on mwdebug1002 |
[production] |
| 07:05 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s) |
[production] |
| 06:37 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s) |
[production] |
| 06:35 |
<ema> |
set max_connections=0 on cp3032 varnish-be |
[production] |
| 06:23 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s) |
[production] |
| 06:21 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s) |
[production] |
| 06:00 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight - T174054 (duration: 00m 46s) |
[production] |
| 05:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1053 - T174509 (duration: 00m 48s) |
[production] |
| 05:35 |
<marostegui> |
Stop MySQL on db1076 for an upgrade |
[production] |
|
2017-10-05
§
|
| 23:40 |
<ejegg> |
updated CiviCRM from cca1a9d90ca41fe9fc75c12095225220dcc43226 to 197b22dce5a6c60a6e3697f5b9ba49de536d0bbb |
[production] |
| 23:37 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.2/resources/lib/jquery/jquery.migrate.js: sync https://gerrit.wikimedia.org/r/#/c/382530 and https://gerrit.wikimedia.org/r/#/c/382529 (duration: 00m 47s) |
[production] |
| 23:34 |
<XioNoX> |
upgrading ps1-d2-eqiad.mgmt.eqiad.wmnet (unused) |
[production] |
| 23:23 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/BaseInterwikiResolver.php: sync https://gerrit.wikimedia.org/r/#/c/382625/ (duration: 00m 47s) |
[production] |
| 23:21 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT https://gerrit.wikimedia.org/r/#/c/382611/ (duration: 00m 47s) |
[production] |
| 22:47 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/: Sync CirrusSearch extension again for good measure (duration: 00m 51s) |
[production] |
| 22:43 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: Sync CirrusSearch extension again for good measure (duration: 00m 57s) |
[production] |
| 22:39 |
<twentyafterfour@tin> |
Synchronized wmf-config/CirrusSearch-common.php: sync https://gerrit.wikimedia.org/r/#/c/382474/ (duration: 00m 47s) |
[production] |
| 22:37 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/: deploy https://gerrit.wikimedia.org/r/#/c/382618/ (duration: 00m 58s) |
[production] |
| 22:13 |
<awight@tin> |
Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 19s) |
[production] |
| 22:13 |
<awight@tin> |
Started deploy [ores/deploy@42c5663]: Cause ORES service restart |
[production] |
| 22:04 |
<bd808@tin> |
Finished deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681) (duration: 00m 33s) |
[production] |
| 22:04 |
<bd808@tin> |
Started deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681) |
[production] |
| 21:35 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358 |
[production] |
| 21:32 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: sync CirrusSearch to deploy https://gerrit.wikimedia.org/r/#/c/382605/ refs T177535 (duration: 01m 06s) |
[production] |
| 21:22 |
<ejegg> |
updated CiviCRM from 59e3e00ecbaa3557fd32dc49259a1ae12edf7414 to cca1a9d90ca41fe9fc75c12095225220dcc43226 |
[production] |
| 21:13 |
<twentyafterfour> |
Getting the train back on track: moving to wmf.2 as soon as https://gerrit.wikimedia.org/r/#/c/382605/1 merges |
[production] |
| 20:47 |
<twentyafterfour> |
1.31.0-wmf.2 is now blocked by T177535 |
[production] |
| 20:28 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.1 refs T174358 |
[production] |
| 20:26 |
<twentyafterfour> |
rolling back to 1.31.0-wmf.1 |
[production] |
| 20:11 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358 |
[production] |