|
2017-11-29
§
|
| 23:20 |
<eileen> |
update civicrm from a1022cf2a64c048382a1ae5da7f1f81da0336d62 to 0f95f3e37f9651a5fd262bf1f5e39929abee0d88 (Benevity import fix) |
[production] |
| 22:23 |
<hashar> |
Nodepool had some troubles spawning new instances from 21:09 to 21:36, and took a while to recover. Issue similar to T170492#3581822 |
[production] |
| 21:51 |
<SMalyshev> |
starting wikidata reindex (T181426) |
[production] |
| 21:23 |
<no_justification> |
docroot sync was for If1afa59a |
[production] |
| 21:21 |
<demon@tin> |
Synchronized docroot/: (no justification provided) (duration: 00m 49s) |
[production] |
| 20:29 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10 |
[production] |
| 20:26 |
<demon@tin> |
Synchronized php: symlink bump (duration: 00m 48s) |
[production] |
| 20:05 |
<urandom> |
restarting cassandra bootstrap of restbase1012-a (T179422) |
[production] |
| 19:55 |
<mutante> |
restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865) |
[production] |
| 19:50 |
<herron> |
beginning rolling cut over of eqiad scb hosts to codfw puppet 4 masters |
[production] |
| 19:15 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/394059/3 (duration: 00m 49s) |
[production] |
| 18:57 |
<herron> |
beginning cutover of codfw lvs2* systems to codfw puppet 4 masters. first standby nodes, then active nodes |
[production] |
| 18:12 |
<marostegui> |
Compress s5 on db1096 - T178359 |
[production] |
| 18:10 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 53s) |
[production] |
| 18:05 |
<awight@tin> |
Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 01m 11s) |
[production] |
| 18:04 |
<awight@tin> |
Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster |
[production] |
| 18:03 |
<herron> |
beginning cut over of codfw cp2* servers to codfw puppet 4 masters |
[production] |
| 17:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1110 with low load (duration: 00m 48s) |
[production] |
| 17:40 |
<godog> |
bootstrapping restbase1012-a - T179422 |
[production] |
| 17:19 |
<herron> |
beginning cut over of cp200[12] to codfw puppet 4 masters |
[production] |
| 17:16 |
<ejegg> |
increased time limit for donation queue consumer from 60 to 70 seconds, reduced thank you job from 55 to 45 seconds |
[production] |
| 17:03 |
<ejegg> |
re-enabled Civi jobs |
[production] |
| 16:58 |
<ejegg> |
disabled Civi jobs for Deadlock-retry update |
[production] |
| 16:46 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Update db1110 ip (duration: 00m 50s) |
[production] |
| 16:43 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Update db1110 ip (duration: 00m 49s) |
[production] |
| 15:02 |
<chasemp> |
purge old qcow2 images from under /home on labtestcontrol2001 |
[production] |
| 15:00 |
<awight> |
Restarting ORES celery workers manually, T181538 |
[production] |
| 14:39 |
<_joe_> |
reloading apache on puppetmasters to pick up the configuration changes |
[production] |
| 14:37 |
<awight@tin> |
Started restart [ores/deploy@e58bfbf]: Restart ORES services (take 2), T181538 |
[production] |
| 14:36 |
<elukey> |
reboot druid100[456] for jvm+kernel updates - T179943 |
[production] |
| 14:35 |
<awight@tin> |
Finished deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 (duration: 00m 16s) |
[production] |
| 14:35 |
<awight@tin> |
Started deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 |
[production] |
| 14:25 |
<zeljkof> |
EU SWAT finished |
[production] |
| 14:18 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:393722|Define throttle rule (T181367)]] (duration: 00m 49s) |
[production] |
| 14:14 |
<herron> |
beginning cut over of cache::misc and codfw cache::canary cp servers to codfw puppet4 masters |
[production] |
| 14:13 |
<moritzm> |
rebooting neodymium for kernel update to 4.9.51 |
[production] |
| 14:12 |
<godog> |
reimage restbase1012 - T179422 |
[production] |
| 14:10 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT T180291 T180128 [[gerrit:393369|AdvancedSearch for arwiki]] (duration: 00m 49s) |
[production] |
| 14:07 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT T180128 [[gerrit:390387|AdvancedSearch for dewiki]] (duration: 00m 50s) |
[production] |
| 13:34 |
<awight@tin> |
Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) (duration: 16m 49s) |
[production] |
| 13:18 |
<elukey> |
reboot kafka100[23] for jvm+kernel updates - T179943 |
[production] |
| 13:17 |
<awight@tin> |
Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) |
[production] |
| 12:34 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Setup s8 on eqiad, with no wikis (duration: 00m 48s) |
[production] |
| 12:24 |
<jynus> |
scap pull on mwdebug1001 |
[production] |
| 12:23 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Pool db1096:3315 (duration: 00m 48s) |
[production] |
| 12:01 |
<marostegui> |
Deploy schema change on db1072 (sanitarium master) on s3 with replication enabled to replicate to labs - T174569 |
[production] |
| 11:30 |
<elukey> |
reboot kafka1001 for kernel + jvm updates - T179943 |
[production] |
| 10:15 |
<akosiaris> |
disable puppet on oresrdb* for merging https://gerrit.wikimedia.org/r/#/c/394022/. T181563 |
[production] |