|
2017-11-29
ยง
|
| 18:57 |
<herron> |
beginning cutover of codfw lvs2* systems to codfw puppet 4 masters. first standby nodes, then active nodes |
[production] |
| 18:12 |
<marostegui> |
Compress s5 on db1096 - T178359 |
[production] |
| 18:10 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 53s) |
[production] |
| 18:05 |
<awight@tin> |
Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 01m 11s) |
[production] |
| 18:04 |
<awight@tin> |
Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster |
[production] |
| 18:03 |
<herron> |
beginning cut over of codfw cp2* servers to codfw puppet 4 masters |
[production] |
| 17:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1110 with low load (duration: 00m 48s) |
[production] |
| 17:40 |
<godog> |
bootstrapping restbase1012-a - T179422 |
[production] |
| 17:19 |
<herron> |
beginning cut over of cp200[12] to codfw puppet 4 masters |
[production] |
| 17:16 |
<ejegg> |
increased time limit for donation queue consumer from 60 to 70 seconds, reduced thank you job from 55 to 45 seconds |
[production] |
| 17:03 |
<ejegg> |
re-enabled Civi jobs |
[production] |
| 16:58 |
<ejegg> |
disabled Civi jobs for Deadlock-retry update |
[production] |
| 16:46 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Update db1110 ip (duration: 00m 50s) |
[production] |
| 16:43 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Update db1110 ip (duration: 00m 49s) |
[production] |
| 15:02 |
<chasemp> |
purge old qcow2 images from under /home on labtestcontrol2001 |
[production] |
| 15:00 |
<awight> |
Restarting ORES celery workers manually, T181538 |
[production] |
| 14:39 |
<_joe_> |
reloading apache on puppetmasters to pick up the configuration changes |
[production] |
| 14:37 |
<awight@tin> |
Started restart [ores/deploy@e58bfbf]: Restart ORES services (take 2), T181538 |
[production] |
| 14:36 |
<elukey> |
reboot druid100[456] for jvm+kernel updates - T179943 |
[production] |
| 14:35 |
<awight@tin> |
Finished deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 (duration: 00m 16s) |
[production] |
| 14:35 |
<awight@tin> |
Started deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 |
[production] |
| 14:25 |
<zeljkof> |
EU SWAT finished |
[production] |
| 14:18 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:393722|Define throttle rule (T181367)]] (duration: 00m 49s) |
[production] |
| 14:14 |
<herron> |
beginning cut over of cache::misc and codfw cache::canary cp servers to codfw puppet4 masters |
[production] |
| 14:13 |
<moritzm> |
rebooting neodymium for kernel update to 4.9.51 |
[production] |
| 14:12 |
<godog> |
reimage restbase1012 - T179422 |
[production] |
| 14:10 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT T180291 T180128 [[gerrit:393369|AdvancedSearch for arwiki]] (duration: 00m 49s) |
[production] |
| 14:07 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT T180128 [[gerrit:390387|AdvancedSearch for dewiki]] (duration: 00m 50s) |
[production] |
| 13:34 |
<awight@tin> |
Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) (duration: 16m 49s) |
[production] |
| 13:18 |
<elukey> |
reboot kafka100[23] for jvm+kernel updates - T179943 |
[production] |
| 13:17 |
<awight@tin> |
Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) |
[production] |
| 12:34 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Setup s8 on eqiad, with no wikis (duration: 00m 48s) |
[production] |
| 12:24 |
<jynus> |
scap pull on mwdebug1001 |
[production] |
| 12:23 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Pool db1096:3315 (duration: 00m 48s) |
[production] |
| 12:01 |
<marostegui> |
Deploy schema change on db1072 (sanitarium master) on s3 with replication enabled to replicate to labs - T174569 |
[production] |
| 11:30 |
<elukey> |
reboot kafka1001 for kernel + jvm updates - T179943 |
[production] |
| 10:15 |
<akosiaris> |
disable puppet on oresrdb* for merging https://gerrit.wikimedia.org/r/#/c/394022/. T181563 |
[production] |
| 09:50 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 48s) |
[production] |
| 09:46 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 49s) |
[production] |
| 09:39 |
<godog> |
upload cassandra-tools-wmf 1.0.2-1 - T181438 |
[production] |
| 09:28 |
<mobrovac@tin> |
Finished deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 (duration: 04m 03s) |
[production] |
| 09:24 |
<mobrovac@tin> |
Started deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 |
[production] |
| 09:04 |
<godog> |
bootstrap restbase1007-c - T179422 |
[production] |
| 08:41 |
<moritzm> |
uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511) |
[production] |
| 08:17 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s) |
[production] |
| 07:54 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3318 and db1055 - T178359 (duration: 00m 48s) |
[production] |
| 07:33 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1099:3318 traffic - T178359 (duration: 00m 45s) |
[production] |
| 07:17 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1099:3318 with low weight - T178359 (duration: 00m 45s) |
[production] |
| 06:51 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1055 - T178359 (duration: 00m 48s) |
[production] |
| 06:42 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 48s) |
[production] |