|
2017-11-29
§
|
| 13:17 |
<awight@tin> |
Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) |
[production] |
| 12:34 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Setup s8 on eqiad, with no wikis (duration: 00m 48s) |
[production] |
| 12:24 |
<jynus> |
scap pull on mwdebug1001 |
[production] |
| 12:23 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Pool db1096:3315 (duration: 00m 48s) |
[production] |
| 12:01 |
<marostegui> |
Deploy schema change on db1072 (sanitarium master) on s3 with replication enabled to replicate to labs - T174569 |
[production] |
| 11:30 |
<elukey> |
reboot kafka1001 for kernel + jvm updates - T179943 |
[production] |
| 10:15 |
<akosiaris> |
disable puppet on oresrdb* for merging https://gerrit.wikimedia.org/r/#/c/394022/. T181563 |
[production] |
| 09:50 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 48s) |
[production] |
| 09:46 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 49s) |
[production] |
| 09:39 |
<godog> |
upload cassandra-tools-wmf 1.0.2-1 - T181438 |
[production] |
| 09:28 |
<mobrovac@tin> |
Finished deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 (duration: 04m 03s) |
[production] |
| 09:24 |
<mobrovac@tin> |
Started deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 |
[production] |
| 09:04 |
<godog> |
bootstrap restbase1007-c - T179422 |
[production] |
| 08:41 |
<moritzm> |
uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511) |
[production] |
| 08:17 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s) |
[production] |
| 07:54 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3318 and db1055 - T178359 (duration: 00m 48s) |
[production] |
| 07:33 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1099:3318 traffic - T178359 (duration: 00m 45s) |
[production] |
| 07:17 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1099:3318 with low weight - T178359 (duration: 00m 45s) |
[production] |
| 06:51 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1055 - T178359 (duration: 00m 48s) |
[production] |
| 06:42 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 48s) |
[production] |
| 06:42 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 49s) |
[production] |
| 06:29 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1055 with lower weight - T178359 (duration: 00m 50s) |
[production] |
| 06:17 |
<ebernhardson@tin> |
Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 04m 27s) |
[production] |
| 06:12 |
<ebernhardson@tin> |
Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) |
[production] |
| 03:39 |
<ejegg> |
reduced CiviMail sample rate to 0.35 |
[production] |
| 03:09 |
<ejegg> |
reduced donation queue consumer time limit from 70 to 60 seconds, increased ty mail batch time limit from 45 to 55 seconds |
[production] |
| 02:50 |
<ejegg> |
reduced donation queue consumer time limit from 75 to 70 seconds, increased ty mail batch time limit from 40 to 45 seconds |
[production] |
| 02:44 |
<ejegg> |
reduced CiviMail record creation rate from 100% to 50% |
[production] |
| 02:24 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.8) (duration: 06m 08s) |
[production] |
| 01:25 |
<mutante> |
snapshot1001 - closed idle screen session |
[production] |
| 01:22 |
<mutante> |
analytics1003 - closed idle screen session |
[production] |
| 01:21 |
<mutante> |
mw1276 - run "scap pull" to get in sync after hardware issue, then pooled again (T181397) |
[production] |
| 01:09 |
<mutante> |
restarting ircecho - it stopped talking |
[production] |
| 01:07 |
<mutante> |
forcing puppet run on all labvirt* machines to clean out Icinga alerts |
[production] |
| 00:44 |
<awight@tin> |
Synchronized php-1.31.0-wmf.10/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 49s) |
[production] |
| 00:33 |
<reedy@tin> |
Synchronized php-1.31.0-wmf.10/includes/logging/LogPager.php: Fix fatal on Special:Log T181565 (duration: 00m 48s) |
[production] |
| 00:32 |
<awight@tin> |
Synchronized php-1.31.0-wmf.8/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 50s) |
[production] |
| 00:18 |
<Jamesofur> |
deleted 6 archived files from servers for legal compliance |
[production] |
| 00:08 |
<ejegg> |
updated fundrasing dashboard from df94248ccf3cc92d9baae7a5dfacca0db6849420 to 6ee656759561d524c1ed8a15ac4da4d0fce887a7 |
[production] |
|
2017-11-28
§
|
| 23:58 |
<hoo> |
Ran scap pull on mwdebug1001 after T181385 related testing |
[production] |
| 23:07 |
<akosiaris@tin> |
Synchronized wmf-config/CommonSettings.php: T181538 (duration: 00m 49s) |
[production] |
| 22:31 |
<akosiaris@tin> |
Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 49s) |
[production] |
| 22:31 |
<akosiaris> |
deploy wmf-config/CommonSettings.php for ORES internal discovery URL, https://gerrit.wikimedia.org/r/#/c/393924/ T181538 |
[production] |
| 21:41 |
<akosiaris> |
disable ORES queue redis persistency by config set appendonly no on oresrdb1001 |
[production] |
| 21:24 |
<hoo> |
Manually killed all remaining Wikidata TTL (RDF) dumpers on snapshot1007. Some shards failed due to the db1110 depool. |
[production] |
| 21:23 |
<hoo> |
Manually killed all remaining Wikidata JSON dumpers on snapshot1007. Some shards failed due to the db1110 depool. |
[production] |
| 20:56 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: removing aawiki from group0 |
[production] |
| 20:42 |
<gehel> |
repooling elastic2004 after RAID controller maintenance - T181412 |
[production] |
| 20:42 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.10 |
[production] |
| 20:28 |
<mutante> |
forcing puppet run on cache misc to revert "failover ORES to codfw" |
[production] |