|
2017-11-29
§
|
| 06:12 |
<ebernhardson@tin> |
Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) |
[production] |
| 03:39 |
<ejegg> |
reduced CiviMail sample rate to 0.35 |
[production] |
| 03:09 |
<ejegg> |
reduced donation queue consumer time limit from 70 to 60 seconds, increased ty mail batch time limit from 45 to 55 seconds |
[production] |
| 02:50 |
<ejegg> |
reduced donation queue consumer time limit from 75 to 70 seconds, increased ty mail batch time limit from 40 to 45 seconds |
[production] |
| 02:44 |
<ejegg> |
reduced CiviMail record creation rate from 100% to 50% |
[production] |
| 02:24 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.8) (duration: 06m 08s) |
[production] |
| 01:25 |
<mutante> |
snapshot1001 - closed idle screen session |
[production] |
| 01:22 |
<mutante> |
analytics1003 - closed idle screen session |
[production] |
| 01:21 |
<mutante> |
mw1276 - run "scap pull" to get in sync after hardware issue, then pooled again (T181397) |
[production] |
| 01:09 |
<mutante> |
restarting ircecho - it stopped talking |
[production] |
| 01:07 |
<mutante> |
forcing puppet run on all labvirt* machines to clean out Icinga alerts |
[production] |
| 00:44 |
<awight@tin> |
Synchronized php-1.31.0-wmf.10/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 49s) |
[production] |
| 00:33 |
<reedy@tin> |
Synchronized php-1.31.0-wmf.10/includes/logging/LogPager.php: Fix fatal on Special:Log T181565 (duration: 00m 48s) |
[production] |
| 00:32 |
<awight@tin> |
Synchronized php-1.31.0-wmf.8/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 50s) |
[production] |
| 00:18 |
<Jamesofur> |
deleted 6 archived files from servers for legal compliance |
[production] |
| 00:08 |
<ejegg> |
updated fundrasing dashboard from df94248ccf3cc92d9baae7a5dfacca0db6849420 to 6ee656759561d524c1ed8a15ac4da4d0fce887a7 |
[production] |
|
2017-11-28
§
|
| 23:58 |
<hoo> |
Ran scap pull on mwdebug1001 after T181385 related testing |
[production] |
| 23:07 |
<akosiaris@tin> |
Synchronized wmf-config/CommonSettings.php: T181538 (duration: 00m 49s) |
[production] |
| 22:31 |
<akosiaris@tin> |
Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 49s) |
[production] |
| 22:31 |
<akosiaris> |
deploy wmf-config/CommonSettings.php for ORES internal discovery URL, https://gerrit.wikimedia.org/r/#/c/393924/ T181538 |
[production] |
| 21:41 |
<akosiaris> |
disable ORES queue redis persistency by config set appendonly no on oresrdb1001 |
[production] |
| 21:24 |
<hoo> |
Manually killed all remaining Wikidata TTL (RDF) dumpers on snapshot1007. Some shards failed due to the db1110 depool. |
[production] |
| 21:23 |
<hoo> |
Manually killed all remaining Wikidata JSON dumpers on snapshot1007. Some shards failed due to the db1110 depool. |
[production] |
| 20:56 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: removing aawiki from group0 |
[production] |
| 20:42 |
<gehel> |
repooling elastic2004 after RAID controller maintenance - T181412 |
[production] |
| 20:42 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.10 |
[production] |
| 20:28 |
<mutante> |
forcing puppet run on cache misc to revert "failover ORES to codfw" |
[production] |
| 20:19 |
<demon@tin> |
Synchronized scap/plugins/prep.py: no-op (duration: 00m 48s) |
[production] |
| 20:16 |
<demon@tin> |
Synchronized dblists/group0.dblist: adding some new wikis (duration: 00m 48s) |
[production] |
| 18:57 |
<demon@tin> |
Finished scap: bootstrap wmf.10 (duration: 35m 20s) |
[production] |
| 18:51 |
<demon@tin> |
Finished deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes (duration: 00m 09s) |
[production] |
| 18:51 |
<demon@tin> |
Started deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes |
[production] |
| 18:40 |
<akosiaris> |
revert weight changes for scb1001, scb1002 T181835 |
[production] |
| 18:39 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=10; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores']) |
[production] |
| 18:39 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=10; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores']) |
[production] |
| 18:35 |
<awight@tin> |
Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2) (duration: 04m 30s) |
[production] |
| 18:32 |
<akosiaris> |
force puppet run on cache::misc boxes T181538 |
[production] |
| 18:30 |
<awight@tin> |
Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2) |
[production] |
| 18:28 |
<urandom> |
(re)bootstrapping cassandra, restbase1007-b - T179422 |
[production] |
| 18:22 |
<demon@tin> |
Started scap: bootstrap wmf.10 |
[production] |
| 18:18 |
<awight@tin> |
Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (duration: 01m 41s) |
[production] |
| 18:17 |
<awight@tin> |
Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster |
[production] |
| 18:00 |
<akosiaris> |
force stop celery-ores-worker on scb1001 |
[production] |
| 17:58 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=5; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores']) |
[production] |
| 17:58 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=5; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores']) |
[production] |
| 17:46 |
<urandom> |
decommissioning cassandra, restbase1007-b - T179422 |
[production] |
| 17:42 |
<urandom> |
restart cassandra, restbase1007, to pickup logstash java deps - T179422 |
[production] |
| 16:45 |
<ejegg> |
updated fundraising dashboard from d8c86e7a2e144e3a665b444e68471f6b864c9d01 to df94248ccf3cc92d9baae7a5dfacca0db6849420 |
[production] |
| 16:25 |
<bblack> |
mw1329 boot to PXE (should come up with new .66 IP) |
[production] |
| 15:39 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: depool db1110 (duration: 00m 44s) |
[production] |