|
2017-12-06
§
|
| 10:55 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet |
[production] |
| 10:55 |
<moritzm> |
reimaging mw2119 to stretch |
[production] |
| 10:45 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801 (duration: 06m 02s) |
[production] |
| 10:39 |
<mobrovac@tin> |
Started deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801 |
[production] |
| 09:34 |
<gehel> |
shuttting down logstash / elasticsearch on logstash100[123] in preparation for decommission -T175830 |
[production] |
| 08:05 |
<moritzm> |
reimaging mw2118 to stretch (now for real, yesterday's reimage logged to SAL was interrupted) |
[production] |
| 07:33 |
<bblack> |
cp4024 - backend restart, mailbox lag |
[production] |
| 02:39 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 53s) |
[production] |
| 01:01 |
<eileen> |
update process-control to 55529ea12d9f16c259fa3 - threshold for $500+ dedupe was too low |
[production] |
| 00:36 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:395584|Enable description usage tracking for all wikis except commons]] T106287 (duration: 00m 48s) |
[production] |
| 00:27 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:395139|Enable RemexHTML on wikis with zero high priority linter errors]] T182042 (duration: 00m 51s) |
[production] |
|
2017-12-05
§
|
| 23:07 |
<eileen> |
update civicrm from d2c70d2b953a90fcadf20cc01bf90c4a648ccbf9 to 85263f625763a454b2664062154c8b791e5a6838 (Major gifts address - choose latest) |
[production] |
| 22:29 |
<eileen> |
update process_control to 3522186 renable jobs to fetch recipient data |
[production] |
| 22:13 |
<mutante> |
restbase1010 failed at reboot with P6431 , after a cold start (power off, power on) it came back though :) (T178177 T141756) |
[production] |
| 22:00 |
<mutante> |
restbase1010 - rebooting for firmware upgrade |
[production] |
| 21:58 |
<mutante> |
restbase1010 - upgraded HP firmware (Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]) T141756 T178177 |
[production] |
| 21:56 |
<urandom> |
draining cassandra instances, restbase1010 - T178177 |
[production] |
| 21:52 |
<mutante> |
restbase1010 - upgrading firmware - Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ] |
[production] |
| 21:31 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 shenanigans / wmf.11 |
[production] |
| 21:12 |
<demon@tin> |
Finished scap: bootstrap wmf.11 (duration: 51m 01s) |
[production] |
| 20:50 |
<twentyafterfour> |
phabricator: running `sudo bin/garbage collect --collector search.ferret.ngram` |
[production] |
| 20:29 |
<twentyafterfour> |
phabricator: running `sudo bin/search ngrams --threshold 0.2` |
[production] |
| 20:21 |
<demon@tin> |
Started scap: bootstrap wmf.11 |
[production] |
| 19:50 |
<mutante> |
forcing puppet ron on ores eqiad to reduce number of celery workers used for stress test |
[production] |
| 19:37 |
<ppchelko@tin> |
Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 (duration: 01m 29s) |
[production] |
| 19:36 |
<ppchelko@tin> |
Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 |
[production] |
| 19:20 |
<gehel> |
moving eventlogging collection by logstash from logstash1003 to logstash1007, no messages **should** be lost - T175830 |
[production] |
| 18:24 |
<ppchelko@tin> |
Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 (duration: 00m 14s) |
[production] |
| 18:24 |
<ppchelko@tin> |
Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 |
[production] |
| 17:27 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet |
[production] |
| 15:56 |
<awight> |
beginning stress test on ores* (non-production) |
[production] |
| 15:54 |
<awight@tin> |
Finished deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] (duration: 01m 01s) |
[production] |
| 15:53 |
<awight@tin> |
Started deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] |
[production] |
| 15:15 |
<ottomata> |
restarrting kafka-jumbo brokers, applying SSL (downtime scheduled) |
[production] |
| 15:03 |
<urandom> |
bootstrapping cassandra, restbase1014-c - T179422 |
[production] |
| 14:54 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 43s) |
[production] |
| 14:41 |
<zeljkof> |
EU SWAT finished |
[production] |
| 14:40 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:395540|Revert "Add category collation for sewiki" (T181503)]] (duration: 00m 44s) |
[production] |
| 14:15 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:395009|Enable RemexHTML on itwiki and dewiki (T181188 T181190)]] (duration: 00m 43s) |
[production] |
| 13:39 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1076 weight (duration: 00m 44s) |
[production] |
| 13:14 |
<moritzm> |
reimaging mw2118/mw2119 (video scalers) to stretch |
[production] |
| 12:21 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1034 (duration: 00m 43s) |
[production] |
| 12:13 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 44s) |
[production] |
| 11:58 |
<marostegui> |
Upgrade MariaDB and kernel on db1076 |
[production] |
| 11:53 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s) |
[production] |
| 10:53 |
<moritzm> |
rebooting meitnerium/archiva.wikimedia.org for update to 4.9.51 |
[production] |
| 10:46 |
<moritzm> |
rebooting einsteinium (icinga.wikimedia.org) for update to 4.9,51 |
[production] |
| 10:45 |
<elukey> |
reboot druid1003 for kernel+jvm updates - T179943 |
[production] |
| 10:33 |
<addshore@tin> |
Synchronized wmf-config/extension-list: [[gerrit:394283|extension-list extension.json entrypoint for ArticlePlaceholder]] (duration: 00m 43s) |
[production] |
| 10:33 |
<moritzm> |
rebooting tegmen for update to 4.9,51 |
[production] |