|
2017-12-05
ยง
|
| 22:13 |
<mutante> |
restbase1010 failed at reboot with P6431 , after a cold start (power off, power on) it came back though :) (T178177 T141756) |
[production] |
| 22:00 |
<mutante> |
restbase1010 - rebooting for firmware upgrade |
[production] |
| 21:58 |
<mutante> |
restbase1010 - upgraded HP firmware (Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]) T141756 T178177 |
[production] |
| 21:56 |
<urandom> |
draining cassandra instances, restbase1010 - T178177 |
[production] |
| 21:52 |
<mutante> |
restbase1010 - upgrading firmware - Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ] |
[production] |
| 21:31 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 shenanigans / wmf.11 |
[production] |
| 21:12 |
<demon@tin> |
Finished scap: bootstrap wmf.11 (duration: 51m 01s) |
[production] |
| 20:50 |
<twentyafterfour> |
phabricator: running `sudo bin/garbage collect --collector search.ferret.ngram` |
[production] |
| 20:29 |
<twentyafterfour> |
phabricator: running `sudo bin/search ngrams --threshold 0.2` |
[production] |
| 20:21 |
<demon@tin> |
Started scap: bootstrap wmf.11 |
[production] |
| 19:50 |
<mutante> |
forcing puppet ron on ores eqiad to reduce number of celery workers used for stress test |
[production] |
| 19:37 |
<ppchelko@tin> |
Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 (duration: 01m 29s) |
[production] |
| 19:36 |
<ppchelko@tin> |
Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 |
[production] |
| 19:20 |
<gehel> |
moving eventlogging collection by logstash from logstash1003 to logstash1007, no messages **should** be lost - T175830 |
[production] |
| 18:24 |
<ppchelko@tin> |
Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 (duration: 00m 14s) |
[production] |
| 18:24 |
<ppchelko@tin> |
Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 |
[production] |
| 17:27 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet |
[production] |
| 15:56 |
<awight> |
beginning stress test on ores* (non-production) |
[production] |
| 15:54 |
<awight@tin> |
Finished deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] (duration: 01m 01s) |
[production] |
| 15:53 |
<awight@tin> |
Started deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] |
[production] |
| 15:15 |
<ottomata> |
restarrting kafka-jumbo brokers, applying SSL (downtime scheduled) |
[production] |
| 15:03 |
<urandom> |
bootstrapping cassandra, restbase1014-c - T179422 |
[production] |
| 14:54 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 43s) |
[production] |
| 14:41 |
<zeljkof> |
EU SWAT finished |
[production] |
| 14:40 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:395540|Revert "Add category collation for sewiki" (T181503)]] (duration: 00m 44s) |
[production] |
| 14:15 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:395009|Enable RemexHTML on itwiki and dewiki (T181188 T181190)]] (duration: 00m 43s) |
[production] |
| 13:39 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1076 weight (duration: 00m 44s) |
[production] |
| 13:14 |
<moritzm> |
reimaging mw2118/mw2119 (video scalers) to stretch |
[production] |
| 12:21 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1034 (duration: 00m 43s) |
[production] |
| 12:13 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 44s) |
[production] |
| 11:58 |
<marostegui> |
Upgrade MariaDB and kernel on db1076 |
[production] |
| 11:53 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s) |
[production] |
| 10:53 |
<moritzm> |
rebooting meitnerium/archiva.wikimedia.org for update to 4.9.51 |
[production] |
| 10:46 |
<moritzm> |
rebooting einsteinium (icinga.wikimedia.org) for update to 4.9,51 |
[production] |
| 10:45 |
<elukey> |
reboot druid1003 for kernel+jvm updates - T179943 |
[production] |
| 10:33 |
<addshore@tin> |
Synchronized wmf-config/extension-list: [[gerrit:394283|extension-list extension.json entrypoint for ArticlePlaceholder]] (duration: 00m 43s) |
[production] |
| 10:33 |
<moritzm> |
rebooting tegmen for update to 4.9,51 |
[production] |
| 10:07 |
<jynus> |
stopping dbstore1002 (s5) and dbstore2001 (s5) for maintenance |
[production] |
| 10:05 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s) |
[production] |
| 10:00 |
<ema> |
cp4021: restart varnish-be due to mbox lag/fetch failures |
[production] |
| 09:49 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359! (duration: 00m 43s) |
[production] |
| 09:42 |
<elukey> |
reboot analytics100[12] for kernel+jvm updates (Hadoop Master nodes) - T179943 |
[production] |
| 09:39 |
<godog> |
bootstrap restbase1014-b - T179422 |
[production] |
| 09:20 |
<marostegui> |
Optimize s7 on db1098 - T178359 |
[production] |
| 09:03 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174569 (duration: 00m 43s) |
[production] |
| 09:01 |
<marostegui> |
Deploy schema change on db1076 (s2) - T174569 |
[production] |
| 08:57 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174569 (duration: 00m 44s) |
[production] |
| 08:54 |
<moritzm> |
enabling test production traffic for mw1259 (stretch-based video scaler) |
[production] |
| 08:24 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet |
[production] |
| 08:08 |
<moritzm> |
installing python updates on trusty |
[production] |