2251-2300 of 10000 results (61ms)
2018-02-21 ยง
18:12 <_joe_> stopped testing on mwdebug1001 for SWAT window [production]
17:43 <ema> eqsin LVSs: upgrade pybal to 1.14.4 [production]
17:34 <_joe_> resuming tests on mwdebug1001 [production]
17:17 <ema> eqiad LVSs: bounce pybal for labweb proxfetch config changes [production]
17:12 <ppchelko@tin> Finished deploy [changeprop/deploy@e9a6bb0]: Use post for ORES precache rules T158437 (duration: 01m 23s) [production]
17:11 <ppchelko@tin> Started deploy [changeprop/deploy@e9a6bb0]: Use post for ORES precache rules T158437 [production]
17:07 <_joe_> finished testing on mwdebug1001 for swat [production]
16:56 <oblivian@puppetmaster1001> conftool action : edit; selector: name=ReadOnly,scope=eqiad [production]
16:40 <_joe_> testing various etcd failure scenarios on mwdebug1001, T185078 [production]
16:39 <ppchelko@tin> Finished deploy [changeprop/deploy@1be63aa]: Simplify ORES precaching by using the new endpoint T158437 (duration: 01m 33s) [production]
16:37 <ppchelko@tin> Started deploy [changeprop/deploy@1be63aa]: Simplify ORES precaching by using the new endpoint T158437 [production]
16:27 <ema> lvs1010: restart pybal [production]
16:00 <godog> restart rsyslogd on lithium and wezen - T136312 [production]
15:50 <gilles@tin> Synchronized wmf-config/filebackend.php: Thumbor private wiki support deployment: [[gerrit:413115|Serve private wiki thumbnails with Thumbor (T169144)]] (duration: 01m 12s) [production]
15:44 <no_justification> pruned old 1.29.x and 1.30.x versions that somehow stuck around. Also 1.31.0-wmf.* cache/ directories for unused branches. T157030 [production]
15:37 <gilles@tin> Synchronized wmf-config/filebackend.php: Thumbor private wiki support deployment: [[gerrit:413168|Serve officewiki thumbnails with Thumbor (T169144)]] (duration: 01m 11s) [production]
15:27 <gilles@tin> Synchronized private/PrivateSettings.php.example: Thumbor private wiki support deployment: [[gerrit:413168|Add Thumbor/Mediawiki shared secret (T169144)]] (duration: 01m 11s) [production]
15:24 <chasemp> reboot labtestservices2002 [production]
15:24 <gilles@tin> Synchronized wmf-config/filebackend.php: Thumbor private wiki support deployment: [[gerrit:413168|Add Thumbor/Mediawiki shared secret (T169144)]] (duration: 01m 12s) [production]
15:19 <gilles> Thumbor private wiki support deployment [production]
15:08 <zeljkof> EU SWAT finished [production]
15:08 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:412983|Removing Mobile beta feedback link (T187712)]] (duration: 01m 12s) [production]
15:03 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:412996|Disable Page Previews EventLogging instrumentation (T185973)]] (duration: 01m 13s) [production]
14:52 <_joe_> rolling restart another 4 api appservers [production]
14:49 <oblivian@tin> Synchronized wmf-config: Serve configuration to mwdebug hosts via etcd (duration: 01m 16s) [production]
14:42 <_joe_> restarted hhvm on mwdebug1001 too [production]
14:38 <_joe_> restarting hhvm on mwdebug1002 [production]
14:06 <_joe_> restarting hhvm on misbehaving api appservers [production]
14:02 <zfilipin@tin> Synchronized wmf-config/throttle.php: SWAT: [[gerrit:413147|Add new throttle rule (T187870)]] (duration: 01m 13s) [production]
13:28 <marostegui> Reboot db2092 for a kernel upgrade [production]
13:26 <moritzm> powercycling ganeti1007 [production]
12:43 <_joe_> rolling restart of hhvm on api servers under high load [production]
12:38 <elukey> restart hhvm on mw1234 - high load [production]
12:35 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Clarify that db1067 is now s1 candidate master - T186321 (duration: 01m 13s) [production]
12:26 <elukey> restart hhvm on mw1231 - high load, hhvm-dump-debug in /home/elukey/hhvm.6759.bt [production]
12:21 <elukey> restart hhvm on mw1227 - high load, hhvm-dump-debug in /home/elukey/hhvm.23382.bt [production]
12:10 <moritzm> uploading retpoline-enabled gcc-4.9 to apt.wikimedia.org / jessie-wikimedia to be able to use it on boron for building Linux (trying to adapt our pbuilder setup to also include security.debian.org ran into a few proxy-related problems and this is really a rare corner case anyway) [production]
12:02 <ema> lvs5003: pybal upgraded to 1.14.4 [production]
12:01 <ema> pybal 1.14.4 uploaded to apt.w.o [production]
11:17 <moritzm> installing db5.3 security updates [production]
11:12 <jynus> cloning db2011 to db2044 [production]
10:40 <kart_> Finished running CLL preference migration script dry-run on terbium (T187677) [production]
10:33 <marostegui> Reload haproxy on dbproxy1005 - T187722 [production]
10:26 <marostegui> Remove db2030 from tendril - T187768 [production]
10:09 <moritzm> installing openssh bugfix updates from jessie/stretch point releases [production]
10:01 <kart_> Running CLL preference migration script dry-run on terbium (T187677) [production]
09:46 <moritzm> installing dbus updates from stretch point release [production]
09:23 <moritzm> installing sqlite security updates on stretch [production]
08:35 <godog> roll-restart thumbor in codfw and eqiad to apply https://gerrit.wikimedia.org/r/c/412980 [production]
08:20 <gilles> foreachwikiindblist "% private.dblist" extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --backend=local-multiwrite --private [production]