51-100 of 10000 results (54ms)
2019-04-23 §
12:15 <godog> swift eqiad-prod: fully decom ms-be1013 - T220590 [production]
11:59 <moritzm> installing clamav security updates on fermium [production]
11:56 <kart_> EU-Midday SWAT is done. [production]
11:54 <kart_> 'SWAT: [[gerrit:505059]] deployment-prep: Use new poolcounter instance, [[gerrit:505060]] deployment-prep: Use new ms-fe host.' [production]
11:53 <kartik@deploy1001> Synchronized wmf-config/LabsServices.php: SWAT: [[gerrit:505643]] (duration: 00m 53s) [production]
11:45 <jijiki> Stop xenon-log, excimer-log and apache on mwlog* [production]
11:43 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:505643]] Turn off logging for CitationUsage and CitationUsagePageLoad (T213969) (duration: 00m 53s) [production]
11:29 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix undefined variable from last SWAT (duration: 00m 54s) [production]
11:27 <moritzm> installing clamav security updates on mendelevium (OTRS host) [production]
11:18 <kartik@deploy1001> Synchronized wmf-config: SWAT: [[gerrit:505220]] Use higher unmodified MT threshold for Indonesian Wikipedia (T221353) (duration: 00m 57s) [production]
10:44 <moritzm> uploaded ferm 2.4-1+wmf2+deb10u1 to buster-wikimedia (T153468) [production]
09:23 <godog> upgrade prometheus to v2 on bast5001, previous metrics will not be available until migration and backfill are complete - T187987 [production]
09:19 <elukey> dumping Kafka consumer offsets' history on logstash1012 for T221202 [production]
09:00 <fdans@deploy1001> Finished deploy [analytics/refinery@0d63671]: deploying changes to pageview definition brought in refinery source 0.0.87 (duration: 14m 09s) [production]
08:54 <fsero> synchronizing old docker_registry content into new one - T221101 [production]
08:46 <fdans@deploy1001> Started deploy [analytics/refinery@0d63671]: deploying changes to pageview definition brought in refinery source 0.0.87 [production]
08:14 <moritzm> removing debmonitor entries for labvirt* hosts [production]
08:06 <moritzm> installing wget security updates on jessie [production]
07:27 <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 Set wgPriorityHintsRatio (duration: 00m 52s) [production]
06:20 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool all slaves in x1 T136427 (duration: 00m 57s) [production]
05:52 <elukey> powercycle wtp2019 - no ssh, mgmt console stuck [production]
05:16 <marostegui> Deploy schema change on x1 master - lag will appear on x1 slaves - T136427 [production]
05:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool all slaves in x1 T136427 (duration: 00m 54s) [production]
2019-04-22 §
18:46 <gilles@deploy1001> Synchronized php-1.34.0-wmf.1/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 53s) [production]
18:22 <XioNoX> Add k8s BGP neighbors on cr1/2-eqiad - T220822 [production]
18:15 <XioNoX> Add k8s BGP neighbors on cr1/2-codfw - T220822 [production]
08:47 <marostegui> finished maintenance window on dbstore1003 and dbstore1005 [production]
08:37 <marostegui> Upgrade dbstore1005 [production]
07:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1099 (duration: 00m 54s) [production]
07:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
07:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
06:40 <marostegui> Upgrade dbstore1003 [production]
06:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
05:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
05:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1099 (duration: 00m 54s) [production]
05:25 <marostegui> Stop MySQL and reboot db1099 to see if memory errors clear up T221502 [production]
05:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099 T221502 (duration: 01m 15s) [production]
2019-04-21 §
05:19 <marostegui> Clean up some space on webperf2001 - T221508 [production]
2019-04-20 §
08:12 <_joe_> depooling mw1261,mw1312 wikidata (at least) not working [production]
07:58 <jijiki> Pool thumbor1001 [production]
07:52 <jijiki> depool thumbor1001, switch back to nginx - T187765 [production]
07:50 <_joe_> restarting php-fpm on mw1312, mw1261 to test the new settings over the weekend [production]
2019-04-19 §
23:17 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2245.codfw.wmnet,cluster=api_appserver [production]
23:16 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2244.codfw.wmnet,cluster=api_appserver [production]
23:10 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2150.codfw.wmnet,service=nginx,cluster=jobrunner [production]
22:55 <mutante> mw2244,mw2245,mw2150 - scap pull [production]
22:53 <mutante> mw2244,mw2245,mw2150 - rebooting for known nutcracker issue after first install [production]
22:47 <mutante> furud - remounted /mnt/hdfs for T221483 [production]
21:42 <mutante> mw2150,mw2244,mw2245: initial puppet run, added to mw roles [production]
19:38 <otto@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: No-op - enabling cirrussearch-request logging in beta (duration: 00m 52s) [production]