6801-6850 of 10000 results (86ms)
2019-04-23 §
05:16 <marostegui> Deploy schema change on x1 master - lag will appear on x1 slaves - T136427 [production]
05:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool all slaves in x1 T136427 (duration: 00m 54s) [production]
2019-04-22 §
18:46 <gilles@deploy1001> Synchronized php-1.34.0-wmf.1/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 53s) [production]
18:22 <XioNoX> Add k8s BGP neighbors on cr1/2-eqiad - T220822 [production]
18:15 <XioNoX> Add k8s BGP neighbors on cr1/2-codfw - T220822 [production]
08:47 <marostegui> finished maintenance window on dbstore1003 and dbstore1005 [production]
08:37 <marostegui> Upgrade dbstore1005 [production]
07:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1099 (duration: 00m 54s) [production]
07:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
07:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
06:40 <marostegui> Upgrade dbstore1003 [production]
06:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
05:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1099 (duration: 00m 53s) [production]
05:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1099 (duration: 00m 54s) [production]
05:25 <marostegui> Stop MySQL and reboot db1099 to see if memory errors clear up T221502 [production]
05:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099 T221502 (duration: 01m 15s) [production]
2019-04-21 §
05:19 <marostegui> Clean up some space on webperf2001 - T221508 [production]
2019-04-20 §
08:12 <_joe_> depooling mw1261,mw1312 wikidata (at least) not working [production]
07:58 <jijiki> Pool thumbor1001 [production]
07:52 <jijiki> depool thumbor1001, switch back to nginx - T187765 [production]
07:50 <_joe_> restarting php-fpm on mw1312, mw1261 to test the new settings over the weekend [production]
2019-04-19 §
23:17 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2245.codfw.wmnet,cluster=api_appserver [production]
23:16 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2244.codfw.wmnet,cluster=api_appserver [production]
23:10 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2150.codfw.wmnet,service=nginx,cluster=jobrunner [production]
22:55 <mutante> mw2244,mw2245,mw2150 - scap pull [production]
22:53 <mutante> mw2244,mw2245,mw2150 - rebooting for known nutcracker issue after first install [production]
22:47 <mutante> furud - remounted /mnt/hdfs for T221483 [production]
21:42 <mutante> mw2150,mw2244,mw2245: initial puppet run, added to mw roles [production]
19:38 <otto@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: No-op - enabling cirrussearch-request logging in beta (duration: 00m 52s) [production]
19:37 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: No-op - enabling cirrussearch-request logging in beta (duration: 00m 53s) [production]
19:36 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: No-op - prep for enabling cirrussearch-request logging in beta (duration: 00m 53s) [production]
16:20 <bblack> wikipedia.org CNAME TTLs increase to 4H - https://gerrit.wikimedia.org/r/c/operations/dns/+/505249 - T208263 [production]
16:18 <ejegg> rolled back payments-wiki from eb3d0f35de to aa8dad50e7 [production]
15:55 <reedy@deploy1001> Synchronized php-1.34.0-wmf.1/includes/logging/LogFormatter.php: T220767 (duration: 00m 53s) [production]
15:54 <bblack> restart pybal on lvs1016 (eqiad primary) for eventscehmas service add [production]
15:54 <reedy@deploy1001> Synchronized php-1.34.0-wmf.1/includes/Linker.php: T220767 (duration: 00m 55s) [production]
15:50 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=schema.* [production]
15:42 <bblack> restart pybal on lvs2003 (codfw primary) for eventscehmas service add [production]
15:39 <bblack> restart pybal on lvs2006 (codfw backup) for eventscehmas service add [production]
15:32 <bblack> restarting pybal on lvs1006 (eqiad backup) for eventschema service add [production]
14:59 <volans> uploaded spicerack_0.0.23-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [production]
12:59 <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 T216598 Enable Priority Hints and Element Timing on eswiki (duration: 00m 56s) [production]
08:45 <akosiaris> restart gerrit to pick up https://gerrit.wikimedia.org/r/504981 [production]
06:39 <elukey> roll restart of druid daemons on druid100[1-3] to pick up new jvm settings [production]
2019-04-18 §
23:16 <mobrovac> evening SWAT completed [production]
23:10 <mobrovac@deploy1001> Synchronized php-1.34.0-wmf.1/extensions/EventBus/includes: (no justification provided) (duration: 00m 54s) [production]
23:10 <ejegg> updated payments-wiki from aa8dad50e7 to eb3d0f35de [production]
23:06 <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add wikimania years namespaces to wgNamespacesWithSubpages - T220950 (duration: 00m 53s) [production]
23:00 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
23:00 <robh@cumin1001> START - Cookbook sre.hosts.decommission [production]