4551-4600 of 10000 results (72ms)
2019-05-23 ยง
13:25 <hashar@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.6 [production]
13:00 <godog> swift eqiad-prod: ms-be1033 weight to 1500 - T223518 [production]
12:04 <moritzm> powercycling mw2268 (stuck after reboot) [production]
11:50 <jbond42> will shortly start rolling reboots of thumbor servers [production]
11:37 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:37 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
11:34 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:34 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:23 <moritzm> rebooting auth1002 for kernel update [production]
11:21 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:21 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:51 <Amir1> Deploying EntitySchema to testwikidatawiki is done [production]
10:50 <Amir1> ladsgroup@mwmaint1002:/srv/mediawiki/php-1.34.0-wmf.5$ mwscript sql.php --wiki=wikidatawiki extensions/EntitySchema/sql/EntitySchema.sql (T216955) [production]
10:50 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:511844|deploy WikibaseSchema to test (T216956)]] (duration: 00m 56s) [production]
10:44 <Amir1> ladsgroup@mwmaint1002:/srv/mediawiki/php-1.34.0-wmf.5$ mwscript sql.php --wiki=testwikidatawiki extensions/EntitySchema/sql/EntitySchema.sql (T216956) [production]
10:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:28 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 57s) [production]
10:15 <_joe_> restarted php7.2-fpm on mw1261 to assess the effect of a larger APCu shm size T223180 [production]
10:00 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:00 <moritzm> rebooting remaining mw servers in codfw (sans mcrouter proxies for now) [production]
10:00 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:51 <hashar@deploy1001> Synchronized php-1.34.0-wmf.6/extensions/Collection: Rename wfAjaxCollectionGetItemList() T224093 (duration: 00m 57s) [production]
09:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 into API (duration: 00m 55s) [production]
09:22 <godog> bounce rsyslog on lithium - listener stuck /T199406 [production]
09:10 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:10 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:10 <moritzm> rebooting scb servers in eqiad [production]
09:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1080 (duration: 00m 55s) [production]
08:49 <hashar@deploy1001> rebuilt and synchronized wikiversions files: Re apply group1 wikis to 1.34.0-wmf.6 T220731 [production]
08:29 <marostegui> Upgrade MySQL and kernel on db1080 [production]
08:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 55s) [production]
08:26 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:26 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:26 <moritzm> rebooting scb servers in codfw [production]
07:58 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1080 (duration: 00m 56s) [production]
07:34 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:34 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:33 <moritzm> rebooting swift frontends in eqiad [production]
07:26 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1136 (duration: 00m 53s) [production]
07:11 <marostegui> Stop MySQL on db1117:3323 to clone db1128 T222682 [production]
06:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1136 (duration: 00m 55s) [production]
06:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2065 from config as it will be moved to m3 to replace db2042 (duration: 00m 55s) [production]
06:28 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2065 from config as it will be moved to m3 to replace db2042 (duration: 00m 56s) [production]
06:14 <mobrovac> start ruwiki dumps to fill the new parsoid tables - T215956 [production]
05:33 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2070 as m5 codfw master - T221533 (duration: 00m 54s) [production]
05:29 <marostegui> Promote db2070 to m5 codfw master instead of db2037 - T221533 [production]
05:20 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Clarify db2107 status - will be the new master (duration: 00m 54s) [production]
05:03 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1136 into s7 T222682 (duration: 00m 55s) [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1136 into s7 T222682 (duration: 00m 55s) [production]