3601-3650 of 10000 results (88ms)
2019-06-06 §
09:31 <moritzm> rebooting mwdebug2002 for some tests [production]
09:31 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:30 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:28 <moritzm> updating qemu on ganeti2004 for some tests [production]
09:24 <gehel@cumin2001> START - Cookbook sre.postgresql.postgres-init [production]
08:38 <marostegui> Stop MySQL on db1117:3322 - this will trigger haproxy alerts - T222682 [production]
07:35 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1121 after upgrade T224852 (duration: 00m 53s) [production]
07:20 <marostegui> Stop MySQL on db1121 for upgrade, this will generate lag on labs hosts for s6 - T224852 [production]
07:16 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2046 to s6 master as db2039 will be decommissioned T221533 (duration: 00m 55s) [production]
06:31 <marostegui> Start topology changes on s6 codfw to promote db2046 as master - T221533 [production]
06:23 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1121 for upgrade T224852 (duration: 00m 55s) [production]
06:15 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1091 after getting its BBU replaced (duration: 00m 54s) [production]
06:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 01m 01s) [production]
05:47 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) [production]
05:41 <marostegui> Upgrade MySQL on s6 codfw hosts in preparation for s6 codfw master failover - T221533 [production]
05:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) [production]
05:18 <marostegui> Remove db2042 from tendril and zarcillo T225090 [production]
05:18 <marostegui> Remove db2042 from tendril and zarcillo [production]
05:14 <marostegui> Stop MySQL on db2042 to copy its content to dbprov2001 as a temporary backup - T225090 [production]
05:11 <marostegui> Disable notifications db2042 - T225090 [production]
05:09 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1091 after getting its BBU replaced T225060 (duration: 00m 56s) [production]
2019-06-05 §
22:15 <chaomodus> restarting gerrit on cobalt due to it being down (seems like Java out of heap space) [production]
20:43 <mforns@deploy1001> Finished deploy [analytics/refinery@0660e70]: deploying analytics/refinery up to 0660e70153dec892ae20bee7119a72cc17e8ec87 (duration: 19m 30s) [production]
20:39 <reedy@deploy1001> Synchronized wmf-config/flaggedrevs.php: Turn off some FR config T225138 (duration: 00m 54s) [production]
20:25 <akosiaris@deploy1001> scap-helm blubberoid finished [production]
20:25 <akosiaris@deploy1001> scap-helm blubberoid cluster codfw completed [production]
20:25 <akosiaris@deploy1001> scap-helm blubberoid cluster eqiad completed [production]
20:25 <akosiaris@deploy1001> scap-helm blubberoid upgrade -f blubberoid-values.yaml production stable/blubberoid [namespace: blubberoid, clusters: eqiad,codfw] [production]
20:23 <mforns@deploy1001> Started deploy [analytics/refinery@0660e70]: deploying analytics/refinery up to 0660e70153dec892ae20bee7119a72cc17e8ec87 [production]
19:57 <hashar> contint1001: docker container prune -f && docker image prune -f # reclaimed 166 MB and 3.4 GB [production]
19:48 <marostegui> Check data consistency on db1091 against db1135 - T225060 [production]
19:45 <reedy@deploy1001> Synchronized wmf-config/flaggedrevs.php: T225115 (duration: 00m 54s) [production]
17:36 <marostegui> Start replication db1091 - T225060 [production]
17:32 <marostegui> Start MySQL with replication stopped on db1091 - T225060 [production]
16:29 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert user-blocks-change to use eventbus and old schema - T211248 (duration: 00m 54s) [production]
16:22 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: use eventgate-main for 2 events on all wikis - T211248 (duration: 00m 55s) [production]
16:11 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add wgEventServiceStreamConfig and switch 2 topics in group0 T222822 (duration: 00m 56s) [production]
16:11 <XioNoX> remove BGP to AS38082 on cr4-ulsfo (left the IXP) [production]
15:46 <reedy@deploy1001> Scap failed!: Call to mwscript eval.php returned: None [production]
15:44 <reedy@deploy1001> Finished scap: Rebuild .8 i18n for FlaggedRevs (duration: 41m 14s) [production]
15:36 <moritzm> installing exim4 security updates [production]
15:03 <reedy@deploy1001> Started scap: Rebuild .8 i18n for FlaggedRevs [production]
14:24 <marostegui> Poweroff db1091 for BBU replacement - T225060 [production]
13:57 <elukey> restart mcrouter on MediaWiki app/api canaries to pick up new config change (timeouts before marking a memcached shard as TKO from 3 to 10) - T203786 [production]
13:56 <jijiki> enabling puppet and pooling on mw* canaries [production]
13:17 <jynus> start es2,es3 backup on codfw [production]
13:17 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.8 [production]
13:03 <hashar> restarting Jenkins [production]
12:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1135 (duration: 00m 54s) [production]
12:46 <Lucas_WMDE> EU SWAT finished [production]