4701-4750 of 10000 results (69ms)
2019-05-22 §
13:36 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:35 <jbond@cumin1001> conftool action : set/pooled=no; selector: name=dns5001.wikimedia.org [production]
13:27 <reedy@deploy1001> Synchronized php-1.34.0-wmf.6/extensions/Collection/templates/: T224092 (duration: 00m 58s) [production]
13:13 <hashar@deploy1001> Synchronized php: group1 wikis to 1.34.0-wmf.6 (duration: 00m 54s) [production]
13:11 <hashar@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.6 [production]
13:06 <urandom> decommissioning restbase1008-a -- T223976 [production]
12:39 <marostegui> Stop replication on db2048 (s1 codfw master) to rebuild revision table - this will generate lag on codfw - T224017 [production]
12:35 <bblack> cp3035: restarting varnish backend [production]
12:34 <marostegui> Stop replication on db1080 to rebuild revision table - T224017 [production]
12:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 to rebuild revision table T224017 (duration: 00m 55s) [production]
11:30 <Amir1> EU SWAT is done [production]
11:30 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:503342|Remove constraint-suggestions beta feature (T220609)]] (duration: 00m 57s) [production]
11:19 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:509878|Add configuration for EntitySchema ShExSimpleUrl (T223120)]] (duration: 00m 56s) [production]
11:14 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:511674|[SDC] Enable depicts qualifiers on testcommons]] (duration: 00m 57s) [production]
10:01 <vgutierrez> restarting varnish-backend on cp3039 [production]
09:52 <mobrovac> start the en, fr and de wiki dumps again to populate the new parsoid table - T215956 [production]
09:43 <mobrovac@deploy1001> Finished deploy [restbase/deploy@b90fb8b]: Temporarily copy from old tables to new ones if the data is not found, the correct way this time - T215956 (duration: 27m 07s) [production]
09:42 <marostegui> Stop MySQL on db2078:m5 to clone db2070 - T221533 [production]
09:16 <mobrovac@deploy1001> Started deploy [restbase/deploy@b90fb8b]: Temporarily copy from old tables to new ones if the data is not found, the correct way this time - T215956 [production]
08:52 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Move db2070 from s1 to m5 (duration: 00m 55s) [production]
08:51 <marostegui@deploy1001> sync-file aborted: Move db2070 from s1 to m5 (duration: 00m 03s) [production]
08:42 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:42 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1086 (duration: 00m 56s) [production]
08:17 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1086 into API (duration: 00m 56s) [production]
08:03 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1086 (duration: 00m 55s) [production]
07:41 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Tackle s8 codfw weights T220170 (duration: 00m 55s) [production]
07:36 <mobrovac> decommission restbase1007-c - T223976 [production]
07:24 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Tackle s4 codfw weights T220170 (duration: 01m 06s) [production]
07:23 <marostegui> Restart MySQL on db2090 to change binlog format T220170 [production]
06:17 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2040 from config T224079 (duration: 00m 55s) [production]
06:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2040 from config T224079 (duration: 00m 56s) [production]
06:13 <marostegui> Remove db2040 from zarcillo and tendril - T224079 [production]
06:01 <marostegui> Stop MySQL on db2040 - T224079 [production]
05:42 <marostegui> Stop MySQL on db1086 to clone db1136 [production]
05:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s) [production]
05:26 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s) [production]
05:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s) [production]
05:09 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1118 from s1 api and pool db1134 instead T224017 (duration: 00m 57s) [production]
04:41 <gilles> purging ruwiki and eswiki to make them get the new origin trial tokens [production]
04:39 <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Renew origin trial tokens (duration: 00m 57s) [production]
03:22 <legoktm> removed 2fa for T224075 [production]
01:46 <aaron@deploy1001> Synchronized php-1.34.0-wmf.5/includes/specials/SpecialWatchlist.php: 68eeaa5b76738a6a07d148391220cdb6c8fd1d23 (duration: 00m 57s) [production]
01:22 <aaron@deploy1001> Synchronized php-1.34.0-wmf.6/includes/specials/SpecialWatchlist.php: 447bf504e498e2c18f29b90f7760514102236e4e (duration: 00m 57s) [production]
2019-05-21 §
23:47 <maxsem@deploy1001> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511668/ (duration: 00m 57s) [production]
23:34 <maxsem@deploy1001> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511667/ (duration: 00m 56s) [production]
22:56 <mutante> ms-be2034 - degraded systemd state was cleared and originally caused by " failed Session 72587 of user debmonitor" [production]
22:56 <mutante> ms-be2034 - sudo systemctl reset-failed [production]
22:51 <urandom> decommissioning restbase1007-b -- T223976 [production]
21:35 <ejegg> updated payments-wiki from d5ef5ad067 to fa005a0640 [production]