|
2019-05-22
§
|
| 12:39 |
<marostegui> |
Stop replication on db2048 (s1 codfw master) to rebuild revision table - this will generate lag on codfw - T224017 |
[production] |
| 12:35 |
<bblack> |
cp3035: restarting varnish backend |
[production] |
| 12:34 |
<marostegui> |
Stop replication on db1080 to rebuild revision table - T224017 |
[production] |
| 12:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1080 to rebuild revision table T224017 (duration: 00m 55s) |
[production] |
| 11:30 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:30 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:503342|Remove constraint-suggestions beta feature (T220609)]] (duration: 00m 57s) |
[production] |
| 11:19 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:509878|Add configuration for EntitySchema ShExSimpleUrl (T223120)]] (duration: 00m 56s) |
[production] |
| 11:14 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:511674|[SDC] Enable depicts qualifiers on testcommons]] (duration: 00m 57s) |
[production] |
| 10:01 |
<vgutierrez> |
restarting varnish-backend on cp3039 |
[production] |
| 09:52 |
<mobrovac> |
start the en, fr and de wiki dumps again to populate the new parsoid table - T215956 |
[production] |
| 09:43 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@b90fb8b]: Temporarily copy from old tables to new ones if the data is not found, the correct way this time - T215956 (duration: 27m 07s) |
[production] |
| 09:42 |
<marostegui> |
Stop MySQL on db2078:m5 to clone db2070 - T221533 |
[production] |
| 09:16 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@b90fb8b]: Temporarily copy from old tables to new ones if the data is not found, the correct way this time - T215956 |
[production] |
| 08:52 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Move db2070 from s1 to m5 (duration: 00m 55s) |
[production] |
| 08:51 |
<marostegui@deploy1001> |
sync-file aborted: Move db2070 from s1 to m5 (duration: 00m 03s) |
[production] |
| 08:42 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:42 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:32 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1086 (duration: 00m 56s) |
[production] |
| 08:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1086 into API (duration: 00m 56s) |
[production] |
| 08:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1086 (duration: 00m 55s) |
[production] |
| 07:41 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tackle s8 codfw weights T220170 (duration: 00m 55s) |
[production] |
| 07:36 |
<mobrovac> |
decommission restbase1007-c - T223976 |
[production] |
| 07:24 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tackle s4 codfw weights T220170 (duration: 01m 06s) |
[production] |
| 07:23 |
<marostegui> |
Restart MySQL on db2090 to change binlog format T220170 |
[production] |
| 06:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2040 from config T224079 (duration: 00m 55s) |
[production] |
| 06:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2040 from config T224079 (duration: 00m 56s) |
[production] |
| 06:13 |
<marostegui> |
Remove db2040 from zarcillo and tendril - T224079 |
[production] |
| 06:01 |
<marostegui> |
Stop MySQL on db2040 - T224079 |
[production] |
| 05:42 |
<marostegui> |
Stop MySQL on db1086 to clone db1136 |
[production] |
| 05:39 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s) |
[production] |
| 05:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s) |
[production] |
| 05:25 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s) |
[production] |
| 05:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1118 from s1 api and pool db1134 instead T224017 (duration: 00m 57s) |
[production] |
| 04:41 |
<gilles> |
purging ruwiki and eswiki to make them get the new origin trial tokens |
[production] |
| 04:39 |
<gilles@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Renew origin trial tokens (duration: 00m 57s) |
[production] |
| 03:22 |
<legoktm> |
removed 2fa for T224075 |
[production] |
| 01:46 |
<aaron@deploy1001> |
Synchronized php-1.34.0-wmf.5/includes/specials/SpecialWatchlist.php: 68eeaa5b76738a6a07d148391220cdb6c8fd1d23 (duration: 00m 57s) |
[production] |
| 01:22 |
<aaron@deploy1001> |
Synchronized php-1.34.0-wmf.6/includes/specials/SpecialWatchlist.php: 447bf504e498e2c18f29b90f7760514102236e4e (duration: 00m 57s) |
[production] |
|
2019-05-21
§
|
| 23:47 |
<maxsem@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511668/ (duration: 00m 57s) |
[production] |
| 23:34 |
<maxsem@deploy1001> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511667/ (duration: 00m 56s) |
[production] |
| 22:56 |
<mutante> |
ms-be2034 - degraded systemd state was cleared and originally caused by " failed Session 72587 of user debmonitor" |
[production] |
| 22:56 |
<mutante> |
ms-be2034 - sudo systemctl reset-failed |
[production] |
| 22:51 |
<urandom> |
decommissioning restbase1007-b -- T223976 |
[production] |
| 21:35 |
<ejegg> |
updated payments-wiki from d5ef5ad067 to fa005a0640 |
[production] |
| 21:21 |
<mutante> |
re-enabling puppet on mc1* hosts |
[production] |
| 20:43 |
<mutante> |
re-enabling puppet on all hosts using memcached class - except mc1* |
[production] |
| 20:31 |
<mutante> |
mc2019 - stopping memcached and letting puppet restart it to confirm no issues after switching to systemd::service |
[production] |
| 20:20 |
<mutante> |
disabling puppet on all servers using class memcached (57) |
[production] |
| 20:06 |
<tzatziki> |
removing (another) two files for legal compliance |
[production] |
| 19:43 |
<tzatziki> |
removing two files for legal compliance |
[production] |