|
2019-05-22
§
|
| 08:42 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:42 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:32 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1086 (duration: 00m 56s) |
[production] |
| 08:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1086 into API (duration: 00m 56s) |
[production] |
| 08:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1086 (duration: 00m 55s) |
[production] |
| 07:41 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tackle s8 codfw weights T220170 (duration: 00m 55s) |
[production] |
| 07:36 |
<mobrovac> |
decommission restbase1007-c - T223976 |
[production] |
| 07:24 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tackle s4 codfw weights T220170 (duration: 01m 06s) |
[production] |
| 07:23 |
<marostegui> |
Restart MySQL on db2090 to change binlog format T220170 |
[production] |
| 06:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2040 from config T224079 (duration: 00m 55s) |
[production] |
| 06:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2040 from config T224079 (duration: 00m 56s) |
[production] |
| 06:13 |
<marostegui> |
Remove db2040 from zarcillo and tendril - T224079 |
[production] |
| 06:01 |
<marostegui> |
Stop MySQL on db2040 - T224079 |
[production] |
| 05:42 |
<marostegui> |
Stop MySQL on db1086 to clone db1136 |
[production] |
| 05:39 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s) |
[production] |
| 05:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s) |
[production] |
| 05:25 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s) |
[production] |
| 05:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1118 from s1 api and pool db1134 instead T224017 (duration: 00m 57s) |
[production] |
| 04:41 |
<gilles> |
purging ruwiki and eswiki to make them get the new origin trial tokens |
[production] |
| 04:39 |
<gilles@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Renew origin trial tokens (duration: 00m 57s) |
[production] |
| 03:22 |
<legoktm> |
removed 2fa for T224075 |
[production] |
| 01:46 |
<aaron@deploy1001> |
Synchronized php-1.34.0-wmf.5/includes/specials/SpecialWatchlist.php: 68eeaa5b76738a6a07d148391220cdb6c8fd1d23 (duration: 00m 57s) |
[production] |
| 01:22 |
<aaron@deploy1001> |
Synchronized php-1.34.0-wmf.6/includes/specials/SpecialWatchlist.php: 447bf504e498e2c18f29b90f7760514102236e4e (duration: 00m 57s) |
[production] |
|
2019-05-21
§
|
| 23:47 |
<maxsem@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511668/ (duration: 00m 57s) |
[production] |
| 23:34 |
<maxsem@deploy1001> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511667/ (duration: 00m 56s) |
[production] |
| 22:56 |
<mutante> |
ms-be2034 - degraded systemd state was cleared and originally caused by " failed Session 72587 of user debmonitor" |
[production] |
| 22:56 |
<mutante> |
ms-be2034 - sudo systemctl reset-failed |
[production] |
| 22:51 |
<urandom> |
decommissioning restbase1007-b -- T223976 |
[production] |
| 21:35 |
<ejegg> |
updated payments-wiki from d5ef5ad067 to fa005a0640 |
[production] |
| 21:21 |
<mutante> |
re-enabling puppet on mc1* hosts |
[production] |
| 20:43 |
<mutante> |
re-enabling puppet on all hosts using memcached class - except mc1* |
[production] |
| 20:31 |
<mutante> |
mc2019 - stopping memcached and letting puppet restart it to confirm no issues after switching to systemd::service |
[production] |
| 20:20 |
<mutante> |
disabling puppet on all servers using class memcached (57) |
[production] |
| 20:06 |
<tzatziki> |
removing (another) two files for legal compliance |
[production] |
| 19:43 |
<tzatziki> |
removing two files for legal compliance |
[production] |
| 19:12 |
<thcipriani> |
gerrit back on 2.15.13 |
[production] |
| 19:09 |
<thcipriani> |
restart gerrit for 2.15.13 update |
[production] |
| 19:08 |
<thcipriani@deploy1001> |
Finished deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (cobalt, restart incoming) (duration: 00m 20s) |
[production] |
| 19:08 |
<thcipriani@deploy1001> |
Started deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (cobalt, restart incoming) |
[production] |
| 19:06 |
<thcipriani@deploy1001> |
Finished deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (gerrit 2001 only) (duration: 00m 11s) |
[production] |
| 19:06 |
<thcipriani@deploy1001> |
Started deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (gerrit 2001 only) |
[production] |
| 18:50 |
<bblack> |
repooling cp1085 frontends (weren't meant to be depooled) |
[production] |
| 18:38 |
<bblack> |
re-pooling eqiad front edge traffic (onto new LVSes from T184293 ) |
[production] |
| 18:35 |
<XioNoX> |
update lvs static routes on cr1/2-eqiad - T184293 |
[production] |
| 18:06 |
<andrewbogott> |
restarting rabbitmq-server on cloudcontrol1003 (turning on HA queues) |
[production] |
| 17:59 |
<bblack> |
rebooting lvs1016 in attempt to clear interface config issues - T224027 |
[production] |
| 17:51 |
<XioNoX> |
add BGP sessions to AS202053 in esams |
[production] |
| 17:31 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected (again, after merging last-minute fixup https://gerrit.wikimedia.org/r/c/operations/puppet/+/511759 ) |
[production] |
| 17:25 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected |
[production] |
| 17:24 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1006, basically no-op |
[production] |