|
2019-09-17
§
|
| 05:23 |
<effie> |
disable puppet on mw* servers for 536979 |
[production] |
| 05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1122 to s2 master and remove read-only from s2 T230785', diff saved to https://phabricator.wikimedia.org/P9113 and previous config saved to /var/cache/conftool/dbconfig/20190917-050133-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s2 as read-only for maintenance T230785', diff saved to https://phabricator.wikimedia.org/P9112 and previous config saved to /var/cache/conftool/dbconfig/20190917-050043-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s2 failover from db1066 to db1122 - T230785 |
[production] |
| 04:56 |
<effie> |
Downtiming HTTPS-blog on icing - T232412 |
[production] |
| 04:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1122 with weight 0 and depool it from API T230785', diff saved to https://phabricator.wikimedia.org/P9111 and previous config saved to /var/cache/conftool/dbconfig/20190917-041441-marostegui.json |
[production] |
| 04:11 |
<marostegui> |
Start s2 pre-switchover steps T230785 |
[production] |
| 00:34 |
<AndyRussG> |
updated fruec from fb29cb7407 to 97128874bf |
[production] |
|
2019-09-16
§
|
| 23:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wgDebugLogFile in VS (duration: 00m 55s) |
[production] |
| 23:52 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wgDebugLogFile in CS (duration: 00m 55s) |
[production] |
| 23:42 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wgUploadThumbnailRenderHttpCustom* in VS (duration: 00m 54s) |
[production] |
| 23:41 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wgUploadThumbnailRenderHttpCustom* in CS (duration: 00m 55s) |
[production] |
| 23:30 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wmgRC2UDPAddress in VS (duration: 00m 55s) |
[production] |
| 23:29 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wmgRC2UDPAddress in CS (duration: 00m 56s) |
[production] |
| 23:24 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wgCopyUploadProxy in VS (duration: 00m 56s) |
[production] |
| 23:21 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wgCopyUploadProxy in CS (duration: 00m 55s) |
[production] |
| 23:13 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T225261 T194019 Adjust CentralNotice CSP for banner previews for FR-tech (duration: 00m 55s) |
[production] |
| 22:59 |
<chaomodus> |
restarted nagios-nrpe-server on notebook1003 |
[production] |
| 22:46 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use __DIR__ rather than global wmfConfgDir (duration: 00m 55s) |
[production] |
| 21:48 |
<ebernhardson> |
unban elastic1027 from production-search-eqiad |
[production] |
| 20:55 |
<XioNoX> |
remove 2 sessions to AS12871 on cr2-esams - T232617 |
[production] |
| 20:20 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:20 |
<dzahn@cumin1001> |
Updating IPMI password on 1 hosts - dzahn@cumin1001 |
[production] |
| 20:20 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 20:19 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:19 |
<dzahn@cumin1001> |
Updating IPMI password on 1 hosts - dzahn@cumin1001 |
[production] |
| 20:19 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 20:19 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:19 |
<dzahn@cumin1001> |
Updating IPMI password on 1 hosts - dzahn@cumin1001 |
[production] |
| 20:19 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 20:19 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:19 |
<dzahn@cumin1001> |
Updating IPMI password on 1 hosts - dzahn@cumin1001 |
[production] |
| 20:18 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 20:15 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:15 |
<dzahn@cumin1001> |
Updating IPMI password on 8 hosts - dzahn@cumin1001 |
[production] |
| 20:14 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 20:10 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:10 |
<dzahn@cumin1001> |
Updating IPMI password on 2 hosts - dzahn@cumin1001 |
[production] |
| 20:09 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 20:08 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:08 |
<dzahn@cumin1001> |
Updating IPMI password on 2 hosts - dzahn@cumin1001 |
[production] |
| 20:08 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 19:55 |
<XioNoX> |
reboot scs-a8-eqiad (at 100% CPU) |
[production] |
| 19:55 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 19:55 |
<dzahn@cumin1001> |
Updating IPMI password on 2 hosts - dzahn@cumin1001 |
[production] |
| 19:54 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 19:53 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 19:53 |
<dzahn@cumin1001> |
Updating IPMI password on 2 hosts - dzahn@cumin1001 |
[production] |
| 19:52 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 19:51 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |