|
2019-09-17
§
|
| 09:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool and stop replication on db1130 db1104 db1085 db1086 (lag will appear on s6 on labsdb) for PDU maintenance - T227539', diff saved to https://phabricator.wikimedia.org/P9116 and previous config saved to /var/cache/conftool/dbconfig/20190917-094827-marostegui.json |
[production] |
| 09:46 |
<marostegui> |
Depool and stop replication on db1130 db1104 db1085 db1086 (lag will appear on s6 on labsdb) for PDU maintenance - T227539 |
[production] |
| 09:30 |
<hashar> |
Restarting CI jenkins |
[production] |
| 09:29 |
<marostegui> |
Downtime db1073 db1130 db1104 db1085 db1086 for the PDU maintenance T227539 |
[production] |
| 09:18 |
<jynus@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:16 |
<mobrovac> |
bootstrap restbase2010-a - T224553 |
[production] |
| 09:15 |
<jynus@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:05 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 100% of users who accept cookies - T219150 (duration: 00m 57s) |
[production] |
| 08:37 |
<vgutierrez> |
upgrading ATS to 8.0.5-1wm8 on cp3034 - T231849 T232724 |
[production] |
| 07:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1074 with just 50 to keep its warmness level just in case T231638', diff saved to https://phabricator.wikimedia.org/P9115 and previous config saved to /var/cache/conftool/dbconfig/20190917-075807-marostegui.json |
[production] |
| 07:48 |
<effie> |
Enable puppet on mw* |
[production] |
| 07:42 |
<elukey> |
reboot analytics-tool1004 (host running superset) for kernel updates |
[production] |
| 07:41 |
<marostegui> |
Stop mysql on db1063 for decommissioning T232564 |
[production] |
| 07:40 |
<marostegui> |
Remove db1063 from puppet and zarcillo T232564 |
[production] |
| 07:29 |
<vgutierrez> |
repooling cp5007 without wikibase configuration - T99531 |
[production] |
| 07:23 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:21 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:19 |
<vgutierrez> |
depooling cp5007 to ensure that wikibase removal goes as expected - T99531 |
[production] |
| 07:10 |
<vgutierrez> |
getting rid of wikibase TLS certificate & nginx configuration on the text cache cluster - T99531 |
[production] |
| 06:56 |
<vgutierrez> |
upgrading ATS to 8.0.5-1wm8 on cp2002, cp4021 and cp5001 - T231849 |
[production] |
| 06:55 |
<vgutierrez> |
uploaded trafficserver 8.0.5-1wm8 to apt.wikimedia.org (stretch) - T231849 |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1066 T233071', diff saved to https://phabricator.wikimedia.org/P9114 and previous config saved to /var/cache/conftool/dbconfig/20190917-065342-marostegui.json |
[production] |
| 06:49 |
<moritzm> |
reimage restbase2010 to Stretch T224553 |
[production] |
| 05:57 |
<vgutierrez> |
upgrading ATS to 8.0.5-1wm7 on cp2002 and cp4021 - T232724 |
[production] |
| 05:56 |
<vgutierrez> |
uploaded trafficserver 8.0.5-1wm7 to apt.wikimedia.org (stretch) - T232298 T232724 |
[production] |
| 05:23 |
<effie> |
disable puppet on mw* servers for 536979 |
[production] |
| 05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1122 to s2 master and remove read-only from s2 T230785', diff saved to https://phabricator.wikimedia.org/P9113 and previous config saved to /var/cache/conftool/dbconfig/20190917-050133-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s2 as read-only for maintenance T230785', diff saved to https://phabricator.wikimedia.org/P9112 and previous config saved to /var/cache/conftool/dbconfig/20190917-050043-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s2 failover from db1066 to db1122 - T230785 |
[production] |
| 04:56 |
<effie> |
Downtiming HTTPS-blog on icing - T232412 |
[production] |
| 04:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1122 with weight 0 and depool it from API T230785', diff saved to https://phabricator.wikimedia.org/P9111 and previous config saved to /var/cache/conftool/dbconfig/20190917-041441-marostegui.json |
[production] |
| 04:11 |
<marostegui> |
Start s2 pre-switchover steps T230785 |
[production] |
| 00:34 |
<AndyRussG> |
updated fruec from fb29cb7407 to 97128874bf |
[production] |
|
2019-09-16
§
|
| 23:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wgDebugLogFile in VS (duration: 00m 55s) |
[production] |
| 23:52 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wgDebugLogFile in CS (duration: 00m 55s) |
[production] |
| 23:42 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wgUploadThumbnailRenderHttpCustom* in VS (duration: 00m 54s) |
[production] |
| 23:41 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wgUploadThumbnailRenderHttpCustom* in CS (duration: 00m 55s) |
[production] |
| 23:30 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wmgRC2UDPAddress in VS (duration: 00m 55s) |
[production] |
| 23:29 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wmgRC2UDPAddress in CS (duration: 00m 56s) |
[production] |
| 23:24 |
<jforrester@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Stop setting wgCopyUploadProxy in VS (duration: 00m 56s) |
[production] |
| 23:21 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set wgCopyUploadProxy in CS (duration: 00m 55s) |
[production] |
| 23:13 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T225261 T194019 Adjust CentralNotice CSP for banner previews for FR-tech (duration: 00m 55s) |
[production] |
| 22:59 |
<chaomodus> |
restarted nagios-nrpe-server on notebook1003 |
[production] |
| 22:46 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use __DIR__ rather than global wmfConfgDir (duration: 00m 55s) |
[production] |
| 21:48 |
<ebernhardson> |
unban elastic1027 from production-search-eqiad |
[production] |
| 20:55 |
<XioNoX> |
remove 2 sessions to AS12871 on cr2-esams - T232617 |
[production] |
| 20:20 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 20:20 |
<dzahn@cumin1001> |
Updating IPMI password on 1 hosts - dzahn@cumin1001 |
[production] |
| 20:20 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 20:19 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |