|
2020-09-03
ยง
|
| 14:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1128 to wikitech master T260324', diff saved to https://phabricator.wikimedia.org/P12488 and previous config saved to /var/cache/conftool/dbconfig/20200903-140411-marostegui.json |
[production] |
| 14:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T260324', diff saved to https://phabricator.wikimedia.org/P12487 and previous config saved to /var/cache/conftool/dbconfig/20200903-140135-marostegui.json |
[production] |
| 14:00 |
<marostegui> |
Failover m5 (wikitech) master - T260324 |
[production] |
| 13:53 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:53 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:43 |
<jmm@deploy1001> |
Finished deploy [debmonitor/deploy@fb64c52]: deploy to new buster host (duration: 00m 18s) |
[production] |
| 13:43 |
<jmm@deploy1001> |
Started deploy [debmonitor/deploy@fb64c52]: deploy to new buster host |
[production] |
| 13:40 |
<jmm@deploy1001> |
Finished deploy [debmonitor/deploy@25dbd20]: deploy to new buster host, now the --force is with me (duration: 01m 29s) |
[production] |
| 13:39 |
<jmm@deploy1001> |
Started deploy [debmonitor/deploy@25dbd20]: deploy to new buster host, now the --force is with me |
[production] |
| 13:32 |
<jmm@deploy1001> |
Finished deploy [debmonitor/deploy@25dbd20]: deploy to new buster host (duration: 00m 05s) |
[production] |
| 13:32 |
<jmm@deploy1001> |
Started deploy [debmonitor/deploy@25dbd20]: deploy to new buster host |
[production] |
| 13:08 |
<marostegui> |
Start pre m5 failover steps T260324 |
[production] |
| 12:46 |
<marostegui> |
Deploy MCR schema change on s7 eqiad master (lag might show up) - T238966 |
[production] |
| 12:30 |
<hnowlan> |
enabling puppet on appservers, finished rollout of api.wikimedia.org https://gerrit.wikimedia.org/r/c/operations/puppet/+/623833 |
[production] |
| 12:19 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Shift weights in s2 codfw to account for db2125 being down T260670', diff saved to https://phabricator.wikimedia.org/P12485 and previous config saved to /var/cache/conftool/dbconfig/20200903-121916-kormat.json |
[production] |
| 12:17 |
<moritzm> |
installing openexr security updates for stretch |
[production] |
| 12:03 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Depool db2125 after hw issue', diff saved to https://phabricator.wikimedia.org/P12483 and previous config saved to /var/cache/conftool/dbconfig/20200903-120304-kormat.json |
[production] |
| 11:45 |
<moritzm> |
installing net-snmp security updates on Stretch |
[production] |
| 11:45 |
<moritzm> |
installing net-snmp security updates on Buster |
[production] |
| 11:33 |
<Urbanecm> |
[urbanecm@mwmaint2001 ~]$ mwscript namespaceDupes.php --wiki=jawikivoyage --fix | phaste # T260320 # P12481 |
[production] |
| 11:28 |
<moritzm> |
installing PHP 7.0 security updates |
[production] |
| 11:28 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 04281a0875d34e1161f44697f732d898ab12d4f0: Add extra namespaces for jawikivoyage (T260320) (duration: 01m 01s) |
[production] |
| 11:26 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: 976d7350a7252610e4ba34e9227e205d085a609a: Lift IP cap on 2020-09-08 for Senior Citizen Write Wikipedia course - cs.wikipedia (T261882) (duration: 01m 01s) |
[production] |
| 11:21 |
<gilles@deploy1001> |
Synchronized static/images/project-logos: T252108 Deploying lossily optimised Wikipedia logos (duration: 01m 20s) |
[production] |
| 10:50 |
<hnowlan> |
disabling apache on appservers for rollout of https://gerrit.wikimedia.org/r/c/operations/puppet/+/623833 |
[production] |
| 10:38 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 10:07 |
<XioNoX> |
re-apply vlan 1118 firewall filter and update OSPF/bootp on cr1/2-eqiad - T261866 |
[production] |
| 09:57 |
<XioNoX> |
rectification: move vlan 1118 from ae2.1118 to xe-3/0/4.1118 on cr1-eqiad - T261866 |
[production] |
| 09:56 |
<XioNoX> |
move vlan 1118 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 |
[production] |
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12480 and previous config saved to /var/cache/conftool/dbconfig/20200903-095510-marostegui.json |
[production] |
| 09:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2087:3316', diff saved to https://phabricator.wikimedia.org/P12479 and previous config saved to /var/cache/conftool/dbconfig/20200903-095015-marostegui.json |
[production] |
| 09:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12478 and previous config saved to /var/cache/conftool/dbconfig/20200903-094857-marostegui.json |
[production] |
| 09:48 |
<XioNoX> |
move VRRP master from cr1-eqiad:ae2.1118 to cr2-eqiad:xe-3/0/4.1118 - T261866 |
[production] |
| 09:46 |
<XioNoX> |
move vlan 1118 IPv4 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 |
[production] |
| 09:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12477 and previous config saved to /var/cache/conftool/dbconfig/20200903-094435-marostegui.json |
[production] |
| 09:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2087:3316', diff saved to https://phabricator.wikimedia.org/P12476 and previous config saved to /var/cache/conftool/dbconfig/20200903-094043-marostegui.json |
[production] |
| 09:38 |
<XioNoX> |
move vlan 1118 IPv6 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 |
[production] |
| 09:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12475 and previous config saved to /var/cache/conftool/dbconfig/20200903-093629-marostegui.json |
[production] |
| 09:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2087:3316', diff saved to https://phabricator.wikimedia.org/P12474 and previous config saved to /var/cache/conftool/dbconfig/20200903-093454-marostegui.json |
[production] |
| 09:32 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:31 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:29 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2087:3316', diff saved to https://phabricator.wikimedia.org/P12473 and previous config saved to /var/cache/conftool/dbconfig/20200903-092549-marostegui.json |
[production] |
| 09:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2087:3316 db2087:3317 T261917', diff saved to https://phabricator.wikimedia.org/P12472 and previous config saved to /var/cache/conftool/dbconfig/20200903-092028-marostegui.json |
[production] |
| 09:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P12471 and previous config saved to /var/cache/conftool/dbconfig/20200903-091834-marostegui.json |
[production] |
| 09:13 |
<XioNoX> |
rolled back: move vlan 1118 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 |
[production] |
| 09:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2122', diff saved to https://phabricator.wikimedia.org/P12470 and previous config saved to /var/cache/conftool/dbconfig/20200903-090901-marostegui.json |
[production] |
| 09:06 |
<XioNoX> |
move vlan 1118 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P12469 and previous config saved to /var/cache/conftool/dbconfig/20200903-090419-marostegui.json |
[production] |
| 09:01 |
<XioNoX> |
force ae2.1118 VRRP master on cr1-eqiad - T261866 |
[production] |