|
2020-09-03
§
|
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P12469 and previous config saved to /var/cache/conftool/dbconfig/20200903-090419-marostegui.json |
[production] |
| 09:01 |
<XioNoX> |
force ae2.1118 VRRP master on cr1-eqiad - T261866 |
[production] |
| 09:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3317, db1098:3316', diff saved to https://phabricator.wikimedia.org/P12468 and previous config saved to /var/cache/conftool/dbconfig/20200903-090007-marostegui.json |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1090:3317', diff saved to https://phabricator.wikimedia.org/P12467 and previous config saved to /var/cache/conftool/dbconfig/20200903-085838-marostegui.json |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2122', diff saved to https://phabricator.wikimedia.org/P12466 and previous config saved to /var/cache/conftool/dbconfig/20200903-085708-marostegui.json |
[production] |
| 08:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2122', diff saved to https://phabricator.wikimedia.org/P12465 and previous config saved to /var/cache/conftool/dbconfig/20200903-084910-marostegui.json |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1090:3312', diff saved to https://phabricator.wikimedia.org/P12464 and previous config saved to /var/cache/conftool/dbconfig/20200903-084836-marostegui.json |
[production] |
| 08:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1090:3317, db1090:3312', diff saved to https://phabricator.wikimedia.org/P12463 and previous config saved to /var/cache/conftool/dbconfig/20200903-084358-marostegui.json |
[production] |
| 08:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2122', diff saved to https://phabricator.wikimedia.org/P12462 and previous config saved to /var/cache/conftool/dbconfig/20200903-084147-marostegui.json |
[production] |
| 08:33 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 08:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2122 T261917', diff saved to https://phabricator.wikimedia.org/P12461 and previous config saved to /var/cache/conftool/dbconfig/20200903-082956-marostegui.json |
[production] |
| 08:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 08:28 |
<moritzm> |
rebooting mwmaint1002 for kernel update |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12460 and previous config saved to /var/cache/conftool/dbconfig/20200903-082655-marostegui.json |
[production] |
| 08:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12459 and previous config saved to /var/cache/conftool/dbconfig/20200903-082034-marostegui.json |
[production] |
| 08:16 |
<marostegui> |
Upgrade db1101 (s7 and s8) |
[production] |
| 08:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12458 and previous config saved to /var/cache/conftool/dbconfig/20200903-081543-marostegui.json |
[production] |
| 08:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318, db1101:3317', diff saved to https://phabricator.wikimedia.org/P12457 and previous config saved to /var/cache/conftool/dbconfig/20200903-081503-marostegui.json |
[production] |
| 08:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12456 and previous config saved to /var/cache/conftool/dbconfig/20200903-081337-marostegui.json |
[production] |
| 08:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12455 and previous config saved to /var/cache/conftool/dbconfig/20200903-080714-marostegui.json |
[production] |
| 08:06 |
<marostegui> |
Upgrade and reboot db1127 |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12454 and previous config saved to /var/cache/conftool/dbconfig/20200903-080634-marostegui.json |
[production] |
| 08:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12453 and previous config saved to /var/cache/conftool/dbconfig/20200903-080024-marostegui.json |
[production] |
| 07:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12452 and previous config saved to /var/cache/conftool/dbconfig/20200903-075443-marostegui.json |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12451 and previous config saved to /var/cache/conftool/dbconfig/20200903-074922-marostegui.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2086:3317 T261917', diff saved to https://phabricator.wikimedia.org/P12450 and previous config saved to /var/cache/conftool/dbconfig/20200903-074827-marostegui.json |
[production] |
| 07:45 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 07:45 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 07:45 |
<marostegui> |
Upgrade and reboot db1094 |
[production] |
| 07:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12449 and previous config saved to /var/cache/conftool/dbconfig/20200903-074426-marostegui.json |
[production] |
| 07:38 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 07:38 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 07:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12448 and previous config saved to /var/cache/conftool/dbconfig/20200903-073718-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12447 and previous config saved to /var/cache/conftool/dbconfig/20200903-073116-marostegui.json |
[production] |
| 07:29 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12446 and previous config saved to /var/cache/conftool/dbconfig/20200903-072716-marostegui.json |
[production] |
| 07:24 |
<hashar> |
contint2001: restarting CI Jenkins for plugins upgrade |
[production] |
| 07:19 |
<marostegui> |
Deploy schema change on s8 eqiad master T237120 |
[production] |
| 07:18 |
<marostegui> |
Stop slave on s8 eqiad master (lag will appear on s8 eqiad) - T237120 |
[production] |
| 07:02 |
<marostegui> |
Stop db2100:3317 and db2121 in sync to reload metawiki.content T261869 |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12445 and previous config saved to /var/cache/conftool/dbconfig/20200903-070104-marostegui.json |
[production] |
| 06:56 |
<hashar> |
contint2001: restarting CI Jenkins |
[production] |
| 06:56 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 06:56 |
<_joe_> |
deployment of mobileapps to pick up changes to envoy config, new helmfile layout |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12444 and previous config saved to /var/cache/conftool/dbconfig/20200903-065105-marostegui.json |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12443 and previous config saved to /var/cache/conftool/dbconfig/20200903-064804-marostegui.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12442 and previous config saved to /var/cache/conftool/dbconfig/20200903-064623-marostegui.json |
[production] |
| 06:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12441 and previous config saved to /var/cache/conftool/dbconfig/20200903-064334-marostegui.json |
[production] |
| 06:24 |
<marostegui> |
Disconnect eqiad -> codfw replication |
[production] |