|
2021-02-02
§
|
| 12:12 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host idp2001.wikimedia.org |
[production] |
| 12:12 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host idp-test2001.wikimedia.org |
[production] |
| 12:11 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host idp-test1001.wikimedia.org |
[production] |
| 11:06 |
<jynus@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1171.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:04 |
<jynus@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1171.eqiad.wmnet with reason: REIMAGE |
[production] |
| 10:30 |
<XioNoX> |
re-enable DE-CIX codfw peering sessions |
[production] |
| 10:17 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1094 to clone db1174 - T258361', diff saved to https://phabricator.wikimedia.org/P14121 and previous config saved to /var/cache/conftool/dbconfig/20210202-100859-marostegui.json |
[production] |
| 10:08 |
<elukey@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:02 |
<hashar> |
Restarted Gerrit primary on gerrit1001 # T273223 |
[production] |
| 10:00 |
<hashar@deploy1001> |
Finished deploy [gerrit/gerrit@c3cd63b]: Gerrit primary on gerrit1001 to v3.2.7 T273223 (duration: 00m 09s) |
[production] |
| 10:00 |
<hashar@deploy1001> |
Started deploy [gerrit/gerrit@c3cd63b]: Gerrit primary on gerrit1001 to v3.2.7 T273223 |
[production] |
| 10:00 |
<hashar> |
Restarted Gerrit replica on gerrit2001 # T273223 |
[production] |
| 09:56 |
<hashar@deploy1001> |
Finished deploy [gerrit/gerrit@c3cd63b]: Gerrit replica on gerrit2001 to v3.2.7 T273223 (duration: 00m 12s) |
[production] |
| 09:56 |
<hashar@deploy1001> |
Started deploy [gerrit/gerrit@c3cd63b]: Gerrit replica on gerrit2001 to v3.2.7 T273223 |
[production] |
| 09:27 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1381.eqiad.wmnet |
[production] |
| 08:56 |
<XioNoX> |
disable DE-CIX codfw peering session |
[production] |
| 08:30 |
<godog> |
swift eqiad-prod: add weight back to sdg on ms-be1054 - T273582 |
[production] |
| 08:02 |
<legoktm> |
depooled mw1381.eqiad.wmnet for perf testing (T273312) |
[production] |
| 07:59 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1381.eqiad.wmnet |
[production] |
| 07:45 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1403.eqiad.wmnet |
[production] |
| 07:45 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1405.eqiad.wmnet |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14118 and previous config saved to /var/cache/conftool/dbconfig/20210202-073105-root.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14117 and previous config saved to /var/cache/conftool/dbconfig/20210202-071602-root.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14116 and previous config saved to /var/cache/conftool/dbconfig/20210202-070057-root.json |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14115 and previous config saved to /var/cache/conftool/dbconfig/20210202-064553-root.json |
[production] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14114 and previous config saved to /var/cache/conftool/dbconfig/20210202-063050-root.json |
[production] |
| 06:24 |
<marostegui> |
Restart mysql on es1022 |
[production] |
| 06:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1022 T266483', diff saved to https://phabricator.wikimedia.org/P14113 and previous config saved to /var/cache/conftool/dbconfig/20210202-062303-marostegui.json |
[production] |
| 04:12 |
<ryankemper> |
[WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 03:36 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@ad9db35]: 0.3.62 (duration: 06m 59s) |
[production] |
| 03:29 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.62` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 03:29 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@ad9db35]: 0.3.62 |
[production] |
| 03:26 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.62`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 03:21 |
<ryankemper> |
`sudo systemctl restart wdqs-blazegraph` on `wdqs1006` |
[production] |
|
2021-02-01
§
|
| 23:54 |
<legoktm@deploy1001> |
Synchronized wmf-config/profiler.php: profiler: Send data to excimer-buster pipeline (T273312) (duration: 00m 57s) |
[production] |
| 23:15 |
<legoktm> |
depooling mw1403 and mw1405 for perf testing |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1405.eqiad.wmnet |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1403.eqiad.wmnet |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1278.eqiad.wmnet |
[production] |
| 23:05 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.28/extensions/Collection/includes/Specials/SpecialCollection.php: 3c7864ca1d5aadc9cd251939c0e23f661faef5e9: Remove unnecessary calls to WikiPage (T273101) (duration: 00m 58s) |
[production] |
| 22:09 |
<sbassett> |
Deployed security patch for T272386 |
[production] |
| 22:05 |
<sbassett> |
Deployed security patch for T270713 |
[production] |
| 22:04 |
<legoktm> |
depooling mw1278.eqiad.wmnet for perf testing |
[production] |
| 22:03 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1278.eqiad.wmnet |
[production] |