|
2021-02-02
§
|
| 07:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14117 and previous config saved to /var/cache/conftool/dbconfig/20210202-071602-root.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14116 and previous config saved to /var/cache/conftool/dbconfig/20210202-070057-root.json |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14115 and previous config saved to /var/cache/conftool/dbconfig/20210202-064553-root.json |
[production] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14114 and previous config saved to /var/cache/conftool/dbconfig/20210202-063050-root.json |
[production] |
| 06:24 |
<marostegui> |
Restart mysql on es1022 |
[production] |
| 06:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1022 T266483', diff saved to https://phabricator.wikimedia.org/P14113 and previous config saved to /var/cache/conftool/dbconfig/20210202-062303-marostegui.json |
[production] |
| 04:12 |
<ryankemper> |
[WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 03:36 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@ad9db35]: 0.3.62 (duration: 06m 59s) |
[production] |
| 03:29 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.62` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 03:29 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@ad9db35]: 0.3.62 |
[production] |
| 03:26 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.62`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 03:21 |
<ryankemper> |
`sudo systemctl restart wdqs-blazegraph` on `wdqs1006` |
[production] |
|
2021-02-01
§
|
| 23:54 |
<legoktm@deploy1001> |
Synchronized wmf-config/profiler.php: profiler: Send data to excimer-buster pipeline (T273312) (duration: 00m 57s) |
[production] |
| 23:15 |
<legoktm> |
depooling mw1403 and mw1405 for perf testing |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1405.eqiad.wmnet |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1403.eqiad.wmnet |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1278.eqiad.wmnet |
[production] |
| 23:05 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.28/extensions/Collection/includes/Specials/SpecialCollection.php: 3c7864ca1d5aadc9cd251939c0e23f661faef5e9: Remove unnecessary calls to WikiPage (T273101) (duration: 00m 58s) |
[production] |
| 22:09 |
<sbassett> |
Deployed security patch for T272386 |
[production] |
| 22:05 |
<sbassett> |
Deployed security patch for T270713 |
[production] |
| 22:04 |
<legoktm> |
depooling mw1278.eqiad.wmnet for perf testing |
[production] |
| 22:03 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1278.eqiad.wmnet |
[production] |
| 22:03 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1277.eqiad.wmnet |
[production] |
| 21:29 |
<dancy@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.27 |
[production] |
| 20:53 |
<andrew@deploy1001> |
Finished deploy [striker/deploy@b6441b8]: Striker hacked fix for T272410 (duration: 00m 57s) |
[production] |
| 20:52 |
<andrew@deploy1001> |
Started deploy [striker/deploy@b6441b8]: Striker hacked fix for T272410 |
[production] |
| 20:27 |
<legoktm> |
depooling mw1277.eqiad.wmnet for perf testing |
[production] |
| 19:42 |
<Urbanecm> |
Morning B&C done |
[production] |
| 19:41 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.28/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php: 1acaba4b3650dfb757d29af5395cc7660c839756: SpecialHomepage: Do not load start-startediting if SE arent enabled (T273243) (duration: 01m 05s) |
[production] |
| 19:38 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.28/extensions/GrowthExperiments/includes/HomepageModules/Banner.php: d39746aa3ed07dfa9173a98d253c61771d5592a1: Banner module: Switch to using activated/unactivated for state (T273084) (duration: 01m 05s) |
[production] |
| 19:23 |
<mutante> |
gerrit2001 - restarting gerrit (replica) |
[production] |
| 19:14 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 6360e7899b7dedc29941783b1cdf76df8db073d7: Enable DiscussionTools as a beta feature on 3 wikis per request (T258554; T265829; T273192) (duration: 01m 04s) |
[production] |
| 19:11 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: a98f08f9582215e8f12f9e9c43f79a1f2fc21a2f: Enable DiscussionTools as a beta feature on wikis with language variants (T272639) (duration: 01m 07s) |
[production] |
| 18:57 |
<mutante> |
restarting gerrit for change 660030 (no ticket) |
[production] |
| 18:44 |
<mutante> |
new Wikimedia project language "mni" added - Meitei is a Sino-Tibetan language and the predominant language and lingua franca of the state of Manipur in northeastern India. |
[production] |
| 18:03 |
<mutante> |
ping3001 - apt-get clean; apt-get autoremove; let it finish kernel upgrade; was out of disk |
[production] |
| 17:59 |
<mutante> |
ping 2001 - apt-get clean; apt autoremove - was out of disk as well |
[production] |
| 17:52 |
<mutante> |
ping1001 - apt-get clean gets back 447M - it was out of disk completely, now 84% usage |
[production] |
| 17:34 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:29 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 17:17 |
<jbond@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deneb.codfw.wmnet |
[production] |
| 17:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5001.wikimedia.org |
[production] |
| 17:15 |
<jbond@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host deneb.codfw.wmnet |
[production] |
| 17:15 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:14 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4001.wikimedia.org |
[production] |