|
2021-02-09
ยง
|
| 10:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1157 for the first time in s3 T258361', diff saved to https://phabricator.wikimedia.org/P14251 and previous config saved to /var/cache/conftool/dbconfig/20210209-102109-marostegui.json |
[production] |
| 10:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3317 (re)pooling @ 75%: Slowly repooling db1090:3317 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14250 and previous config saved to /var/cache/conftool/dbconfig/20210209-101939-root.json |
[production] |
| 10:19 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1019.eqiad.wmnet |
[production] |
| 10:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3312 (re)pooling @ 75%: Slowly repooling db1090:3312 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14249 and previous config saved to /var/cache/conftool/dbconfig/20210209-101911-root.json |
[production] |
| 10:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1157 to dbctl, depooled T258361', diff saved to https://phabricator.wikimedia.org/P14248 and previous config saved to /var/cache/conftool/dbconfig/20210209-101556-marostegui.json |
[production] |
| 10:13 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc1019.eqiad.wmnet |
[production] |
| 10:12 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 10:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3317 (re)pooling @ 50%: Slowly repooling db1090:3317 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14247 and previous config saved to /var/cache/conftool/dbconfig/20210209-100436-root.json |
[production] |
| 10:04 |
<elukey> |
stop mysql replication an-coord1001 -> an-coord1002, an-coord1001 -> db1108 |
[analytics] |
| 10:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3312 (re)pooling @ 50%: Slowly repooling db1090:3312 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14246 and previous config saved to /var/cache/conftool/dbconfig/20210209-100407-root.json |
[production] |
| 09:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3317 (re)pooling @ 25%: Slowly repooling db1090:3317 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14245 and previous config saved to /var/cache/conftool/dbconfig/20210209-094932-root.json |
[production] |
| 09:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3312 (re)pooling @ 25%: Slowly repooling db1090:3312 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14244 and previous config saved to /var/cache/conftool/dbconfig/20210209-094904-root.json |
[production] |
| 09:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3317 (re)pooling @ 10%: Slowly repooling db1090:3317 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14243 and previous config saved to /var/cache/conftool/dbconfig/20210209-093429-root.json |
[production] |
| 09:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3312 (re)pooling @ 10%: Slowly repooling db1090:3312 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14242 and previous config saved to /var/cache/conftool/dbconfig/20210209-093400-root.json |
[production] |
| 09:22 |
<godog> |
swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836 |
[production] |
| 08:44 |
<XioNoX> |
repool esams - T272342 |
[production] |
| 08:30 |
<XioNoX> |
rollback redirect ns2 to authdns1001 - T252631 |
[production] |
| 08:29 |
<elukey> |
leave hdfs safemode to let distcp do its job |
[analytics] |
| 08:25 |
<elukey> |
set hdfs safemode on for the Analytics cluster |
[analytics] |
| 08:19 |
<elukey> |
umount /mnt/hdfs from all nodes using it |
[analytics] |
| 08:16 |
<joal> |
Kill flink yarn app |
[analytics] |
| 08:09 |
<XioNoX> |
alright, brace yourself, esams switch stack is going to go down |
[production] |
| 08:08 |
<elukey> |
stop jupyterhub on stat100x |
[analytics] |
| 08:07 |
<elukey> |
stop hive on an-coord100[1,2] - prep step for bigtop upgrade |
[analytics] |
| 08:05 |
<elukey> |
stop oozie an-coord1001 - prep step for bigtop upgrade |
[analytics] |
| 08:03 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on 32 hosts with reason: switch upgrade |
[production] |
| 08:03 |
<elukey> |
stop presto-server on an-presto100x and an-coord1001 - prep step for bigtop upgrade |
[analytics] |
| 08:02 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:30:00 on 32 hosts with reason: switch upgrade |
[production] |
| 07:54 |
<XioNoX> |
redirect ns2 to authdns1001 - T252631 |
[production] |
| 07:47 |
<hashar@deploy1001> |
Finished deploy [integration/docroot@672e79f]: build: Add /scap/log to gitignore (duration: 00m 06s) |
[production] |
| 07:47 |
<hashar@deploy1001> |
Started deploy [integration/docroot@672e79f]: build: Add /scap/log to gitignore |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1081 from dbctl T273040', diff saved to https://phabricator.wikimedia.org/P14241 and previous config saved to /var/cache/conftool/dbconfig/20210209-073455-marostegui.json |
[production] |
| 07:28 |
<elukey> |
roll out new apt bigtop changes across all hadoop-related nodes |
[analytics] |
| 07:20 |
<ryankemper> |
[WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good |
[production] |
| 07:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 100%: Slowly repooling db1111 after onsite maintenance', diff saved to https://phabricator.wikimedia.org/P14240 and previous config saved to /var/cache/conftool/dbconfig/20210209-072038-root.json |
[production] |
| 07:19 |
<joal> |
Killing yarn users applications |
[analytics] |
| 07:12 |
<elukey> |
stop airflow on an-airflow1001 (prep step for bigtop) |
[analytics] |
| 07:09 |
<elukey> |
stop namenode on an-worker1124 (backup cluster), create two new partitions for backup and namenode, restart namenode |
[analytics] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 75%: Slowly repooling db1111 after onsite maintenance', diff saved to https://phabricator.wikimedia.org/P14239 and previous config saved to /var/cache/conftool/dbconfig/20210209-070534-root.json |
[production] |
| 07:04 |
<XioNoX> |
depool disable 2 uplinks on asw2-esams - T272342 |
[production] |
| 06:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 50%: Slowly repooling db1111 after onsite maintenance', diff saved to https://phabricator.wikimedia.org/P14238 and previous config saved to /var/cache/conftool/dbconfig/20210209-065031-root.json |
[production] |
| 06:48 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 06:48 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 06:48 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 06:47 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@582b070]: 0.3.63 (duration: 06m 46s) |
[production] |
| 06:44 |
<XioNoX> |
depool esams for network maintenance - T272342 |
[production] |
| 06:41 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.63` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 06:40 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@582b070]: 0.3.63 |
[production] |
| 06:40 |
<ryankemper> |
Pooled `wdqs1007` and depooled `wdqs1005` (`1005` is ~12 hours behind) |
[production] |
| 06:38 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.63`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |