|
2021-02-02
ยง
|
| 10:17 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1094 to clone db1174 - T258361', diff saved to https://phabricator.wikimedia.org/P14121 and previous config saved to /var/cache/conftool/dbconfig/20210202-100859-marostegui.json |
[production] |
| 10:08 |
<elukey@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:02 |
<hashar> |
Restarted Gerrit primary on gerrit1001 # T273223 |
[production] |
| 10:00 |
<hashar@deploy1001> |
Finished deploy [gerrit/gerrit@c3cd63b]: Gerrit primary on gerrit1001 to v3.2.7 T273223 (duration: 00m 09s) |
[production] |
| 10:00 |
<hashar@deploy1001> |
Started deploy [gerrit/gerrit@c3cd63b]: Gerrit primary on gerrit1001 to v3.2.7 T273223 |
[production] |
| 10:00 |
<hashar> |
Restarted Gerrit replica on gerrit2001 # T273223 |
[production] |
| 09:56 |
<hashar@deploy1001> |
Finished deploy [gerrit/gerrit@c3cd63b]: Gerrit replica on gerrit2001 to v3.2.7 T273223 (duration: 00m 12s) |
[production] |
| 09:56 |
<hashar@deploy1001> |
Started deploy [gerrit/gerrit@c3cd63b]: Gerrit replica on gerrit2001 to v3.2.7 T273223 |
[production] |
| 09:27 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1381.eqiad.wmnet |
[production] |
| 08:56 |
<XioNoX> |
disable DE-CIX codfw peering session |
[production] |
| 08:42 |
<elukey> |
decommission an-worker1117 from the Hadoop cluster, to move it under the Backup cluster |
[analytics] |
| 08:30 |
<godog> |
swift eqiad-prod: add weight back to sdg on ms-be1054 - T273582 |
[production] |
| 08:02 |
<legoktm> |
depooled mw1381.eqiad.wmnet for perf testing (T273312) |
[production] |
| 07:59 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1381.eqiad.wmnet |
[production] |
| 07:45 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1403.eqiad.wmnet |
[production] |
| 07:45 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1405.eqiad.wmnet |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14118 and previous config saved to /var/cache/conftool/dbconfig/20210202-073105-root.json |
[production] |
| 07:29 |
<dcaro> |
large VM wcqs-beta-01 is exhausting the hosts disk space (cloudvirt-wdqs1001) (T273579) |
[wikidata-query] |
| 07:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14117 and previous config saved to /var/cache/conftool/dbconfig/20210202-071602-root.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14116 and previous config saved to /var/cache/conftool/dbconfig/20210202-070057-root.json |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14115 and previous config saved to /var/cache/conftool/dbconfig/20210202-064553-root.json |
[production] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14114 and previous config saved to /var/cache/conftool/dbconfig/20210202-063050-root.json |
[production] |
| 06:24 |
<marostegui> |
Restart mysql on es1022 |
[production] |
| 06:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1022 T266483', diff saved to https://phabricator.wikimedia.org/P14113 and previous config saved to /var/cache/conftool/dbconfig/20210202-062303-marostegui.json |
[production] |
| 04:12 |
<ryankemper> |
[WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 03:36 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@ad9db35]: 0.3.62 (duration: 06m 59s) |
[production] |
| 03:36 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikimania-scholarships] |
| 03:33 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikidata-primary-sources-tool] |
| 03:29 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.62` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 03:29 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@ad9db35]: 0.3.62 |
[production] |
| 03:27 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikidata-realtime-dumps] |
| 03:26 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.62`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 03:21 |
<ryankemper> |
`sudo systemctl restart wdqs-blazegraph` on `wdqs1006` |
[production] |
| 03:12 |
<andrewbogott> |
running /usr/local/sbin/wmcs-purge-backups and /usr/local/sbin/wmcs-backup-instances on cloudvirt1024 to see why the backup job paged |
[admin] |
| 02:49 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikidata-federation] |
| 02:47 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[puppet] |
| 02:44 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[ogvjs-integration] |
| 02:41 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[meza] |
| 01:38 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[lizenzhinweisgenerator] |
| 01:37 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[ign2commons] |
| 01:35 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[finding-glams] |
| 01:33 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[commons-corruption-checker] |
| 01:29 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[blog] |
| 01:26 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge#SHUTDOWN_asyncwiki |
[asyncwiki] |