|
2021-02-02
§
|
| 07:45 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1405.eqiad.wmnet |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14118 and previous config saved to /var/cache/conftool/dbconfig/20210202-073105-root.json |
[production] |
| 07:29 |
<dcaro> |
large VM wcqs-beta-01 is exhausting the hosts disk space (cloudvirt-wdqs1001) (T273579) |
[wikidata-query] |
| 07:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14117 and previous config saved to /var/cache/conftool/dbconfig/20210202-071602-root.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14116 and previous config saved to /var/cache/conftool/dbconfig/20210202-070057-root.json |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14115 and previous config saved to /var/cache/conftool/dbconfig/20210202-064553-root.json |
[production] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repool es1022 after a restart', diff saved to https://phabricator.wikimedia.org/P14114 and previous config saved to /var/cache/conftool/dbconfig/20210202-063050-root.json |
[production] |
| 06:24 |
<marostegui> |
Restart mysql on es1022 |
[production] |
| 06:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1022 T266483', diff saved to https://phabricator.wikimedia.org/P14113 and previous config saved to /var/cache/conftool/dbconfig/20210202-062303-marostegui.json |
[production] |
| 04:12 |
<ryankemper> |
[WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 03:40 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 03:36 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@ad9db35]: 0.3.62 (duration: 06m 59s) |
[production] |
| 03:36 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikimania-scholarships] |
| 03:33 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikidata-primary-sources-tool] |
| 03:29 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.62` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 03:29 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@ad9db35]: 0.3.62 |
[production] |
| 03:27 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikidata-realtime-dumps] |
| 03:26 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.62`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 03:21 |
<ryankemper> |
`sudo systemctl restart wdqs-blazegraph` on `wdqs1006` |
[production] |
| 03:12 |
<andrewbogott> |
running /usr/local/sbin/wmcs-purge-backups and /usr/local/sbin/wmcs-backup-instances on cloudvirt1024 to see why the backup job paged |
[admin] |
| 02:49 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[wikidata-federation] |
| 02:47 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[puppet] |
| 02:44 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[ogvjs-integration] |
| 02:41 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[meza] |
| 01:38 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[lizenzhinweisgenerator] |
| 01:37 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[ign2commons] |
| 01:35 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[finding-glams] |
| 01:33 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[commons-corruption-checker] |
| 01:29 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge |
[blog] |
| 01:26 |
<andrewbogott> |
deleting project and all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge#SHUTDOWN_asyncwiki |
[asyncwiki] |
|
2021-02-01
§
|
| 23:54 |
<legoktm@deploy1001> |
Synchronized wmf-config/profiler.php: profiler: Send data to excimer-buster pipeline (T273312) (duration: 00m 57s) |
[production] |
| 23:15 |
<legoktm> |
depooling mw1403 and mw1405 for perf testing |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1405.eqiad.wmnet |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1403.eqiad.wmnet |
[production] |
| 23:14 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1278.eqiad.wmnet |
[production] |
| 23:05 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.28/extensions/Collection/includes/Specials/SpecialCollection.php: 3c7864ca1d5aadc9cd251939c0e23f661faef5e9: Remove unnecessary calls to WikiPage (T273101) (duration: 00m 58s) |
[production] |
| 22:09 |
<sbassett> |
Deployed security patch for T272386 |
[production] |
| 22:05 |
<sbassett> |
Deployed security patch for T270713 |
[production] |
| 22:04 |
<legoktm> |
depooling mw1278.eqiad.wmnet for perf testing |
[production] |
| 22:03 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1278.eqiad.wmnet |
[production] |
| 22:03 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1277.eqiad.wmnet |
[production] |
| 21:35 |
<wm-bot> |
<lucaswerkmeister> deployed f4e7ba98a7 (stop referrer-URL comparison) |
[tools.lexeme-forms] |
| 21:29 |
<dancy@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.27 |
[production] |
| 21:27 |
<razzi> |
rebalance kafka partitions for eqiad.mediawiki.job.cdnPurge |
[analytics] |
| 21:27 |
<razzi> |
rebalance kafka partitions for codfw.mediawiki.job.cdnPurge |
[analytics] |
| 20:53 |
<andrew@deploy1001> |
Finished deploy [striker/deploy@b6441b8]: Striker hacked fix for T272410 (duration: 00m 57s) |
[production] |