|
2021-03-15
§
|
| 11:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 100%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P14847 and previous config saved to /var/cache/conftool/dbconfig/20210315-115826-root.json |
[production] |
| 11:51 |
<jdrewniak@deploy1002> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:672371| Bumping portals to master (T128546)]] (duration: 00m 58s) |
[production] |
| 11:50 |
<jdrewniak@deploy1002> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:672371| Bumping portals to master (T128546)]] (duration: 00m 59s) |
[production] |
| 11:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 75%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P14846 and previous config saved to /var/cache/conftool/dbconfig/20210315-114323-root.json |
[production] |
| 11:34 |
<moritzm> |
restarting FPM on mw canaries to pick up new libtiff |
[production] |
| 11:30 |
<aborrero@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 11:28 |
<aborrero@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 11:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 50%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P14844 and previous config saved to /var/cache/conftool/dbconfig/20210315-112819-root.json |
[production] |
| 11:22 |
<moritzm> |
installing tiff security updates |
[production] |
| 11:17 |
<moritzm> |
installing golang-1.7 security updates |
[production] |
| 11:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 25%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P14843 and previous config saved to /var/cache/conftool/dbconfig/20210315-111315-root.json |
[production] |
| 11:00 |
<volans> |
upgraded spicerack on cumin1001 to 0.0.49-1+deb10u1 |
[production] |
| 10:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1129', diff saved to https://phabricator.wikimedia.org/P14842 and previous config saved to /var/cache/conftool/dbconfig/20210315-105855-marostegui.json |
[production] |
| 10:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1076 (re)pooling @ 100%: Repool db1076', diff saved to https://phabricator.wikimedia.org/P14841 and previous config saved to /var/cache/conftool/dbconfig/20210315-105820-root.json |
[production] |
| 10:56 |
<volans@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on cumin2001.codfw.wmnet with reason: test |
[production] |
| 10:55 |
<volans@cumin2001> |
START - Cookbook sre.hosts.downtime for 0:05:00 on cumin2001.codfw.wmnet with reason: test |
[production] |
| 10:52 |
<arturo> |
extending storage quota: `aborrero@cloudcontrol1005:~$ sudo wmcs-openstack quota set --gigabytes 350 cyberbot` (T277333) |
[cyberbot] |
| 10:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1076 (re)pooling @ 75%: Repool db1076', diff saved to https://phabricator.wikimedia.org/P14840 and previous config saved to /var/cache/conftool/dbconfig/20210315-104316-root.json |
[production] |
| 10:42 |
<moritzm> |
installing pygments security updates on buster |
[production] |
| 10:33 |
<volans> |
upgraded spicerack on cumin2001 to 0.0.49-1+deb10u1 |
[production] |
| 10:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1076 (re)pooling @ 50%: Repool db1076', diff saved to https://phabricator.wikimedia.org/P14839 and previous config saved to /var/cache/conftool/dbconfig/20210315-102813-root.json |
[production] |
| 10:26 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db1114 (re)pooling @ 100%: schema change T267767', diff saved to https://phabricator.wikimedia.org/P14838 and previous config saved to /var/cache/conftool/dbconfig/20210315-102648-kormat.json |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1076 (re)pooling @ 25%: Repool db1076', diff saved to https://phabricator.wikimedia.org/P14837 and previous config saved to /var/cache/conftool/dbconfig/20210315-101309-root.json |
[production] |
| 10:11 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db1114 (re)pooling @ 50%: schema change T267767', diff saved to https://phabricator.wikimedia.org/P14836 and previous config saved to /var/cache/conftool/dbconfig/20210315-101143-kormat.json |
[production] |
| 10:03 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db1114 depooling: schema change T267767', diff saved to https://phabricator.wikimedia.org/P14835 and previous config saved to /var/cache/conftool/dbconfig/20210315-100337-kormat.json |
[production] |
| 10:03 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1114.eqiad.wmnet with reason: schema change T267767 |
[production] |
| 10:02 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db1114.eqiad.wmnet with reason: schema change T267767 |
[production] |
| 09:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P14834 and previous config saved to /var/cache/conftool/dbconfig/20210315-095607-marostegui.json |
[production] |
| 09:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 100%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P14833 and previous config saved to /var/cache/conftool/dbconfig/20210315-094920-root.json |
[production] |
| 09:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 75%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P14832 and previous config saved to /var/cache/conftool/dbconfig/20210315-093416-root.json |
[production] |
| 09:23 |
<vgutierrez> |
rolling restart of LVS cluster to bump depool_threshold to 0.8 on text & upload clusters - T274888 |
[production] |
| 09:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 50%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P14831 and previous config saved to /var/cache/conftool/dbconfig/20210315-091912-root.json |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 25%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P14830 and previous config saved to /var/cache/conftool/dbconfig/20210315-090409-root.json |
[production] |
| 08:54 |
<marostegui> |
Stop MySQL on db1136 T277007 |
[production] |
| 08:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136 T277007', diff saved to https://phabricator.wikimedia.org/P14829 and previous config saved to /var/cache/conftool/dbconfig/20210315-085409-marostegui.json |
[production] |
| 08:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1146:3312', diff saved to https://phabricator.wikimedia.org/P14828 and previous config saved to /var/cache/conftool/dbconfig/20210315-083555-marostegui.json |
[production] |
| 08:33 |
<godog> |
swift eqiad-prod remove decom hosts from account/container rings - T272836 T276193 |
[production] |
| 08:33 |
<marostegui> |
Repool labsdb1009 T276980 |
[production] |
| 08:25 |
<hashar> |
Reloaded Zuul for https://gerrit.wikimedia.org/r/670782 |
[releng] |
| 08:25 |
<elukey> |
stop/start hdfs-balancer on an-launcher1002 with bw 200MB |
[analytics] |
| 08:13 |
<hashar> |
Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/672118 |
[releng] |
| 07:48 |
<joal> |
Manually start mediawiki-history-drop-snapshot.service to check the run succeeds |
[analytics] |
| 07:47 |
<joal> |
Drop hive wmf.mediawiki_wikitext_history snapshot partitions (2020-08, 2020-09, 2020-10, 2020-11) |
[analytics] |
| 07:22 |
<elukey> |
powercycle ms-be1038 - no ssh, no tty available in mgmt serial console, irrecoverable error saved in ilo's system logs |
[production] |