|
2021-05-12
§
|
| 10:46 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 180 days, 0:00:00 on cloudvirt1038.eqiad.wmnet with reason: T276922 |
[production] |
| 10:32 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: Initial deploy to cumin2002 - volans@cumin2002 |
[production] |
| 10:31 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2004.codfw.wmnet |
[production] |
| 10:29 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host poolcounter2004.codfw.wmnet |
[production] |
| 10:14 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2003.codfw.wmnet |
[production] |
| 10:01 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host poolcounter2003.codfw.wmnet |
[production] |
| 10:01 |
<effie> |
reboot poolcounter2003 and poolcounter2004 |
[production] |
| 09:55 |
<volans@cumin2002> |
START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: Initial deploy to cumin2002 - volans@cumin2002 |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317', diff saved to https://phabricator.wikimedia.org/P15940 and previous config saved to /var/cache/conftool/dbconfig/20210512-093333-marostegui.json |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 100%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15939 and previous config saved to /var/cache/conftool/dbconfig/20210512-093308-root.json |
[production] |
| 09:20 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1074.eqiad.wmnet |
[production] |
| 09:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 75%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15938 and previous config saved to /var/cache/conftool/dbconfig/20210512-091804-root.json |
[production] |
| 09:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1074.eqiad.wmnet |
[production] |
| 09:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 50%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15937 and previous config saved to /var/cache/conftool/dbconfig/20210512-090301-root.json |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 25%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15936 and previous config saved to /var/cache/conftool/dbconfig/20210512-084757-root.json |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1074 from dbctl T281959', diff saved to https://phabricator.wikimedia.org/P15935 and previous config saved to /var/cache/conftool/dbconfig/20210512-084755-marostegui.json |
[production] |
| 08:23 |
<jbond42> |
rolling restart of ats |
[production] |
| 07:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15934 and previous config saved to /var/cache/conftool/dbconfig/20210512-071017-root.json |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15933 and previous config saved to /var/cache/conftool/dbconfig/20210512-070202-marostegui.json |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15932 and previous config saved to /var/cache/conftool/dbconfig/20210512-065513-root.json |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 50%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15931 and previous config saved to /var/cache/conftool/dbconfig/20210512-064009-root.json |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15930 and previous config saved to /var/cache/conftool/dbconfig/20210512-062506-root.json |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P15929 and previous config saved to /var/cache/conftool/dbconfig/20210512-062118-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2121 and db2108 in s7 T282535', diff saved to https://phabricator.wikimedia.org/P15928 and previous config saved to /var/cache/conftool/dbconfig/20210512-062046-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 100%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15927 and previous config saved to /var/cache/conftool/dbconfig/20210512-061702-root.json |
[production] |
| 06:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Move db2148 to also serve vslow in s2 T282535', diff saved to https://phabricator.wikimedia.org/P15926 and previous config saved to /var/cache/conftool/dbconfig/20210512-060817-marostegui.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 75%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15925 and previous config saved to /var/cache/conftool/dbconfig/20210512-060158-root.json |
[production] |
| 05:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 50%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15924 and previous config saved to /var/cache/conftool/dbconfig/20210512-054655-root.json |
[production] |
| 05:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 25%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15923 and previous config saved to /var/cache/conftool/dbconfig/20210512-053151-root.json |
[production] |
| 05:00 |
<marostegui> |
Stop MySQL on labsdb1009 labsdb1010 labsdb1011 T282524 T282523 T282522 |
[production] |
| 04:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1181', diff saved to https://phabricator.wikimedia.org/P15922 and previous config saved to /var/cache/conftool/dbconfig/20210512-044728-marostegui.json |
[production] |
| 04:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2121 T282535', diff saved to https://phabricator.wikimedia.org/P15920 and previous config saved to /var/cache/conftool/dbconfig/20210512-044222-marostegui.json |
[production] |
| 04:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2108 T282535', diff saved to https://phabricator.wikimedia.org/P15919 and previous config saved to /var/cache/conftool/dbconfig/20210512-044109-marostegui.json |
[production] |
| 04:38 |
<marostegui> |
Drop testing mailman3 databases T281548 |
[production] |
| 04:36 |
<Amir1> |
importing archives of wikitech-l (T280322) |
[production] |
| 01:42 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host |
[production] |
| 01:42 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host |
[production] |
| 01:35 |
<mutante> |
people2002 - created new VM resembling people2001, signed puppet cert request, initial puppet run T280989 |
[production] |
| 01:19 |
<tstarling@deploy1002> |
Synchronized php-1.37.0-wmf.5/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 08s) |
[production] |
| 01:17 |
<tstarling@deploy1002> |
Synchronized php-1.37.0-wmf.4/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 16s) |
[production] |
| 00:54 |
<mutante> |
made public_html dirs on people1002 readonly to make it obvious it is not the active backend anymore |
[production] |
| 00:51 |
<mutante> |
[people1002:/home] $ sudo find . -type d -name public_html -exec chmod 555 {} \; |
[production] |