|
2021-05-12
§
|
| 10:01 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host poolcounter2003.codfw.wmnet |
[production] |
| 10:01 |
<effie> |
reboot poolcounter2003 and poolcounter2004 |
[production] |
| 09:55 |
<volans@cumin2002> |
START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: Initial deploy to cumin2002 - volans@cumin2002 |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317', diff saved to https://phabricator.wikimedia.org/P15940 and previous config saved to /var/cache/conftool/dbconfig/20210512-093333-marostegui.json |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 100%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15939 and previous config saved to /var/cache/conftool/dbconfig/20210512-093308-root.json |
[production] |
| 09:20 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1074.eqiad.wmnet |
[production] |
| 09:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 75%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15938 and previous config saved to /var/cache/conftool/dbconfig/20210512-091804-root.json |
[production] |
| 09:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1074.eqiad.wmnet |
[production] |
| 09:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 50%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15937 and previous config saved to /var/cache/conftool/dbconfig/20210512-090301-root.json |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 25%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15936 and previous config saved to /var/cache/conftool/dbconfig/20210512-084757-root.json |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1074 from dbctl T281959', diff saved to https://phabricator.wikimedia.org/P15935 and previous config saved to /var/cache/conftool/dbconfig/20210512-084755-marostegui.json |
[production] |
| 08:23 |
<jbond42> |
rolling restart of ats |
[production] |
| 07:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15934 and previous config saved to /var/cache/conftool/dbconfig/20210512-071017-root.json |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15933 and previous config saved to /var/cache/conftool/dbconfig/20210512-070202-marostegui.json |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15932 and previous config saved to /var/cache/conftool/dbconfig/20210512-065513-root.json |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 50%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15931 and previous config saved to /var/cache/conftool/dbconfig/20210512-064009-root.json |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15930 and previous config saved to /var/cache/conftool/dbconfig/20210512-062506-root.json |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P15929 and previous config saved to /var/cache/conftool/dbconfig/20210512-062118-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2121 and db2108 in s7 T282535', diff saved to https://phabricator.wikimedia.org/P15928 and previous config saved to /var/cache/conftool/dbconfig/20210512-062046-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 100%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15927 and previous config saved to /var/cache/conftool/dbconfig/20210512-061702-root.json |
[production] |
| 06:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Move db2148 to also serve vslow in s2 T282535', diff saved to https://phabricator.wikimedia.org/P15926 and previous config saved to /var/cache/conftool/dbconfig/20210512-060817-marostegui.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 75%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15925 and previous config saved to /var/cache/conftool/dbconfig/20210512-060158-root.json |
[production] |
| 05:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 50%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15924 and previous config saved to /var/cache/conftool/dbconfig/20210512-054655-root.json |
[production] |
| 05:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1181 (re)pooling @ 25%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15923 and previous config saved to /var/cache/conftool/dbconfig/20210512-053151-root.json |
[production] |
| 05:00 |
<marostegui> |
Stop MySQL on labsdb1009 labsdb1010 labsdb1011 T282524 T282523 T282522 |
[production] |
| 04:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1181', diff saved to https://phabricator.wikimedia.org/P15922 and previous config saved to /var/cache/conftool/dbconfig/20210512-044728-marostegui.json |
[production] |
| 04:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2121 T282535', diff saved to https://phabricator.wikimedia.org/P15920 and previous config saved to /var/cache/conftool/dbconfig/20210512-044222-marostegui.json |
[production] |
| 04:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2108 T282535', diff saved to https://phabricator.wikimedia.org/P15919 and previous config saved to /var/cache/conftool/dbconfig/20210512-044109-marostegui.json |
[production] |
| 04:38 |
<marostegui> |
Drop testing mailman3 databases T281548 |
[production] |
| 04:36 |
<Amir1> |
importing archives of wikitech-l (T280322) |
[production] |
| 01:42 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host |
[production] |
| 01:42 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host |
[production] |
| 01:35 |
<mutante> |
people2002 - created new VM resembling people2001, signed puppet cert request, initial puppet run T280989 |
[production] |
| 01:19 |
<tstarling@deploy1002> |
Synchronized php-1.37.0-wmf.5/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 08s) |
[production] |
| 01:17 |
<tstarling@deploy1002> |
Synchronized php-1.37.0-wmf.4/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 16s) |
[production] |
| 00:54 |
<mutante> |
made public_html dirs on people1002 readonly to make it obvious it is not the active backend anymore |
[production] |
| 00:51 |
<mutante> |
[people1002:/home] $ sudo find . -type d -name public_html -exec chmod 555 {} \; |
[production] |
|
2021-05-11
§
|
| 23:19 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: ec37795eba5faa0c1a1dddb29504941205e155b4: Change namespace names and aliases on tiwiki and tiwiktionary (T263840) (duration: 01m 07s) |
[production] |
| 23:16 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 5bc40acfa6514f7940af0a6cef9974140680f4b9: ptwiki: Use celebration logos in new vector (T281925) (duration: 01m 06s) |
[production] |
| 23:08 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: eac843a69574feebbf962959e5eb9811a2a83bc4: Make DT source mode toolbar available as beta on all wikis (T279124) (duration: 01m 12s) |
[production] |
| 23:06 |
<urbanecm@deploy1002> |
Synchronized static/images/mobile/copyright/wikipedia-pt-20.png: 60e6e4e960ee6cb31df9ce08fdeaedb647ce3afb: ptwiki: Add wikipedia-pt-20.png (T281925) (duration: 01m 08s) |
[production] |
| 23:02 |
<urbanecm@deploy1002> |
Synchronized static/images/mobile/copyright/: e35199baf01f423015905b8fec9e419ed3529787: Adding square logo and wordmark for ptwiki 20 years celebration (T281925) (duration: 01m 50s) |
[production] |
| 22:14 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lists1002.wikimedia.org |
[production] |
| 22:05 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts lists1002.wikimedia.org |
[production] |
| 21:37 |
<Urbanecm> |
Start server-side upload for 3 video files (T282566, T282565, T282559) |
[production] |
| 21:37 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1012.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:34 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1012.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:52 |
<legoktm> |
upgraded mailman3 on lists1001 |
[production] |
| 20:37 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host people2002.codfw.wmnet |
[production] |
| 20:24 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@270c753] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 06m 57s) |
[production] |