|
2019-11-20
ยง
|
| 11:36 |
<urbanecm@deploy1001> |
Finished scap: SWAT: 44ec4e4: e1baf0e: 3c02aa7: Namespace changes (duration: 06m 15s) |
[production] |
| 11:30 |
<urbanecm@deploy1001> |
Started scap: SWAT: 44ec4e4: e1baf0e: 3c02aa7: Namespace changes |
[production] |
| 11:27 |
<ema> |
cp2010: ats-backend-restart to clear backend restart alert |
[production] |
| 11:21 |
<ema> |
depool cp2012 and reimage as text_ats T227432 |
[production] |
| 11:15 |
<ema> |
pool cp2010 with ATS backend T227432 |
[production] |
| 10:54 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:52 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:36 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@daa7808]: Revert switching test2.wp to Parsoid/JS - T238716 (duration: 13m 56s) |
[production] |
| 10:34 |
<ema> |
depool cp2010 and reimage as text_ats T227432 |
[production] |
| 10:30 |
<marostegui> |
Upgrade db1116 |
[production] |
| 10:22 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@daa7808]: Revert switching test2.wp to Parsoid/JS - T238716 |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P9694 and previous config saved to /var/cache/conftool/dbconfig/20191120-101727-marostegui.json |
[production] |
| 10:14 |
<marostegui> |
Compress db2095:3314 |
[production] |
| 10:07 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@c677063]: Switch test2.wp back to Parsoid/JS temporarily - T238716 (duration: 14m 54s) |
[production] |
| 09:56 |
<marostegui> |
Compress db2106 |
[production] |
| 09:52 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@c677063]: Switch test2.wp back to Parsoid/JS temporarily - T238716 |
[production] |
| 09:48 |
<marostegui> |
Compress dbstore1005:3318 |
[production] |
| 09:47 |
<marostegui> |
Compress dbstore1004:3314 |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1094 after upgrade', diff saved to https://phabricator.wikimedia.org/P9693 and previous config saved to /var/cache/conftool/dbconfig/20191120-093308-marostegui.json |
[production] |
| 09:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1094 after upgrade', diff saved to https://phabricator.wikimedia.org/P9692 and previous config saved to /var/cache/conftool/dbconfig/20191120-092337-marostegui.json |
[production] |
| 09:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1094 after upgrade', diff saved to https://phabricator.wikimedia.org/P9691 and previous config saved to /var/cache/conftool/dbconfig/20191120-090739-marostegui.json |
[production] |
| 08:55 |
<marostegui> |
Upgrade db1094 |
[production] |
| 08:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1094 for upgrade', diff saved to https://phabricator.wikimedia.org/P9690 and previous config saved to /var/cache/conftool/dbconfig/20191120-085448-marostegui.json |
[production] |
| 08:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 08:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:43 |
<marostegui> |
Promote db2132 as m1-codfw master - T238183 |
[production] |
| 07:19 |
<marostegui> |
Upgrade db2062 |
[production] |
| 07:19 |
<marostegui> |
Upgrade db2078 |
[production] |
| 07:14 |
<marostegui> |
Deploy schema change on s3 (testwikidatawiki) directly on s3 primary master T237120 |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1136', diff saved to https://phabricator.wikimedia.org/P9688 and previous config saved to /var/cache/conftool/dbconfig/20191120-070511-marostegui.json |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give more traffic to db1136', diff saved to https://phabricator.wikimedia.org/P9687 and previous config saved to /var/cache/conftool/dbconfig/20191120-065718-marostegui.json |
[production] |
| 06:44 |
<marostegui> |
Upgrade db2118 (s7 codfw master) |
[production] |
| 06:41 |
<marostegui> |
Repool labsdb1011 |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1136 into s7 api', diff saved to https://phabricator.wikimedia.org/P9686 and previous config saved to /var/cache/conftool/dbconfig/20191120-064022-marostegui.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1136 after upgrade', diff saved to https://phabricator.wikimedia.org/P9685 and previous config saved to /var/cache/conftool/dbconfig/20191120-063628-marostegui.json |
[production] |
| 06:28 |
<marostegui> |
Upgrade db1136 |
[production] |
| 06:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136 for upgrade', diff saved to https://phabricator.wikimedia.org/P9684 and previous config saved to /var/cache/conftool/dbconfig/20191120-062749-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317 after upgrade', diff saved to https://phabricator.wikimedia.org/P9683 and previous config saved to /var/cache/conftool/dbconfig/20191120-062029-marostegui.json |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3314 for compression', diff saved to https://phabricator.wikimedia.org/P9682 and previous config saved to /var/cache/conftool/dbconfig/20191120-061938-marostegui.json |
[production] |
| 05:58 |
<marostegui> |
Stop MySQL on db1101:3317, db1101:3318 for upgrade and schema change |
[production] |
| 05:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317 and db1101:3318 for upgrade and schema change', diff saved to https://phabricator.wikimedia.org/P9681 and previous config saved to /var/cache/conftool/dbconfig/20191120-055732-marostegui.json |
[production] |
| 05:55 |
<marostegui> |
Depool labsdb1011 for upgrade |
[production] |
| 05:54 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Repool db1105:3311 db1097:3314 db1098:3316 db1098:3317 after compression', diff saved to https://phabricator.wikimedia.org/P9680 and previous config saved to /var/cache/conftool/dbconfig/20191120-055426-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P9679 and previous config saved to /var/cache/conftool/dbconfig/20191120-054840-marostegui.json |
[production] |
| 03:16 |
<tgr> |
T208369 ran mwscript extensions/GrowthExperiments/maintenance/deleteOldSurveys.php kowiki --cutoff 350 |
[production] |
| 02:57 |
<vgutierrez> |
restarting pybal on lvs2002 |
[production] |
| 02:54 |
<vgutierrez> |
restarting pybal on lvs2005 |
[production] |
| 02:32 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 02:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 00:10 |
<mutante> |
phab2001 - restart ssh-phab service after repooling it after buster reinstall, it wasn't listening on the IPv6 IP,causing LVS/pybal alerts |
[production] |