|
2019-12-19
ยง
|
| 09:49 |
<jmm@deploy1001> |
Finished deploy [debmonitor/deploy@c056c3c]: debmonitor release v0.2.1 - T237978 (duration: 02m 24s) |
[production] |
| 09:48 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:47 |
<jmm@deploy1001> |
Started deploy [debmonitor/deploy@c056c3c]: debmonitor release v0.2.1 - T237978 |
[production] |
| 09:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2084:3314, db2084:3315 T241103', diff saved to https://phabricator.wikimedia.org/P9966 and previous config saved to /var/cache/conftool/dbconfig/20191219-094135-marostegui.json |
[production] |
| 09:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P9965 and previous config saved to /var/cache/conftool/dbconfig/20191219-093959-marostegui.json |
[production] |
| 09:39 |
<ema> |
pool cp1087 with ATS backend T227432 |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093, s6 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9964 and previous config saved to /var/cache/conftool/dbconfig/20191219-093116-marostegui.json |
[production] |
| 09:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1081', diff saved to https://phabricator.wikimedia.org/P9963 and previous config saved to /var/cache/conftool/dbconfig/20191219-092920-marostegui.json |
[production] |
| 09:23 |
<ema> |
depool cp3064 and reimage as text_ats T227432 |
[production] |
| 09:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9962 and previous config saved to /var/cache/conftool/dbconfig/20191219-092257-marostegui.json |
[production] |
| 09:15 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<onimisionipe> |
running maps osm-replicate process manually on maps1004 - T239728 |
[production] |
| 09:13 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9961 and previous config saved to /var/cache/conftool/dbconfig/20191219-091205-marostegui.json |
[production] |
| 09:06 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9960 and previous config saved to /var/cache/conftool/dbconfig/20191219-090455-marostegui.json |
[production] |
| 09:03 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:55 |
<ema> |
depool cp1087 and reimage as text_ats T227432 |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081, s4 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9959 and previous config saved to /var/cache/conftool/dbconfig/20191219-084544-marostegui.json |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 - depooled by mistake', diff saved to https://phabricator.wikimedia.org/P9958 and previous config saved to /var/cache/conftool/dbconfig/20191219-084518-marostegui.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081, s4 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9957 and previous config saved to /var/cache/conftool/dbconfig/20191219-084346-marostegui.json |
[production] |
| 08:31 |
<ema> |
pool cp1085 with ATS backend T227432 |
[production] |
| 08:27 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size=5 --from-id=30398836 # T237984 |
[production] |
| 08:21 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:18 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:17 |
<marostegui> |
Restart mysql on labsdb1010 (after depooling it) |
[production] |
| 08:11 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:09 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1014', diff saved to https://phabricator.wikimedia.org/P9956 and previous config saved to /var/cache/conftool/dbconfig/20191219-080519-marostegui.json |
[production] |
| 07:53 |
<ema> |
depool cp1085 and reimage as text_ats T227432 |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1014', diff saved to https://phabricator.wikimedia.org/P9955 and previous config saved to /var/cache/conftool/dbconfig/20191219-074840-marostegui.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1076', diff saved to https://phabricator.wikimedia.org/P9954 and previous config saved to /var/cache/conftool/dbconfig/20191219-074800-marostegui.json |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1014', diff saved to https://phabricator.wikimedia.org/P9953 and previous config saved to /var/cache/conftool/dbconfig/20191219-074122-marostegui.json |
[production] |
| 07:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1076', diff saved to https://phabricator.wikimedia.org/P9952 and previous config saved to /var/cache/conftool/dbconfig/20191219-073907-marostegui.json |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1014', diff saved to https://phabricator.wikimedia.org/P9951 and previous config saved to /var/cache/conftool/dbconfig/20191219-073430-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1076', diff saved to https://phabricator.wikimedia.org/P9950 and previous config saved to /var/cache/conftool/dbconfig/20191219-073151-marostegui.json |
[production] |
| 07:30 |
<ema> |
cp: rolling ats-backend-restart to disable compress plugin everywhere T238494 |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1014 for upgrade', diff saved to https://phabricator.wikimedia.org/P9949 and previous config saved to /var/cache/conftool/dbconfig/20191219-072728-marostegui.json |
[production] |
| 07:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1076', diff saved to https://phabricator.wikimedia.org/P9948 and previous config saved to /var/cache/conftool/dbconfig/20191219-072413-marostegui.json |
[production] |
| 07:19 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:17 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1076, s2 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9947 and previous config saved to /var/cache/conftool/dbconfig/20191219-071514-marostegui.json |
[production] |
| 07:10 |
<marostegui> |
Upgrade db1115 (this will make dbtree fail for a few minutes) |
[production] |
| 06:51 |
<XioNoX> |
cr2-eqdfw:delete chassis alarm management-ethernet - T241105 |
[production] |
| 06:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1008 after upgrade (duration: 01m 02s) |
[production] |
| 06:37 |
<marostegui> |
Upgrade pc1008 |
[production] |
| 06:36 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool pc1008 for upgrade (duration: 01m 03s) |
[production] |
| 06:15 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:13 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:13 |
<marostegui> |
Upgrade db2122, db2084 |
[production] |