|
2019-12-19
ยง
|
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9961 and previous config saved to /var/cache/conftool/dbconfig/20191219-091205-marostegui.json |
[production] |
| 09:06 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9960 and previous config saved to /var/cache/conftool/dbconfig/20191219-090455-marostegui.json |
[production] |
| 09:03 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:55 |
<ema> |
depool cp1087 and reimage as text_ats T227432 |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081, s4 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9959 and previous config saved to /var/cache/conftool/dbconfig/20191219-084544-marostegui.json |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 - depooled by mistake', diff saved to https://phabricator.wikimedia.org/P9958 and previous config saved to /var/cache/conftool/dbconfig/20191219-084518-marostegui.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081, s4 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9957 and previous config saved to /var/cache/conftool/dbconfig/20191219-084346-marostegui.json |
[production] |
| 08:31 |
<ema> |
pool cp1085 with ATS backend T227432 |
[production] |
| 08:27 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size=5 --from-id=30398836 # T237984 |
[production] |
| 08:21 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:18 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:17 |
<marostegui> |
Restart mysql on labsdb1010 (after depooling it) |
[production] |
| 08:11 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:09 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1014', diff saved to https://phabricator.wikimedia.org/P9956 and previous config saved to /var/cache/conftool/dbconfig/20191219-080519-marostegui.json |
[production] |
| 07:53 |
<ema> |
depool cp1085 and reimage as text_ats T227432 |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1014', diff saved to https://phabricator.wikimedia.org/P9955 and previous config saved to /var/cache/conftool/dbconfig/20191219-074840-marostegui.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1076', diff saved to https://phabricator.wikimedia.org/P9954 and previous config saved to /var/cache/conftool/dbconfig/20191219-074800-marostegui.json |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1014', diff saved to https://phabricator.wikimedia.org/P9953 and previous config saved to /var/cache/conftool/dbconfig/20191219-074122-marostegui.json |
[production] |
| 07:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1076', diff saved to https://phabricator.wikimedia.org/P9952 and previous config saved to /var/cache/conftool/dbconfig/20191219-073907-marostegui.json |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1014', diff saved to https://phabricator.wikimedia.org/P9951 and previous config saved to /var/cache/conftool/dbconfig/20191219-073430-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1076', diff saved to https://phabricator.wikimedia.org/P9950 and previous config saved to /var/cache/conftool/dbconfig/20191219-073151-marostegui.json |
[production] |
| 07:30 |
<ema> |
cp: rolling ats-backend-restart to disable compress plugin everywhere T238494 |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1014 for upgrade', diff saved to https://phabricator.wikimedia.org/P9949 and previous config saved to /var/cache/conftool/dbconfig/20191219-072728-marostegui.json |
[production] |
| 07:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1076', diff saved to https://phabricator.wikimedia.org/P9948 and previous config saved to /var/cache/conftool/dbconfig/20191219-072413-marostegui.json |
[production] |
| 07:19 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:17 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1076, s2 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9947 and previous config saved to /var/cache/conftool/dbconfig/20191219-071514-marostegui.json |
[production] |
| 07:10 |
<marostegui> |
Upgrade db1115 (this will make dbtree fail for a few minutes) |
[production] |
| 06:51 |
<XioNoX> |
cr2-eqdfw:delete chassis alarm management-ethernet - T241105 |
[production] |
| 06:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1008 after upgrade (duration: 01m 02s) |
[production] |
| 06:37 |
<marostegui> |
Upgrade pc1008 |
[production] |
| 06:36 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool pc1008 for upgrade (duration: 01m 03s) |
[production] |
| 06:15 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:13 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:13 |
<marostegui> |
Upgrade db2122, db2084 |
[production] |
| 05:09 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:07 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:02 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:00 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:59 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:58 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:29 |
<kart_> |
Update cxserver to 2019-12-11-144337-production (T233405, T238118) |
[production] |
| 04:25 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 04:21 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 04:19 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 04:12 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:10 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:03 |
<mutante> |
LDAP - added mstyles to archiva-deployers (T240865) |
[production] |