|
2019-12-19
ยง
|
| 12:35 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:32 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Cleanup: Remove very old 'error_append_string' INI override (duration: 01m 02s) |
[production] |
| 12:30 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Cleanup: Move core DB/SQL-related config closer together (duration: 01m 02s) |
[production] |
| 12:03 |
<moritzm> |
installing netflow4001 |
[production] |
| 11:43 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size=5 --from-id=4089887 # T237984 - Will stop after P433 |
[production] |
| 11:28 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:26 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:05 |
<moritzm> |
removing kubestagetcd1001-1003 from debmonitor T224568 |
[production] |
| 11:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P9974 and previous config saved to /var/cache/conftool/dbconfig/20191219-110404-marostegui.json |
[production] |
| 10:50 |
<ema> |
pool cp3064 with ATS backend T227432 |
[production] |
| 10:40 |
<effie> |
pool mw1320 |
[production] |
| 10:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P9973 and previous config saved to /var/cache/conftool/dbconfig/20191219-102316-marostegui.json |
[production] |
| 10:21 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:18 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P9972 and previous config saved to /var/cache/conftool/dbconfig/20191219-101024-marostegui.json |
[production] |
| 10:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1093', diff saved to https://phabricator.wikimedia.org/P9971 and previous config saved to /var/cache/conftool/dbconfig/20191219-100938-marostegui.json |
[production] |
| 10:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P9970 and previous config saved to /var/cache/conftool/dbconfig/20191219-095959-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P9969 and previous config saved to /var/cache/conftool/dbconfig/20191219-095559-marostegui.json |
[production] |
| 09:53 |
<effie> |
depool mw1320 |
[production] |
| 09:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104, s8 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9968 and previous config saved to /var/cache/conftool/dbconfig/20191219-095158-marostegui.json |
[production] |
| 09:50 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P9967 and previous config saved to /var/cache/conftool/dbconfig/20191219-094945-marostegui.json |
[production] |
| 09:49 |
<jmm@deploy1001> |
Finished deploy [debmonitor/deploy@c056c3c]: debmonitor release v0.2.1 - T237978 (duration: 02m 24s) |
[production] |
| 09:48 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:47 |
<jmm@deploy1001> |
Started deploy [debmonitor/deploy@c056c3c]: debmonitor release v0.2.1 - T237978 |
[production] |
| 09:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2084:3314, db2084:3315 T241103', diff saved to https://phabricator.wikimedia.org/P9966 and previous config saved to /var/cache/conftool/dbconfig/20191219-094135-marostegui.json |
[production] |
| 09:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P9965 and previous config saved to /var/cache/conftool/dbconfig/20191219-093959-marostegui.json |
[production] |
| 09:39 |
<ema> |
pool cp1087 with ATS backend T227432 |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093, s6 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9964 and previous config saved to /var/cache/conftool/dbconfig/20191219-093116-marostegui.json |
[production] |
| 09:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1081', diff saved to https://phabricator.wikimedia.org/P9963 and previous config saved to /var/cache/conftool/dbconfig/20191219-092920-marostegui.json |
[production] |
| 09:23 |
<ema> |
depool cp3064 and reimage as text_ats T227432 |
[production] |
| 09:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9962 and previous config saved to /var/cache/conftool/dbconfig/20191219-092257-marostegui.json |
[production] |
| 09:15 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<onimisionipe> |
running maps osm-replicate process manually on maps1004 - T239728 |
[production] |
| 09:13 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9961 and previous config saved to /var/cache/conftool/dbconfig/20191219-091205-marostegui.json |
[production] |
| 09:06 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P9960 and previous config saved to /var/cache/conftool/dbconfig/20191219-090455-marostegui.json |
[production] |
| 09:03 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:55 |
<ema> |
depool cp1087 and reimage as text_ats T227432 |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081, s4 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9959 and previous config saved to /var/cache/conftool/dbconfig/20191219-084544-marostegui.json |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 - depooled by mistake', diff saved to https://phabricator.wikimedia.org/P9958 and previous config saved to /var/cache/conftool/dbconfig/20191219-084518-marostegui.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081, s4 candidate master, for upgrade', diff saved to https://phabricator.wikimedia.org/P9957 and previous config saved to /var/cache/conftool/dbconfig/20191219-084346-marostegui.json |
[production] |
| 08:31 |
<ema> |
pool cp1085 with ATS backend T227432 |
[production] |
| 08:27 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size=5 --from-id=30398836 # T237984 |
[production] |
| 08:21 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:18 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:17 |
<marostegui> |
Restart mysql on labsdb1010 (after depooling it) |
[production] |
| 08:11 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:09 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |