|
2020-07-27
ยง
|
| 16:59 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:58 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:57 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephosd1003.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephosd1002.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephosd1001.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1003.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:49 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1001.eqiad.wmnet |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephosd1003.wikimedia.org |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephosd1002.wikimedia.org |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephosd1001.wikimedia.org |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephosd1001.eqiad.wmnet |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephosd1002.eqiad.wmnet |
[production] |
| 16:47 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephosd1003.eqiad.wmnet |
[production] |
| 16:44 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cumin1001.eqiad.wmnet |
[production] |
| 16:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2087:3316, db2087:3317 after on-site maintenance T258587', diff saved to https://phabricator.wikimedia.org/P12063 and previous config saved to /var/cache/conftool/dbconfig/20200727-163311-marostegui.json |
[production] |
| 16:05 |
<marostegui> |
Will show up on labsdb hosts for s5 |
[production] |
| 16:04 |
<marostegui> |
Stop MySQL on db1082 for onsite maintenance - T258910 |
[production] |
| 15:03 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:03 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:57 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:55 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Reduce db1146:3314 weight while db1144:3314 is depooled', diff saved to https://phabricator.wikimedia.org/P12060 and previous config saved to /var/cache/conftool/dbconfig/20200727-145010-marostegui.json |
[production] |
| 14:48 |
<marostegui> |
Deploy MCR change on db1144:3314 |
[production] |
| 14:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3314 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12059 and previous config saved to /var/cache/conftool/dbconfig/20200727-144807-marostegui.json |
[production] |
| 14:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1149', diff saved to https://phabricator.wikimedia.org/P12058 and previous config saved to /var/cache/conftool/dbconfig/20200727-144034-marostegui.json |
[production] |
| 14:21 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:19 |
<XioNoX> |
standardize cr1-codfw interfaces |
[production] |
| 14:19 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:07 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:04 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:57 |
<moritzm> |
upgrading idp2001 to CAS 6.1.7.1 |
[production] |
| 13:19 |
<XioNoX> |
standardize some cr2-esams interfaces |
[production] |
| 13:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More weight to db1089 in main traffic', diff saved to https://phabricator.wikimedia.org/P12057 and previous config saved to /var/cache/conftool/dbconfig/20200727-131123-marostegui.json |
[production] |
| 13:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1105:3311 with normal weight and pool db1089 into vslow', diff saved to https://phabricator.wikimedia.org/P12056 and previous config saved to /var/cache/conftool/dbconfig/20200727-130954-marostegui.json |
[production] |
| 13:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12055 and previous config saved to /var/cache/conftool/dbconfig/20200727-130713-marostegui.json |
[production] |
| 13:03 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:01 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1105:3311 with less weight', diff saved to https://phabricator.wikimedia.org/P12054 and previous config saved to /var/cache/conftool/dbconfig/20200727-125824-marostegui.json |
[production] |
| 12:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12053 and previous config saved to /var/cache/conftool/dbconfig/20200727-125351-marostegui.json |
[production] |
| 12:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1105:3311 with less weight', diff saved to https://phabricator.wikimedia.org/P12052 and previous config saved to /var/cache/conftool/dbconfig/20200727-125207-marostegui.json |
[production] |
| 12:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12051 and previous config saved to /var/cache/conftool/dbconfig/20200727-125045-marostegui.json |
[production] |
| 12:41 |
<marostegui> |
Compress innodb on db1106, this will generate lag on enwiki on labsdb hosts (wiki replicas) T254462 |
[production] |
| 12:38 |
<moritzm> |
disable puppet on idp1001/2001 |
[production] |
| 12:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 and pool db1105:3311 as vslow T254462', diff saved to https://phabricator.wikimedia.org/P12050 and previous config saved to /var/cache/conftool/dbconfig/20200727-123833-marostegui.json |
[production] |
| 12:37 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet |
[production] |
| 12:37 |
<akosiaris@cumin1001> |
conftool action : set/weight=0; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet |
[production] |
| 12:37 |
<akosiaris@cumin1001> |
conftool action : set/weight=10; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet |
[production] |
| 12:37 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet |
[production] |