1051-1100 of 10000 results (28ms)
2020-07-27 ยง
16:48 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cloudcephosd1002.eqiad.wmnet [production]
16:47 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cloudcephosd1003.eqiad.wmnet [production]
16:44 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cumin1001.eqiad.wmnet [production]
16:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2087:3316, db2087:3317 after on-site maintenance T258587', diff saved to https://phabricator.wikimedia.org/P12063 and previous config saved to /var/cache/conftool/dbconfig/20200727-163311-marostegui.json [production]
16:05 <marostegui> Will show up on labsdb hosts for s5 [production]
16:04 <marostegui> Stop MySQL on db1082 for onsite maintenance - T258910 [production]
15:03 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:03 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:57 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:55 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Reduce db1146:3314 weight while db1144:3314 is depooled', diff saved to https://phabricator.wikimedia.org/P12060 and previous config saved to /var/cache/conftool/dbconfig/20200727-145010-marostegui.json [production]
14:48 <marostegui> Deploy MCR change on db1144:3314 [production]
14:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1144:3314 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12059 and previous config saved to /var/cache/conftool/dbconfig/20200727-144807-marostegui.json [production]
14:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1149', diff saved to https://phabricator.wikimedia.org/P12058 and previous config saved to /var/cache/conftool/dbconfig/20200727-144034-marostegui.json [production]
14:21 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:19 <XioNoX> standardize cr1-codfw interfaces [production]
14:19 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:07 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:04 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:57 <moritzm> upgrading idp2001 to CAS 6.1.7.1 [production]
13:19 <XioNoX> standardize some cr2-esams interfaces [production]
13:11 <marostegui@cumin1001> dbctl commit (dc=all): 'More weight to db1089 in main traffic', diff saved to https://phabricator.wikimedia.org/P12057 and previous config saved to /var/cache/conftool/dbconfig/20200727-131123-marostegui.json [production]
13:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1105:3311 with normal weight and pool db1089 into vslow', diff saved to https://phabricator.wikimedia.org/P12056 and previous config saved to /var/cache/conftool/dbconfig/20200727-130954-marostegui.json [production]
13:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12055 and previous config saved to /var/cache/conftool/dbconfig/20200727-130713-marostegui.json [production]
13:03 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:01 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1105:3311 with less weight', diff saved to https://phabricator.wikimedia.org/P12054 and previous config saved to /var/cache/conftool/dbconfig/20200727-125824-marostegui.json [production]
12:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12053 and previous config saved to /var/cache/conftool/dbconfig/20200727-125351-marostegui.json [production]
12:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1105:3311 with less weight', diff saved to https://phabricator.wikimedia.org/P12052 and previous config saved to /var/cache/conftool/dbconfig/20200727-125207-marostegui.json [production]
12:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12051 and previous config saved to /var/cache/conftool/dbconfig/20200727-125045-marostegui.json [production]
12:41 <marostegui> Compress innodb on db1106, this will generate lag on enwiki on labsdb hosts (wiki replicas) T254462 [production]
12:38 <moritzm> disable puppet on idp1001/2001 [production]
12:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1106 and pool db1105:3311 as vslow T254462', diff saved to https://phabricator.wikimedia.org/P12050 and previous config saved to /var/cache/conftool/dbconfig/20200727-123833-marostegui.json [production]
12:37 <akosiaris@cumin1001> conftool action : set/pooled=no; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet [production]
12:37 <akosiaris@cumin1001> conftool action : set/weight=0; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet [production]
12:37 <akosiaris@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet [production]
12:37 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet [production]
12:36 <akosiaris@cumin1001> conftool action : set/weight=0; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet [production]
12:31 <XioNoX> standardize cr2-codfw interfaces [production]
12:28 <volans@deploy1001> Finished deploy [debmonitor/deploy@25dbd20]: Release v0.2.7 (duration: 00m 27s) [production]
12:28 <volans@deploy1001> Started deploy [debmonitor/deploy@25dbd20]: Release v0.2.7 [production]
12:25 <jbond42> upload new cas package to buster-wikimedia [production]
12:25 <jbond42> upload new cas package [production]
12:23 <ema> A:cp rolling varnish-frontend restart to actually discard old VCL still pointing at varnishcheck/check T255015 T236754 [production]
12:21 <moritzm> installing ruby-json security updates [production]
12:16 <moritzm> installing batik security updates [production]
11:59 <marostegui> Deploy MCR schema change on db1149 [production]
11:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1149 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12049 and previous config saved to /var/cache/conftool/dbconfig/20200727-115818-marostegui.json [production]
11:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1138', diff saved to https://phabricator.wikimedia.org/P12048 and previous config saved to /var/cache/conftool/dbconfig/20200727-115739-marostegui.json [production]
11:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1138', diff saved to https://phabricator.wikimedia.org/P12047 and previous config saved to /var/cache/conftool/dbconfig/20200727-115258-marostegui.json [production]