2201-2250 of 10000 results (32ms)
2021-04-19 §
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 50%: Repool db1082', diff saved to https://phabricator.wikimedia.org/P15433 and previous config saved to /var/cache/conftool/dbconfig/20210419-074953-root.json [production]
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15432 and previous config saved to /var/cache/conftool/dbconfig/20210419-074950-root.json [production]
07:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1085 (re)pooling @ 25%: Repool db1085', diff saved to https://phabricator.wikimedia.org/P15431 and previous config saved to /var/cache/conftool/dbconfig/20210419-074510-root.json [production]
07:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1085 T272008', diff saved to https://phabricator.wikimedia.org/P15430 and previous config saved to /var/cache/conftool/dbconfig/20210419-074155-marostegui.json [production]
07:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 30%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15429 and previous config saved to /var/cache/conftool/dbconfig/20210419-073725-root.json [production]
07:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 25%: Repool db1082', diff saved to https://phabricator.wikimedia.org/P15428 and previous config saved to /var/cache/conftool/dbconfig/20210419-073449-root.json [production]
07:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 50%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15427 and previous config saved to /var/cache/conftool/dbconfig/20210419-073446-root.json [production]
07:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P15426 and previous config saved to /var/cache/conftool/dbconfig/20210419-073425-root.json [production]
07:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 20%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15425 and previous config saved to /var/cache/conftool/dbconfig/20210419-072221-root.json [production]
07:21 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
07:19 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
07:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15424 and previous config saved to /var/cache/conftool/dbconfig/20210419-071943-root.json [production]
07:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P15423 and previous config saved to /var/cache/conftool/dbconfig/20210419-071921-root.json [production]
07:17 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
07:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1082 T272008', diff saved to https://phabricator.wikimedia.org/P15422 and previous config saved to /var/cache/conftool/dbconfig/20210419-071701-marostegui.json [production]
07:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 15%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15421 and previous config saved to /var/cache/conftool/dbconfig/20210419-070718-root.json [production]
07:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 10%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15420 and previous config saved to /var/cache/conftool/dbconfig/20210419-070439-root.json [production]
07:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P15419 and previous config saved to /var/cache/conftool/dbconfig/20210419-070418-root.json [production]
07:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121 T272008', diff saved to https://phabricator.wikimedia.org/P15418 and previous config saved to /var/cache/conftool/dbconfig/20210419-070035-marostegui.json [production]
06:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 100%: Repool db1074', diff saved to https://phabricator.wikimedia.org/P15417 and previous config saved to /var/cache/conftool/dbconfig/20210419-065627-root.json [production]
06:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 10%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15416 and previous config saved to /var/cache/conftool/dbconfig/20210419-065213-root.json [production]
06:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P15415 and previous config saved to /var/cache/conftool/dbconfig/20210419-064914-root.json [production]
06:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112 T272008', diff saved to https://phabricator.wikimedia.org/P15414 and previous config saved to /var/cache/conftool/dbconfig/20210419-064600-marostegui.json [production]
06:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 75%: Repool db1074', diff saved to https://phabricator.wikimedia.org/P15413 and previous config saved to /var/cache/conftool/dbconfig/20210419-064123-root.json [production]
06:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 50%: Repool db1074', diff saved to https://phabricator.wikimedia.org/P15412 and previous config saved to /var/cache/conftool/dbconfig/20210419-062620-root.json [production]
06:17 <_joe_> upgrading envoy everywhere in eqiad T280317 [production]
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 25%: Repool db1074', diff saved to https://phabricator.wikimedia.org/P15411 and previous config saved to /var/cache/conftool/dbconfig/20210419-061116-root.json [production]
06:10 <_joe_> upgrading envoy everywhere in codfw T280317 [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1179 in s3 for the first time with minimal weight T275633', diff saved to https://phabricator.wikimedia.org/P15410 and previous config saved to /var/cache/conftool/dbconfig/20210419-060321-marostegui.json [production]
06:01 <_joe_> rolling out further envoy upgrades T280317 [production]
05:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 10%: Repool db1074', diff saved to https://phabricator.wikimedia.org/P15409 and previous config saved to /var/cache/conftool/dbconfig/20210419-055613-root.json [production]
05:53 <marostegui> Stop sanitarium master on s2 (lag will show up on clouddb* labsdb* hosts) T272008 [production]
05:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 T272008', diff saved to https://phabricator.wikimedia.org/P15408 and previous config saved to /var/cache/conftool/dbconfig/20210419-055240-marostegui.json [production]
05:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1106', diff saved to https://phabricator.wikimedia.org/P15407 and previous config saved to /var/cache/conftool/dbconfig/20210419-054831-marostegui.json [production]
05:42 <marostegui> Stop sanitarium master on s1 (lag will show up on clouddb* labsdb* hosts) T272008 [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1106 T272008', diff saved to https://phabricator.wikimedia.org/P15406 and previous config saved to /var/cache/conftool/dbconfig/20210419-054158-marostegui.json [production]
05:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1179 in s3 for the first time with minimal weight T275633', diff saved to https://phabricator.wikimedia.org/P15405 and previous config saved to /var/cache/conftool/dbconfig/20210419-053730-marostegui.json [production]
05:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1179 in s3 for the first time with minimal weight T275633', diff saved to https://phabricator.wikimedia.org/P15404 and previous config saved to /var/cache/conftool/dbconfig/20210419-053127-marostegui.json [production]
05:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1179 to dbctl T275633', diff saved to https://phabricator.wikimedia.org/P15403 and previous config saved to /var/cache/conftool/dbconfig/20210419-053050-marostegui.json [production]
05:05 <marostegui> Restart m2 database master T280251 [production]
2021-04-18 §
06:40 <Amir1> cleaning watchlist of User:Mr._Ibrahem in wikidatawiki (in main ns only) [production]
2021-04-17 §
16:16 <Amir1> cleaning SuccuBot's watchlist in wikidatawiki [production]
00:53 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet [production]
00:48 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1307.eqiad.wmnet [production]
00:23 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1402.eqiad.wmnet [production]
00:22 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1403.eqiad.wmnet [production]
00:18 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1403.eqiad.wmnet [production]
00:18 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1402.eqiad.wmnet [production]
00:14 <ryankemper> T267927 `sudo run-puppet-agent` and `sudo pool` on `wdqs2003` [production]
00:11 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1307.eqiad.wmnet with reason: REIMAGE [production]