1601-1650 of 10000 results (79ms)
2022-07-31 §
22:54 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:53 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:53 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
22:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:19 <vgutierrez@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=cp5001.eqsin.wmnet [production]
18:14 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp5001.eqsin.wmnet with reason: depooled: faulty DIMM: T314256 [production]
18:13 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on cp5001.eqsin.wmnet with reason: depooled: faulty DIMM: T314256 [production]
18:12 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp5001.eqsin.wmnet,service=ats-tls [production]
18:12 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp5001.eqsin.wmnet,service=varnish-fe [production]
18:12 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp5001.eqsin.wmnet,service=ats-be [production]
2022-07-30 §
01:44 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reimage (bullseye upgrade) - bking@cumin1001 - T289135 [production]
01:44 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2028.codfw.wmnet with OS bullseye [production]
00:55 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2028.codfw.wmnet with OS bullseye [production]
2022-07-29 §
22:43 <Krinkle> krinkle@mwmaint1002$ mwscript findBadBlobs.php nlwiktionary; mark 2371 blobs from May 2004 as "Invalid gzip, T265989" [production]
22:37 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2041.codfw.wmnet with OS bullseye [production]
22:20 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2041.codfw.wmnet with reason: host reimage [production]
22:17 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2041.codfw.wmnet with reason: host reimage [production]
22:09 <Krinkle> findBadBlobs.php nlwiktionary --revisions 22 --mark 'Invalid gzip, T265989' [production]
22:01 <mutante> phab1001 - rsync -avp --bwlimit=1000 /srv/repos/ rsync://phab1004.eqiad.wmnet/phabricator-srv-repos (running slowly inside a screen session as root) (T313360, T280597) [production]
21:57 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2041.codfw.wmnet with OS bullseye [production]
21:06 <mutante> phab1004 - mkdir /srv/repos ; mkdir /srv/dumps [production]
20:46 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host elastic2029.codfw.wmnet with OS bullseye [production]
20:29 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2029.codfw.wmnet with reason: host reimage [production]
20:26 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2029.codfw.wmnet with reason: host reimage [production]
20:18 <mutante> authdns-update - adding gerrit-replica-new.wikimedia.org [production]
20:13 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2029.codfw.wmnet with OS bullseye [production]
18:28 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2057.codfw.wmnet with OS bullseye [production]
18:06 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2057.codfw.wmnet with reason: host reimage [production]
18:02 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2057.codfw.wmnet with reason: host reimage [production]
17:47 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2057.codfw.wmnet with OS bullseye [production]
17:41 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@85585b0]: (no justification provided) (duration: 00m 09s) [production]
17:41 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@85585b0]: (no justification provided) [production]
17:10 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2042.codfw.wmnet with OS bullseye [production]
16:53 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2042.codfw.wmnet with reason: host reimage [production]
16:50 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2042.codfw.wmnet with reason: host reimage [production]
16:30 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2042.codfw.wmnet with OS bullseye [production]
16:21 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host elastic2058.codfw.wmnet with OS bullseye [production]
15:58 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2058.codfw.wmnet with reason: host reimage [production]
15:55 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2058.codfw.wmnet with reason: host reimage [production]
15:40 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2058.codfw.wmnet with OS bullseye [production]
15:37 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2030.codfw.wmnet with OS bullseye [production]
15:19 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2030.codfw.wmnet with reason: host reimage [production]
15:17 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2030.codfw.wmnet with reason: host reimage [production]
15:03 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2030.codfw.wmnet with OS bullseye [production]
15:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1174', diff saved to https://phabricator.wikimedia.org/P32112 and previous config saved to /var/cache/conftool/dbconfig/20220729-150256-root.json [production]
15:00 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2043.codfw.wmnet with OS bullseye [production]
14:59 <marostegui> dbmaint s7@eqiad T314140 [production]
14:39 <marostegui> dbmaint s3@eqiad T314140 [production]
14:37 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2043.codfw.wmnet with reason: host reimage [production]
14:34 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2043.codfw.wmnet with reason: host reimage [production]