6101-6150 of 10000 results (99ms)
2023-06-01 ยง
14:41 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mwlog1002.eqiad.wmnet with reason: host reimage [production]
14:40 <fabfur@cumin1001> START - Cookbook sre.cdn.run-puppet-restart-varnish rolling custom on A:cp-upload_drmrs and A:cp [production]
14:39 <ayounsi@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [production]
14:39 <ayounsi@cumin1001> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [production]
14:36 <fabfur@cumin1001> START - Cookbook sre.cdn.run-puppet-restart-varnish rolling custom on A:cp-text_drmrs and A:cp [production]
14:34 <aborrero@cumin2002> START - Cookbook sre.hosts.reimage for host cloudcontrol2004-dev.codfw.wmnet with OS bullseye [production]
14:29 <moritzm> installing imagemagick security updates on buster [production]
14:16 <herron@cumin1001> START - Cookbook sre.hosts.reimage for host mwlog1002.eqiad.wmnet with OS bullseye [production]
14:14 <fabfur> Disabled puppet on A:cp-drmrs for T323557 [production]
14:13 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@3c9cc85]: (no justification provided) (duration: 00m 11s) [production]
14:13 <mforns@deploy1002> Started deploy [airflow-dags/analytics@3c9cc85]: (no justification provided) [production]
14:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2158 (T336886)', diff saved to https://phabricator.wikimedia.org/P48700 and previous config saved to /var/cache/conftool/dbconfig/20230601-141317-ladsgroup.json [production]
14:11 <claime> Removing obsolete mediawiki-services-function-evaluator from registry - T337505 [production]
13:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P48699 and previous config saved to /var/cache/conftool/dbconfig/20230601-135811-ladsgroup.json [production]
13:52 <moritzm> installing sysstat security updates [production]
13:52 <jelto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
13:51 <jelto@deploy1002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
13:50 <jelto@deploy1002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
13:50 <jelto@deploy1002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
13:49 <jelto@deploy1002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
13:49 <jelto@deploy1002> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
13:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P48698 and previous config saved to /var/cache/conftool/dbconfig/20230601-134304-ladsgroup.json [production]
13:29 <moritzm> installing openssl security updates on bullseye [production]
13:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2158 (T336886)', diff saved to https://phabricator.wikimedia.org/P48697 and previous config saved to /var/cache/conftool/dbconfig/20230601-132758-ladsgroup.json [production]
13:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2158 (T336886)', diff saved to https://phabricator.wikimedia.org/P48695 and previous config saved to /var/cache/conftool/dbconfig/20230601-132319-ladsgroup.json [production]
13:23 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
13:23 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
13:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
13:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
13:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T336886)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20230601-132238-ladsgroup.json [production]
13:21 <claime> Removing obsolete mediawiki-services-function-orchestrator from registry - T337505 [production]
13:13 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:925766|beta: Stop setting unused $wgCampaignEventsUseNewTrackingToolsSchema (T336362)]], [[gerrit:923305|Set $wgCampaignEventsUseNewTrackingToolsSchema to true in prod (T336364)]] (duration: 11m 08s) [production]
13:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P48694 and previous config saved to /var/cache/conftool/dbconfig/20230601-130732-ladsgroup.json [production]
13:04 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20 days, 0:00:00 on wdqs2021.codfw.wmnet with reason: attempting WDQS stack on bullseye [production]
13:04 <urbanecm@deploy1002> urbanecm and daimona: Backport for [[gerrit:925766|beta: Stop setting unused $wgCampaignEventsUseNewTrackingToolsSchema (T336362)]], [[gerrit:923305|Set $wgCampaignEventsUseNewTrackingToolsSchema to true in prod (T336364)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
13:03 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 20 days, 0:00:00 on wdqs2021.codfw.wmnet with reason: attempting WDQS stack on bullseye [production]
13:02 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:925766|beta: Stop setting unused $wgCampaignEventsUseNewTrackingToolsSchema (T336362)]], [[gerrit:923305|Set $wgCampaignEventsUseNewTrackingToolsSchema to true in prod (T336364)]] [production]
12:58 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
12:57 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
12:52 <jelto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
12:52 <jelto@deploy1002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
12:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P48693 and previous config saved to /var/cache/conftool/dbconfig/20230601-125226-ladsgroup.json [production]
12:50 <jelto@deploy1002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
12:49 <jelto@deploy1002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
12:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T336886)', diff saved to https://phabricator.wikimedia.org/P48692 and previous config saved to /var/cache/conftool/dbconfig/20230601-123720-ladsgroup.json [production]
12:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2151 (T336886)', diff saved to https://phabricator.wikimedia.org/P48691 and previous config saved to /var/cache/conftool/dbconfig/20230601-123236-ladsgroup.json [production]
12:32 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]
12:32 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]
12:29 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance [production]
12:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance [production]