501-550 of 10000 results (78ms)
2023-09-07 ยง
13:14 <ayounsi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: atlas2001.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1001" [production]
13:12 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
13:08 <ayounsi@cumin1001> START - Cookbook sre.hosts.decommission for hosts atlas2001.wikimedia.org [production]
12:35 <filippo@deploy1002> helmfile [staging] DONE helmfile.d/services/opentelemetry-collector: apply [production]
12:34 <filippo@deploy1002> helmfile [staging] START helmfile.d/services/opentelemetry-collector: apply [production]
12:23 <filippo@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
12:23 <filippo@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
12:04 <cgoubert@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
12:04 <claime> Starting eqiad jobrunner reboots [production]
12:02 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
12:02 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
11:59 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [production]
11:29 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2050.codfw.wmnet [production]
11:29 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1050.eqiad.wmnet [production]
11:23 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1050.eqiad.wmnet [production]
11:22 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2050.codfw.wmnet [production]
11:13 <cgoubert@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
11:11 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2065.codfw.wmnet [production]
11:10 <cgoubert@deploy1002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
11:09 <cgoubert@deploy1002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
11:05 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2065.codfw.wmnet [production]
11:04 <cgoubert@deploy1002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
11:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2065.codfw.wmnet [production]
10:56 <urbanecm> mwmaint1002: `/usr/local/bin/mw-cli-wrapper /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/growthexperiments.dblist extensions/GrowthExperiments/maintenance/refreshUserImpactData.php --registeredWithin=1year --editedWithin=2week --hasEditsAtLeast=3 --ignoreIfUpdatedWithin=1second --verbose --use-job-queue` (T344428, testing with r955319 deployed) [production]
10:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2065.codfw.wmnet [production]
10:54 <cgoubert@deploy1002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: sync [production]
10:51 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:955331|Pin pagelinks normalization stage to old in production (T345732)]] (duration: 09m 05s) [production]
10:46 <urbanecm@deploy1002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
10:45 <ladsgroup@deploy1002> ladsgroup: Continuing with sync [production]
10:44 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:955331|Pin pagelinks normalization stage to old in production (T345732)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
10:42 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:955331|Pin pagelinks normalization stage to old in production (T345732)]] [production]
10:35 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw[1441-1442,1451].eqiad.wmnet [production]
10:35 <cgoubert@cumin1001> START - Cookbook sre.hosts.remove-downtime for mw[1441-1442,1451].eqiad.wmnet [production]
10:35 <urbanecm@deploy1002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:35 <urbanecm@deploy1002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
10:33 <cgoubert@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1) [production]
10:32 <fnegri@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2001-dev.codfw.wmnet with reason: host reimage [production]
10:29 <fnegri@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2001-dev.codfw.wmnet with reason: host reimage [production]
10:24 <urbanecm@deploy1002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:24 <urbanecm@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
10:23 <urbanecm@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:23 <urbanecm@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
10:21 <urbanecm@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:10 <fnegri@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcontrol2001-dev.codfw.wmnet with OS bookworm [production]
10:08 <jbond@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserver1002.eqiad.wmnet with OS bookworm [production]
10:05 <hashar@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.41.0-wmf.25 refs T343727 [production]
10:03 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1134.eqiad.wmnet with OS bullseye [production]
09:58 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetserver1002.eqiad.wmnet with reason: host reimage [production]
09:55 <jbond@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on puppetserver1002.eqiad.wmnet with reason: host reimage [production]
09:54 <hashar@deploy1002> Finished scap: Backport for [[gerrit:955062|RevisionReviewForm: allow setting `null` tag (T345804)]] (duration: 07m 54s) [production]