2151-2200 of 10000 results (110ms)
2024-07-31 ยง
14:45 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2223.mgmt.codfw.wmnet with reboot policy GRACEFUL [production]
14:33 <marostegui@cumin1002> dbctl commit (dc=all): 'db2148 (re)pooling @ 5%: Repooling', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20240731-143340-root.json [production]
14:33 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host db2223.mgmt.codfw.wmnet with reboot policy GRACEFUL [production]
14:21 <sukhe> [done] upgrade cp4044 to ATS 9.2.5: T339134 [production]
14:21 <sukhe@cumin1002> END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on P{cp4044*} and (A:cp-eqiad or A:cp-text_eqiad or A:cp-upload_eqiad or A:cp-codfw or A:cp-text_codfw or A:cp-upload_codfw or A:cp-esams or A:cp-text_esams or A:cp-upload_esams or A:cp-ulsfo or A:cp-text_ulsfo or A:cp-upload_ulsfo or A:cp-eqsin or A:cp-text_eqsin or A:cp-upload_eqsin or A:cp- [production]
14:20 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2148', diff saved to https://phabricator.wikimedia.org/P67160 and previous config saved to /var/cache/conftool/dbconfig/20240731-141959-marostegui.json [production]
14:19 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
14:19 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
14:17 <sukhe@cumin1002> START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on P{cp4044*} and (A:cp-eqiad or A:cp-text_eqiad or A:cp-upload_eqiad or A:cp-codfw or A:cp-text_codfw or A:cp-upload_codfw or A:cp-esams or A:cp-text_esams or A:cp-upload_esams or A:cp-ulsfo or A:cp-text_ulsfo or A:cp-upload_ulsfo or A:cp-eqsin or A:cp-text_eqsin or A:cp-upload_eqsin or A:cp-drmrs or A:cp-text_ [production]
13:54 <Lucas_WMDE> UTC afternoon backport+config window done [production]
13:53 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Finished scap: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]] (duration: 11m 31s) [production]
13:49 <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, otto: Continuing with sync [production]
13:49 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s6 [production]
13:46 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
13:45 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
13:44 <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, otto: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:42 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]] [production]
13:40 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=dns6001.wikimedia.org,service=recdns [reason: [done] pdns-rec upgrade] [production]
13:39 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=dns6001.wikimedia.org,service=recdns [reason: pdns-rec upgrade] [production]
13:39 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Finished scap: Backport for [[gerrit:1058496|TranslatablePage: Store source page ids as string in WAN cache (T366455)]], [[gerrit:1058495|TranslatablePage: Store source page ids as string in WAN cache (T366455)]] (duration: 12m 34s) [production]
13:39 <sukhe> upgrade pdns-recursor to 4.8.8 from from 4.8.7 on dns6001 [production]
13:34 <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, abi: Continuing with sync [production]
13:28 <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, abi: Backport for [[gerrit:1058496|TranslatablePage: Store source page ids as string in WAN cache (T366455)]], [[gerrit:1058495|TranslatablePage: Store source page ids as string in WAN cache (T366455)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:27 <cdanis@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
13:26 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1058496|TranslatablePage: Store source page ids as string in WAN cache (T366455)]], [[gerrit:1058495|TranslatablePage: Store source page ids as string in WAN cache (T366455)]] [production]
13:25 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Finished scap: Backport for [[gerrit:1058196|Fix tracking parameter casing (T370045)]] (duration: 12m 30s) [production]
13:25 <ayounsi@cumin1002> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1002.eqiad.wmnet with reason: Release v0.7.0 - ayounsi@cumin1002 [production]
13:24 <cdanis@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
13:21 <cdanis@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
13:20 <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, joelyrookewmde: Continuing with sync [production]
13:19 <ayounsi@cumin1002> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1002.eqiad.wmnet with reason: Release v0.7.0 - ayounsi@cumin1002 [production]
13:18 <cdanis@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
13:16 <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, joelyrookewmde: Backport for [[gerrit:1058196|Fix tracking parameter casing (T370045)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:13 <fabfur> running `sudo cumin -b 1 -s300 A:cp-ulsfo 'depool-cdn && sleep 30 && enable-puppet "T370741" && run-puppet-agent && pool-cdn'` (T370741) [production]
13:12 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1058196|Fix tracking parameter casing (T370045)]] [production]
12:57 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet [reason: pooling after cookbook depooled as puppet was disabled] [production]
12:57 <elukey> update debmonitor-server and python3-debmonitor to bookworm-wikimedia - T368744 [production]
12:54 <sukhe@cumin1002> END (FAIL) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=1) Rolling upgrade/restart of Apache Traffic Server on P{cp4044*} and (A:cp-eqiad or A:cp-text_eqiad or A:cp-upload_eqiad or A:cp-codfw or A:cp-text_codfw or A:cp-upload_codfw or A:cp-esams or A:cp-text_esams or A:cp-upload_esams or A:cp-ulsfo or A:cp-text_ulsfo or A:cp-upload_ulsfo or A:cp-eqsin or A:cp-text_eqsin or A:cp-upload_eqsin or A:cp- [production]
12:53 <sukhe> upgrade cp4044 to ATS 9.2.5: T339134 [production]
12:53 <sukhe@cumin1002> START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on P{cp4044*} and (A:cp-eqiad or A:cp-text_eqiad or A:cp-upload_eqiad or A:cp-codfw or A:cp-text_codfw or A:cp-upload_codfw or A:cp-esams or A:cp-text_esams or A:cp-upload_esams or A:cp-ulsfo or A:cp-text_ulsfo or A:cp-upload_ulsfo or A:cp-eqsin or A:cp-text_eqsin or A:cp-upload_eqsin or A:cp-drmrs or A:cp-text_ [production]
12:50 <fabfur@cumin1002> conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [production]
12:50 <fabfur> repool cp4037, haproxy configuration modified to exclude benthos logging (T370741) [production]
12:46 <cdanis@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
12:44 <klausman@deploy1003> helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
12:39 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
12:39 <fabfur> temporary depooling cp4037 to test remove all Benthos resources (T370741) [production]
12:34 <ayounsi@cumin1002> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Release v4.0.8 to future netbox prod - ayounsi@cumin1002 - T336275 [production]
12:33 <cdanis@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
12:30 <fabfur> temporary disabling puppet on cp-ulsfo to test remove benthos from cp4037 (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1057823) (T370741) [production]
12:25 <ayounsi@cumin1002> START - Cookbook sre.deploy.python-code netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Release v4.0.8 to future netbox prod - ayounsi@cumin1002 - T336275 [production]