1-50 of 10000 results (37ms)
2022-10-04 ยง
13:49 <jbond@cumin1001> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "sync data - jbond@cumin1001" [production]
13:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db2181 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35347 and previous config saved to /var/cache/conftool/dbconfig/20221004-134947-root.json [production]
13:49 <hnowlan@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=sessionstore,name=codfw [production]
13:48 <sukhe> disable Puppet on A:cp and A:eqsin for T309651 [production]
13:47 <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync data - jbond@cumin1001" [production]
13:42 <awight> EU backport window finished. [production]
13:40 <filippo@cumin1001> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "sync-mgmt - filippo@cumin1001" [production]
13:38 <filippo@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001" [production]
13:38 <jmm@cumin2002> END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad [production]
13:38 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:37 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:37 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:36 <awight@deploy1002> Finished scap: Backport for [[gerrit:836804|Wire new event stream for maps interactions (T315972 T318678)]] (duration: 06m 49s) [production]
13:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:35 <jmm@cumin2002> START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad [production]
13:35 <filippo@cumin1001> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "filippo test - filippo@cumin1001" [production]
13:34 <filippo@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "filippo test - filippo@cumin1001" [production]
13:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db2181 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35346 and previous config saved to /var/cache/conftool/dbconfig/20221004-133442-root.json [production]
13:32 <ayounsi@cumin1001> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: update to wmf-netbox - try 2 - CR826559 - ayounsi@cumin1001 [production]
13:31 <jbond> re-enable puppet post deploy a puppetmaster change 838144 [production]
13:30 <ayounsi@cumin1001> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: update to wmf-netbox - try 2 - CR826559 - ayounsi@cumin1001 [production]
13:30 <ayounsi@cumin1001> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: update to wmf-netbx CR826559 - ayounsi@cumin1001 [production]
13:30 <awight@deploy1002> awight and awight: Backport for [[gerrit:836804|Wire new event stream for maps interactions (T315972 T318678)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
13:29 <awight@deploy1002> Started scap: Backport for [[gerrit:836804|Wire new event stream for maps interactions (T315972 T318678)]] [production]
13:28 <ayounsi@cumin1001> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: update to wmf-netbx CR826559 - ayounsi@cumin1001 [production]
13:27 <awight@deploy1002> Finished scap: Backport for [[gerrit:837757|ukwiki: Create flood group (T319243)]] (duration: 05m 16s) [production]
13:24 <jbond> disable puppet to deploy a puppetmaster change 838144 [production]
13:22 <awight@deploy1002> awight and stang: Backport for [[gerrit:837757|ukwiki: Create flood group (T319243)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
13:21 <awight@deploy1002> Started scap: Backport for [[gerrit:837757|ukwiki: Create flood group (T319243)]] [production]
13:21 <awight@deploy1002> Finished scap: Backport for [[gerrit:837756|throttle: Add throttle rule for 2022-10-13 (T319244)]] (duration: 12m 48s) [production]
13:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db2181 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35345 and previous config saved to /var/cache/conftool/dbconfig/20221004-131937-root.json [production]
13:16 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:14 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
13:13 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
13:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:11 <awight@deploy1002> awight and stang: Backport for [[gerrit:837756|throttle: Add throttle rule for 2022-10-13 (T319244)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
13:08 <awight@deploy1002> Started scap: Backport for [[gerrit:837756|throttle: Add throttle rule for 2022-10-13 (T319244)]] [production]
13:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db2181 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35343 and previous config saved to /var/cache/conftool/dbconfig/20221004-130432-root.json [production]
12:58 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage [production]
12:56 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage [production]
12:53 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage [production]
12:53 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage [production]
12:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db2181 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35342 and previous config saved to /var/cache/conftool/dbconfig/20221004-124927-root.json [production]
12:37 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye [production]
12:37 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS bullseye [production]
12:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db2181 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35341 and previous config saved to /var/cache/conftool/dbconfig/20221004-123422-root.json [production]
12:31 <cgoubert@deploy1002> Finished deploy [docker-pkg/deploy@24fbee1]: Release 3.0.3 # T310458 (duration: 00m 58s) [production]
12:30 <cgoubert@deploy1002> Started deploy [docker-pkg/deploy@24fbee1]: Release 3.0.3 # T310458 [production]