7201-7250 of 10000 results (83ms)
2022-12-02 §
07:49 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
07:43 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:43 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
07:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: Maint done', diff saved to https://phabricator.wikimedia.org/P42209 and previous config saved to /var/cache/conftool/dbconfig/20221202-074300-ladsgroup.json [production]
07:41 <moritzm> draining ganeti5001 for eventual decom T322048 [production]
07:41 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:41 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
07:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: Maint done', diff saved to https://phabricator.wikimedia.org/P42208 and previous config saved to /var/cache/conftool/dbconfig/20221202-072755-ladsgroup.json [production]
07:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: Maint done', diff saved to https://phabricator.wikimedia.org/P42207 and previous config saved to /var/cache/conftool/dbconfig/20221202-071250-ladsgroup.json [production]
06:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1163 (re)pooling @ 10%: Maint done', diff saved to https://phabricator.wikimedia.org/P42206 and previous config saved to /var/cache/conftool/dbconfig/20221202-065745-ladsgroup.json [production]
06:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P42204 and previous config saved to /var/cache/conftool/dbconfig/20221202-061259-marostegui.json [production]
00:09 <rzl@cumin1001> conftool action : set/pooled=no; selector: name=mw14(45|46).eqiad.wmnet,cluster=jobrunner [production]
00:09 <rzl@cumin1001> conftool action : set/pooled=no; selector: name=mw14(39|40).eqiad.wmnet,cluster=videoscaler [production]
00:07 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5004.wikimedia.org with OS buster [production]
2022-12-01 §
23:47 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[1347-1348].eqiad.wmnet [production]
23:47 <rzl@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:47 <rzl@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1347-1348].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:45 <rzl@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1347-1348].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:43 <rzl@cumin1001> START - Cookbook sre.dns.netbox [production]
23:37 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1347-1348].eqiad.wmnet [production]
23:35 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[1327-1346].eqiad.wmnet [production]
23:35 <rzl@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:35 <rzl@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1327-1346].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:34 <rzl@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1327-1346].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:31 <rzl@cumin1001> START - Cookbook sre.dns.netbox [production]
22:59 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1327-1346].eqiad.wmnet [production]
22:57 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:856008|GrowthExperiments: Remove unused config variable GEMentorDashboardUseVue]] (duration: 07m 28s) [production]
22:57 <rzl> rzl@puppetmaster1001:~$ sudo puppet node deactivate mw1320.eqiad.wmnet # T306162 [production]
22:56 <rzl> rzl@puppetmaster1001:~$ sudo puppet node deactivate mw1312.eqiad.wmnet # T306162 [production]
22:54 <rzl@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw[1307-1326].eqiad.wmnet [production]
22:54 <rzl@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:54 <rzl@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1307-1326].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
22:50 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:856008|GrowthExperiments: Remove unused config variable GEMentorDashboardUseVue]] [production]
22:49 <urbanecm@deploy1002> backport aborted: (duration: 00m 03s) [production]
22:42 <andrewbogott> upgradedwikitech-static-ord (aka wikitech-static) to Debian Buster, installed php7.4, upgraded MW to 1_39. Will delete the rackspace backup image in a few days. [production]
22:19 <rzl@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1307-1326].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
22:07 <rzl@cumin1001> START - Cookbook sre.dns.netbox [production]
22:02 <cwhite> restart swift-proxy on thanos::frontend eqiad [production]
22:01 <brennen> end of utc late backport & config window [production]
21:46 <brennen@deploy1002> Finished scap: Backport for [[gerrit:859568|GrowthExperiments: Enable user impact refresh script on pilot wikis (T322541)]] (duration: 07m 48s) [production]
21:40 <brennen@deploy1002> brennen and kharlan: Backport for [[gerrit:859568|GrowthExperiments: Enable user impact refresh script on pilot wikis (T322541)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
21:38 <brennen@deploy1002> Started scap: Backport for [[gerrit:859568|GrowthExperiments: Enable user impact refresh script on pilot wikis (T322541)]] [production]
21:34 <brennen@deploy1002> Finished scap: Backport for [[gerrit:863011|New configs for android schemas]] (duration: 09m 49s) [production]
21:26 <brennen@deploy1002> brennen and sharvaniharan: Backport for [[gerrit:863011|New configs for android schemas]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:25 <andrewbogott> saving an image of wikitech-static-ord (aka wikitech-static) before upgrading the host to Buster [production]
21:25 <brennen@deploy1002> Started scap: Backport for [[gerrit:863011|New configs for android schemas]] [production]
21:22 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1307-1326].eqiad.wmnet [production]
21:21 <brennen@deploy1002> Finished scap: Backport for [[gerrit:861853|Start writing to cul_actor on test wikis (T233004)]] (duration: 14m 56s) [production]
21:13 <rzl@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts mw[1307-1326].eqiad.wmnet [production]
21:10 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1307-1326].eqiad.wmnet [production]