101-150 of 10000 results (55ms)
2022-12-02 §
06:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P42204 and previous config saved to /var/cache/conftool/dbconfig/20221202-061259-marostegui.json [production]
00:09 <rzl@cumin1001> conftool action : set/pooled=no; selector: name=mw14(45|46).eqiad.wmnet,cluster=jobrunner [production]
00:09 <rzl@cumin1001> conftool action : set/pooled=no; selector: name=mw14(39|40).eqiad.wmnet,cluster=videoscaler [production]
00:07 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5004.wikimedia.org with OS buster [production]
2022-12-01 §
23:47 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[1347-1348].eqiad.wmnet [production]
23:47 <rzl@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:47 <rzl@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1347-1348].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:45 <rzl@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1347-1348].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:43 <rzl@cumin1001> START - Cookbook sre.dns.netbox [production]
23:37 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1347-1348].eqiad.wmnet [production]
23:35 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[1327-1346].eqiad.wmnet [production]
23:35 <rzl@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:35 <rzl@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1327-1346].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:34 <rzl@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1327-1346].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
23:31 <rzl@cumin1001> START - Cookbook sre.dns.netbox [production]
22:59 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1327-1346].eqiad.wmnet [production]
22:57 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:856008|GrowthExperiments: Remove unused config variable GEMentorDashboardUseVue]] (duration: 07m 28s) [production]
22:57 <rzl> rzl@puppetmaster1001:~$ sudo puppet node deactivate mw1320.eqiad.wmnet # T306162 [production]
22:56 <rzl> rzl@puppetmaster1001:~$ sudo puppet node deactivate mw1312.eqiad.wmnet # T306162 [production]
22:54 <rzl@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw[1307-1326].eqiad.wmnet [production]
22:54 <rzl@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:54 <rzl@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1307-1326].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
22:50 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:856008|GrowthExperiments: Remove unused config variable GEMentorDashboardUseVue]] [production]
22:49 <urbanecm@deploy1002> backport aborted: (duration: 00m 03s) [production]
22:42 <andrewbogott> upgradedwikitech-static-ord (aka wikitech-static) to Debian Buster, installed php7.4, upgraded MW to 1_39. Will delete the rackspace backup image in a few days. [production]
22:19 <rzl@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1307-1326].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - rzl@cumin1001" [production]
22:07 <rzl@cumin1001> START - Cookbook sre.dns.netbox [production]
22:02 <cwhite> restart swift-proxy on thanos::frontend eqiad [production]
22:01 <brennen> end of utc late backport & config window [production]
21:46 <brennen@deploy1002> Finished scap: Backport for [[gerrit:859568|GrowthExperiments: Enable user impact refresh script on pilot wikis (T322541)]] (duration: 07m 48s) [production]
21:40 <brennen@deploy1002> brennen and kharlan: Backport for [[gerrit:859568|GrowthExperiments: Enable user impact refresh script on pilot wikis (T322541)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
21:38 <brennen@deploy1002> Started scap: Backport for [[gerrit:859568|GrowthExperiments: Enable user impact refresh script on pilot wikis (T322541)]] [production]
21:34 <brennen@deploy1002> Finished scap: Backport for [[gerrit:863011|New configs for android schemas]] (duration: 09m 49s) [production]
21:26 <brennen@deploy1002> brennen and sharvaniharan: Backport for [[gerrit:863011|New configs for android schemas]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:25 <andrewbogott> saving an image of wikitech-static-ord (aka wikitech-static) before upgrading the host to Buster [production]
21:25 <brennen@deploy1002> Started scap: Backport for [[gerrit:863011|New configs for android schemas]] [production]
21:22 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1307-1326].eqiad.wmnet [production]
21:21 <brennen@deploy1002> Finished scap: Backport for [[gerrit:861853|Start writing to cul_actor on test wikis (T233004)]] (duration: 14m 56s) [production]
21:13 <rzl@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts mw[1307-1326].eqiad.wmnet [production]
21:10 <rzl@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw[1307-1326].eqiad.wmnet [production]
21:08 <brennen@deploy1002> brennen and zabe: Backport for [[gerrit:861853|Start writing to cul_actor on test wikis (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:06 <brennen@deploy1002> Started scap: Backport for [[gerrit:861853|Start writing to cul_actor on test wikis (T233004)]] [production]
20:47 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for gitlab1004.wikimedia.org [production]
20:47 <aokoth@cumin1001> START - Cookbook sre.hosts.remove-downtime for gitlab1004.wikimedia.org [production]
20:27 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1061.eqiad.wmnet with OS bullseye [production]
20:16 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns5004.wikimedia.org with reason: host reimage [production]
20:12 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1061.eqiad.wmnet with reason: host reimage [production]
20:12 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns5004.wikimedia.org with reason: host reimage [production]
20:09 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1061.eqiad.wmnet with reason: host reimage [production]
20:00 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1004.wikimedia.org with reason: upgrade gitlab1004 to new version https://phabricator.wikmiedia.org/T324195 [production]