1301-1350 of 10000 results (95ms)
2023-05-12 §
09:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2131', diff saved to https://phabricator.wikimedia.org/P48205 and previous config saved to /var/cache/conftool/dbconfig/20230512-093950-root.json [production]
09:18 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.8 refs T330214 [production]
09:13 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=eqiad,name=mw1494.eqiad.wmnet [production]
09:13 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=eqiad,name=mw146[7-9].eqiad.wmnet [production]
09:08 <hashar@deploy1002> Finished scap: Backport for [[gerrit:919178|Reset the cached skin in RequestContext::setUser() (T336504)]] (duration: 16m 27s) [production]
08:54 <hashar@deploy1002> hashar: Backport for [[gerrit:919178|Reset the cached skin in RequestContext::setUser() (T336504)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
08:52 <hashar@deploy1002> Started scap: Backport for [[gerrit:919178|Reset the cached skin in RequestContext::setUser() (T336504)]] [production]
08:03 <_joe_> restarting envoy on all jobrunners pooled in the jobrunner cluster T336554 [production]
08:00 <_joe_> do it also on mw1438 [production]
07:59 <_joe_> restaring envoyproxy on mw1439 to rebalance connections (see T336554) [production]
07:57 <taavi@deploy1002> Finished scap: Backport for [[gerrit:919269|Disable Graph (again) (T336556)]] (duration: 12m 29s) [production]
07:47 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13335 [production]
07:46 <taavi@deploy1002> taavi: Backport for [[gerrit:919269|Disable Graph (again) (T336556)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
07:45 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 13335 [production]
07:45 <volans@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling update on A:netbox [production]
07:44 <taavi@deploy1002> Started scap: Backport for [[gerrit:919269|Disable Graph (again) (T336556)]] [production]
07:43 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 20940 [production]
07:32 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 20940 [production]
07:28 <volans@cumin1001> START - Cookbook sre.netbox.update-extras rolling update on A:netbox [production]
07:27 <volans@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling update on A:netbox-canary [production]
07:27 <volans@cumin1001> START - Cookbook sre.netbox.update-extras rolling update on A:netbox-canary [production]
05:33 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=eqiad,name=mw1495.eqiad.wmnet [production]
05:32 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=eqiad,name=mw1466.eqiad.wmnet [production]
05:32 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=eqiad,name=mw1458.eqiad.wmnet [production]
05:31 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=eqiad,name=mw1461.eqiad.wmnet [production]
02:33 <ejegg> payments-wiki upgraded from d1c5fefc to c0da741f [production]
02:32 <ejegg> SmashPig upgraded from a9fa7a2c to 5460dbe2 [production]
01:08 <denisse@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts prometheus6001.drmrs.wmnet [production]
01:08 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:08 <denisse@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: prometheus6001.drmrs.wmnet decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
01:07 <denisse@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: prometheus6001.drmrs.wmnet decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
01:01 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
00:57 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts prometheus6001.drmrs.wmnet [production]
00:51 <denisse@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts prometheus5001.eqsin.wmnet [production]
00:51 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:51 <denisse@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: prometheus5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
00:50 <denisse@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: prometheus5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
00:48 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
00:44 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts prometheus5001.eqsin.wmnet [production]
00:32 <denisse> manually removing prometheus4001.ulsfo.wmnet from the Ganeti master after a failed step in the decommission cookbook - T335585 [production]
00:22 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on prometheus3001.esams.wmnet with reason: maintenance [production]
00:22 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on prometheus3001.esams.wmnet with reason: maintenance [production]
2023-05-11 §
23:39 <mutante> LDAP - added uid lorenjohnson to groups wmde nda T335858 [production]
23:39 <mutante> LDAP - added uid roti to groups wmde and nda T336435 [production]
23:24 <rzl@cumin2002> conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw14(3[789]|4[056]|57)\.eqiad\.wmnet [production]
23:22 <rzl@cumin2002> conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw14(5[89]|6[016789]|9[45])\.eqiad\.wmnet [production]
23:22 <rzl@cumin2002> conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw14(3[789]|4[056]57)\.eqiad\.wmnet [production]
23:07 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudswift1002'] [production]
22:46 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudswift1002'] [production]
22:41 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudswift1001'] [production]