1-50 of 10000 results (89ms)
2025-03-27 §
04:06 <cwhite> restart grafana-server on grafana1002 - appears hung [production]
2025-03-26 §
23:06 <toyofuku@deploy1003> Finished scap sync-world: Backport for [[gerrit:1131469|Restore simplified watchlist for logged in users (T388445)]] (duration: 12m 29s) [production]
22:59 <toyofuku@deploy1003> jdlrobson, toyofuku: Continuing with sync [production]
22:59 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic1123.eqiad.wmnet with OS bullseye [production]
22:58 <toyofuku@deploy1003> jdlrobson, toyofuku: Backport for [[gerrit:1131469|Restore simplified watchlist for logged in users (T388445)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:57 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp6009.drmrs.wmnet} and A:cp [production]
22:56 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp6008.drmrs.wmnet} and A:cp [production]
22:54 <toyofuku@deploy1003> Started scap sync-world: Backport for [[gerrit:1131469|Restore simplified watchlist for logged in users (T388445)]] [production]
22:51 <toyofuku@deploy1003> Finished scap sync-world: Backport for [[gerrit:1131365|Set wgMinervaDonateBanner to default base true (T388438)]] (duration: 15m 15s) [production]
22:50 <brett@cumin2002> START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp6008.drmrs.wmnet} and A:cp [production]
22:50 <brett@cumin2002> START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp6009.drmrs.wmnet} and A:cp [production]
22:44 <toyofuku@deploy1003> ksarabia, toyofuku: Continuing with sync [production]
22:42 <toyofuku@deploy1003> ksarabia, toyofuku: Backport for [[gerrit:1131365|Set wgMinervaDonateBanner to default base true (T388438)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:38 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2310.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
22:36 <toyofuku@deploy1003> Started scap sync-world: Backport for [[gerrit:1131365|Set wgMinervaDonateBanner to default base true (T388438)]] [production]
22:32 <toyofuku@deploy1003> Finished scap sync-world: Backport for [[gerrit:1130771|Web features should not be ambiguously configured (T388445)]] (duration: 23m 41s) [production]
22:32 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host elastic1124.eqiad.wmnet with OS bullseye [production]
22:31 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host wikikube-worker2310.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
22:31 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:29 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2310.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
22:26 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2301.codfw.wmnet with OS bookworm [production]
22:26 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2302.codfw.wmnet with OS bookworm [production]
22:25 <toyofuku@deploy1003> toyofuku, jdlrobson: Continuing with sync [production]
22:20 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:18 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:18 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host wikikube-worker2310.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
22:18 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host elastic1125.eqiad.wmnet with OS bullseye [production]
22:18 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1125.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:17 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2310 [production]
22:17 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:17 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2310 [production]
22:17 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:15 <toyofuku@deploy1003> toyofuku, jdlrobson: Backport for [[gerrit:1130771|Web features should not be ambiguously configured (T388445)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:15 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:13 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp6010.drmrs.wmnet} and A:cp [production]
22:13 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:12 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host elastic1125.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:12 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:12 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2310 to codfw - jhancock@cumin2002" [production]
22:12 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp6007.drmrs.wmnet} and A:cp [production]
22:12 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2310 to codfw - jhancock@cumin2002" [production]
22:10 <topranks> reset configuration on cr1-drmrs to enable external connections T389071 [production]
22:09 <toyofuku@deploy1003> Started scap sync-world: Backport for [[gerrit:1130771|Web features should not be ambiguously configured (T388445)]] [production]
22:08 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
22:06 <brett@cumin2002> START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp6007.drmrs.wmnet} and A:cp [production]
22:06 <brett@cumin2002> START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp6010.drmrs.wmnet} and A:cp [production]
22:04 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1125.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:01 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host elastic1124.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:01 <topranks> resetting PIC0 on cr1-drmrs to enable et-0/0/1 T389071 [production]
21:55 <topranks> disabling external Internet peers in BGP on cr1-drmrs T389071 [production]