151-200 of 10000 results (80ms)
2024-01-11 §
08:42 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1201.eqiad.wmnet with reason: host reimage [production]
02:50 <urandom> decommissioning cassandra, restbase2014-{a,b,c} — T352469 [production]
2024-01-10 §
22:29 <herron@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad [production]
22:05 <herron@cumin1002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad [production]
21:54 <herron@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw [production]
21:36 <Dreamy_Jazz> UTC late deploys done [production]
21:33 <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:989569|Add comment to clarify which rate limits apply to temporary users (T331576)]] (duration: 08m 05s) [production]
21:28 <herron@cumin1002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw [production]
21:27 <dreamyjazz@deploy2002> dreamyjazz and tchanders: Continuing with sync [production]
21:27 <dreamyjazz@deploy2002> dreamyjazz and tchanders: Backport for [[gerrit:989569|Add comment to clarify which rate limits apply to temporary users (T331576)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:25 <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:989569|Add comment to clarify which rate limits apply to temporary users (T331576)]] [production]
21:19 <taavi@deploy2002> Finished scap: Backport for [[gerrit:989262|Disable max width for index namespace (T352162)]] (duration: 14m 19s) [production]
21:12 <taavi@deploy2002> toyofuku and taavi: Continuing with sync [production]
21:08 <taavi@deploy2002> toyofuku and taavi: Backport for [[gerrit:989262|Disable max width for index namespace (T352162)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:05 <taavi@deploy2002> Started scap: Backport for [[gerrit:989262|Disable max width for index namespace (T352162)]] [production]
20:22 <sukhe> enable puppet on lvs2013: T352758 [production]
19:29 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:29 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove old records for mr1-codfw core links - cmooney@cumin1002" [production]
19:28 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove old records for mr1-codfw core links - cmooney@cumin1002" [production]
19:26 <jhuneidi@deploy2002> Synchronized php: group1 wikis to 1.42.0-wmf.13 refs T350089 (duration: 07m 58s) [production]
19:24 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
19:18 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.13 refs T350089 [production]
19:00 <topranks> disabling OSPF connection from mr1-codfw to codfw core routers T348164 [production]
18:40 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [production]
18:38 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus2006.codfw.wmnet with reason: memory upgrade [production]
18:37 <filippo@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on prometheus2006.codfw.wmnet with reason: memory upgrade [production]
18:37 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [production]
18:37 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [production]
18:36 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [production]
18:35 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for prometheus2005.codfw.wmnet [production]
18:35 <filippo@cumin1002> START - Cookbook sre.hosts.remove-downtime for prometheus2005.codfw.wmnet [production]
18:24 <sukhe> stop pybal on lvs2013: T352758 [production]
17:59 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on prometheus2005.codfw.wmnet with reason: memory upgrade [production]
17:58 <filippo@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on prometheus2005.codfw.wmnet with reason: memory upgrade [production]
17:54 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1377.eqiad.wmnet with OS bullseye [production]
17:47 <pfischer@deploy2002> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
17:46 <pfischer@deploy2002> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
17:44 <pfischer@deploy2002> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
17:44 <pfischer@deploy2002> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
17:40 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs2014.codfw.wmnet [production]
17:34 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1377.eqiad.wmnet with reason: host reimage [production]
17:31 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1377.eqiad.wmnet with reason: host reimage [production]
17:28 <sukhe@cumin2002> START - Cookbook sre.hosts.reboot-single for host lvs2014.codfw.wmnet [production]
17:27 <sukhe> enable puppet on lvs2014: T352758 [production]
17:16 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host mw1377.eqiad.wmnet with OS bullseye [production]
17:15 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1378.eqiad.wmnet with OS bullseye [production]
17:14 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:14 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for sandbox1-a-codfw irb.2201 gw - cmooney@cumin1002" [production]
17:14 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for sandbox1-a-codfw irb.2201 gw - cmooney@cumin1002" [production]
17:09 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]