601-650 of 10000 results (85ms)
2023-10-12 §
08:35 <godog> add 200G to prometheus/ops in eqiad [production]
08:28 <hashar@deploy2002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.30 refs T347081 [production]
08:15 <aikochou@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
06:59 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Arturo Borrero Gonzalez out of all services on: 2156 hosts [production]
06:58 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Arturo Borrero Gonzalez out of all services on: 2156 hosts [production]
06:46 <elukey@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
00:09 <pt1979@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1063.eqiad.wmnet with OS bullseye [production]
2023-10-11 §
23:23 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS bullseye [production]
23:22 <pt1979@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
23:09 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
23:05 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host moss-be2003.codfw.wmnet with OS bullseye [production]
22:47 <eileen> civicrm upgraded from f2f1e23e to ceaeaa19 [production]
22:46 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye [production]
22:18 <ebernhardson@deploy2002> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
22:18 <ebernhardson@deploy2002> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
22:15 <ebernhardson@deploy2002> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
22:15 <ebernhardson@deploy2002> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
22:05 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS bookworm [production]
21:49 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage [production]
21:47 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage [production]
21:30 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for apifeatureusage2001.codfw.wmnet,apifeatureusage1001.eqiad.wmnet [production]
21:30 <ryankemper@cumin1001> START - Cookbook sre.hosts.remove-downtime for apifeatureusage2001.codfw.wmnet,apifeatureusage1001.eqiad.wmnet [production]
21:30 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bookworm [production]
21:26 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS bookworm [production]
21:20 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on apifeatureusage2001.codfw.wmnet with reason: reboot T348418 [production]
21:20 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on apifeatureusage2001.codfw.wmnet with reason: reboot T348418 [production]
21:11 <ryankemper> T348418 Rebooting `apifeatureusage1001.eqiad.wmnet` [production]
21:09 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage [production]
21:07 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage [production]
21:06 <taavi@deploy2002> Finished scap: Backport for [[gerrit:965250|Set WRITE_NEW for CA wikis on OATHAuth multiple devices (T242031)]] (duration: 10m 33s) [production]
21:01 <taavi@deploy2002> taavi: Continuing with sync [production]
20:57 <taavi@deploy2002> taavi: Backport for [[gerrit:965250|Set WRITE_NEW for CA wikis on OATHAuth multiple devices (T242031)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:55 <taavi@deploy2002> Started scap: Backport for [[gerrit:965250|Set WRITE_NEW for CA wikis on OATHAuth multiple devices (T242031)]] [production]
20:54 <cstone> payments-wiki upgraded from d6ad0376 to aa5cd24d [production]
20:54 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS bookworm [production]
20:45 <bking@deploy2002> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
20:45 <bking@deploy2002> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
20:44 <bking@deploy2002> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
20:43 <bking@deploy2002> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
20:40 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS bookworm [production]
20:24 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage [production]
20:22 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage [production]
20:19 <samtar@deploy2002> Finished scap: Backport for [[gerrit:965205|Remove override to allow mobile edit notices to display on all wikis (T316178)]] (duration: 08m 18s) [production]
20:14 <samtar@deploy2002> kemayo and samtar: Continuing with sync [production]
20:13 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye [production]
20:13 <samtar@deploy2002> kemayo and samtar: Backport for [[gerrit:965205|Remove override to allow mobile edit notices to display on all wikis (T316178)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:12 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-be2003.codfw.wmnet with OS bullseye [production]
20:11 <samtar@deploy2002> Started scap: Backport for [[gerrit:965205|Remove override to allow mobile edit notices to display on all wikis (T316178)]] [production]
20:11 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye [production]
20:09 <samtar@deploy2002> Finished scap: Backport for [[gerrit:963084|Enable Edit Check on initial partner wikis (T347908)]] (duration: 07m 32s) [production]