601-650 of 10000 results (27ms)
2025-07-30 §
08:38 <gkyziridis@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
08:38 <jynus@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2184.codfw.wmnet with reason: replication will stop [production]
08:36 <jynus@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2183.codfw.wmnet with reason: upgrade mariadb [production]
08:36 <elukey@cumin1003> START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS bookworm [production]
08:32 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T399728)', diff saved to https://phabricator.wikimedia.org/P80278 and previous config saved to /var/cache/conftool/dbconfig/20250730-083252-fceratto.json [production]
08:32 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [tools]
08:28 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2010.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
08:28 <elukey@cumin1003> START - Cookbook sre.hosts.provision for host sretest2010.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
08:27 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db1190 (T399728)', diff saved to https://phabricator.wikimedia.org/P80276 and previous config saved to /var/cache/conftool/dbconfig/20250730-082758-fceratto.json [production]
08:27 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
08:27 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T399728)', diff saved to https://phabricator.wikimedia.org/P80275 and previous config saved to /var/cache/conftool/dbconfig/20250730-082735-fceratto.json [production]
08:22 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [tools]
08:22 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [toolsbeta]
08:13 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
08:12 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P80274 and previous config saved to /var/cache/conftool/dbconfig/20250730-081228-fceratto.json [production]
08:09 <elukey@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS bookworm [production]
08:05 <mlitn@deploy1003> Finished scap sync-world: Backport for [[gerrit:1171239|Add new MediaSearch config/coefficients (T385286)]] (duration: 09m 42s) [production]
08:03 <elukey@cumin1003> START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS bookworm [production]
08:03 <jelto@cumin1003> END (PASS) - Cookbook sre.gitlab.failover (exit_code=0) Failover of gitlab from gitlab2002.wikimedia.org to gitlab1004.wikimedia.org [production]
08:01 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2010.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
08:00 <mlitn@deploy1003> mlitn: Continuing with sync [production]
07:58 <mlitn@deploy1003> mlitn: Backport for [[gerrit:1171239|Add new MediaSearch config/coefficients (T385286)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
07:57 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P80273 and previous config saved to /var/cache/conftool/dbconfig/20250730-075720-fceratto.json [production]
07:56 <jelto@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:56 <jelto@cumin1003> START - Cookbook sre.dns.wipe-cache 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:55 <mlitn@deploy1003> Started scap sync-world: Backport for [[gerrit:1171239|Add new MediaSearch config/coefficients (T385286)]] [production]
07:53 <jelto@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:53 <jelto@cumin1003> START - Cookbook sre.dns.wipe-cache 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:51 <jelto@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:51 <jelto@cumin1003> START - Cookbook sre.dns.wipe-cache 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:50 <jelto@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:50 <jelto@cumin1003> START - Cookbook sre.dns.wipe-cache 'https://gitlab.wikimedia.org/ https://gitlab-replica-b.wikimedia.org/' on all recursors [production]
07:50 <jelto@dns1004> END - running authdns-update [production]
07:50 <elukey@cumin1003> START - Cookbook sre.hosts.provision for host sretest2010.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
07:49 <jelto@dns1004> START - running authdns-update [production]
07:42 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T399728)', diff saved to https://phabricator.wikimedia.org/P80272 and previous config saved to /var/cache/conftool/dbconfig/20250730-074213-fceratto.json [production]
07:35 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db1160 (T399728)', diff saved to https://phabricator.wikimedia.org/P80271 and previous config saved to /var/cache/conftool/dbconfig/20250730-073517-fceratto.json [production]
07:35 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
07:31 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
06:37 <jelto@cumin1003> START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab2002.wikimedia.org to gitlab1004.wikimedia.org [production]
01:11 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 10m 52s) [production]
01:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
2025-07-29 §
23:10 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2035.codfw.wmnet with OS bookworm [production]
22:48 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2035.codfw.wmnet with reason: host reimage [production]
22:42 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2035.codfw.wmnet with reason: host reimage [production]
22:24 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-reload reloading wikidata_main on wdqs1022.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20250714/ using stat1009.eqiad.wmnet) [production]
22:23 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host logstash2035 [production]
22:23 <cwhite@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host logstash2035 [production]
22:19 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2091.codfw.wmnet with OS bullseye [production]
22:15 <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1172397|Enable DiscussionTools thanks on existing "report incident" wikis (T366095)]] (duration: 12m 28s) [production]