501-550 of 10000 results (169ms)
2024-11-04 ยง
16:06 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2160.codfw.wmnet with reason: cloning db2135@db2235 [production]
16:05 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 3:00:00 on db2160.codfw.wmnet with reason: cloning db2135@db2235 [production]
16:05 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
16:05 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm [production]
16:02 <arnaudb@cumin1002> START - Cookbook sre.mysql.clone of db2135.codfw.wmnet onto db2235.codfw.wmnet [production]
16:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:00 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P70879 and previous config saved to /var/cache/conftool/dbconfig/20241104-160050-ladsgroup.json [production]
16:00 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db[2135,2235].codfw.wmnet with reason: cloning db2135@db2235 [production]
16:00 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 3:00:00 on db[2135,2235].codfw.wmnet with reason: cloning db2135@db2235 [production]
15:58 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
15:54 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1013.eqiad.wmnet with reason: host reimage [production]
15:51 <vgutierrez@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs1013.eqiad.wmnet with reason: host reimage [production]
15:47 <pt1979@cumin2002> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
15:46 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
15:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T376905)', diff saved to https://phabricator.wikimedia.org/P70878 and previous config saved to /var/cache/conftool/dbconfig/20241104-154543-ladsgroup.json [production]
15:40 <vgutierrez@cumin1002> START - Cookbook sre.hosts.reimage for host lvs1013.eqiad.wmnet with OS bookworm [production]
15:36 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1178 (T376905)', diff saved to https://phabricator.wikimedia.org/P70877 and previous config saved to /var/cache/conftool/dbconfig/20241104-153613-ladsgroup.json [production]
15:36 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
15:35 <vgutierrez> upload liberica 0.1 to apt.wm.o (bookworm) - T377127 [production]
15:35 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
15:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T376905)', diff saved to https://phabricator.wikimedia.org/P70876 and previous config saved to /var/cache/conftool/dbconfig/20241104-153548-ladsgroup.json [production]
15:29 <sukhe> running authdns-update to move CN traffic to eqsin from ulsfo: T378744 [production]
15:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P70874 and previous config saved to /var/cache/conftool/dbconfig/20241104-152041-ladsgroup.json [production]
15:05 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P70873 and previous config saved to /var/cache/conftool/dbconfig/20241104-150534-ladsgroup.json [production]
14:50 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T376905)', diff saved to https://phabricator.wikimedia.org/P70872 and previous config saved to /var/cache/conftool/dbconfig/20241104-145027-ladsgroup.json [production]
14:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1177 (T376905)', diff saved to https://phabricator.wikimedia.org/P70871 and previous config saved to /var/cache/conftool/dbconfig/20241104-144101-ladsgroup.json [production]
14:40 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
14:40 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
14:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T376905)', diff saved to https://phabricator.wikimedia.org/P70870 and previous config saved to /var/cache/conftool/dbconfig/20241104-144037-ladsgroup.json [production]
14:38 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:36 <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084765|Exclude affiliates from P&E dashboard integration for CampaignEvents Extension (T377252)]] (duration: 23m 39s) [production]
14:28 <lucaswerkmeister-wmde@deploy2002> mhorsey, lucaswerkmeister-wmde: Continuing with sync [production]
14:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P70869 and previous config saved to /var/cache/conftool/dbconfig/20241104-142530-ladsgroup.json [production]
14:24 <moritzm> uploaded php7.4 7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2+icu67u3 to component/icu67 (backports of latest security fixes to our PHP 7.4 build) [production]
14:23 <lucaswerkmeister-wmde@deploy2002> mhorsey, lucaswerkmeister-wmde: Backport for [[gerrit:1084765|Exclude affiliates from P&E dashboard integration for CampaignEvents Extension (T377252)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:12 <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1084765|Exclude affiliates from P&E dashboard integration for CampaignEvents Extension (T377252)]] [production]
14:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P70868 and previous config saved to /var/cache/conftool/dbconfig/20241104-141023-ladsgroup.json [production]
13:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T376905)', diff saved to https://phabricator.wikimedia.org/P70867 and previous config saved to /var/cache/conftool/dbconfig/20241104-135516-ladsgroup.json [production]
13:51 <marostegui> Start schema change on redacteddb1001:s8 T367856 (this will make replication in s8 lag for around 2-3 days) [production]
13:50 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Schema change T367856 [production]
13:50 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Schema change T367856 [production]
13:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1172 (T376905)', diff saved to https://phabricator.wikimedia.org/P70866 and previous config saved to /var/cache/conftool/dbconfig/20241104-134605-ladsgroup.json [production]
13:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
13:45 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
13:40 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:40 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70865 and previous config saved to /var/cache/conftool/dbconfig/20241104-134021-ladsgroup.json [production]
13:25 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1039.eqiad.wmnet to cluster eqiad and group B [production]
13:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P70864 and previous config saved to /var/cache/conftool/dbconfig/20241104-132513-ladsgroup.json [production]
13:24 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1039.eqiad.wmnet to cluster eqiad and group B [production]