201-250 of 10000 results (94ms)
2024-11-04 ยง
15:51 <vgutierrez@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs1013.eqiad.wmnet with reason: host reimage [production]
15:47 <pt1979@cumin2002> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
15:46 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
15:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T376905)', diff saved to https://phabricator.wikimedia.org/P70878 and previous config saved to /var/cache/conftool/dbconfig/20241104-154543-ladsgroup.json [production]
15:40 <vgutierrez@cumin1002> START - Cookbook sre.hosts.reimage for host lvs1013.eqiad.wmnet with OS bookworm [production]
15:36 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1178 (T376905)', diff saved to https://phabricator.wikimedia.org/P70877 and previous config saved to /var/cache/conftool/dbconfig/20241104-153613-ladsgroup.json [production]
15:36 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
15:35 <vgutierrez> upload liberica 0.1 to apt.wm.o (bookworm) - T377127 [production]
15:35 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
15:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T376905)', diff saved to https://phabricator.wikimedia.org/P70876 and previous config saved to /var/cache/conftool/dbconfig/20241104-153548-ladsgroup.json [production]
15:29 <sukhe> running authdns-update to move CN traffic to eqsin from ulsfo: T378744 [production]
15:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P70874 and previous config saved to /var/cache/conftool/dbconfig/20241104-152041-ladsgroup.json [production]
15:05 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P70873 and previous config saved to /var/cache/conftool/dbconfig/20241104-150534-ladsgroup.json [production]
14:50 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T376905)', diff saved to https://phabricator.wikimedia.org/P70872 and previous config saved to /var/cache/conftool/dbconfig/20241104-145027-ladsgroup.json [production]
14:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1177 (T376905)', diff saved to https://phabricator.wikimedia.org/P70871 and previous config saved to /var/cache/conftool/dbconfig/20241104-144101-ladsgroup.json [production]
14:40 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
14:40 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
14:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T376905)', diff saved to https://phabricator.wikimedia.org/P70870 and previous config saved to /var/cache/conftool/dbconfig/20241104-144037-ladsgroup.json [production]
14:38 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:36 <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084765|Exclude affiliates from P&E dashboard integration for CampaignEvents Extension (T377252)]] (duration: 23m 39s) [production]
14:28 <lucaswerkmeister-wmde@deploy2002> mhorsey, lucaswerkmeister-wmde: Continuing with sync [production]
14:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P70869 and previous config saved to /var/cache/conftool/dbconfig/20241104-142530-ladsgroup.json [production]
14:24 <moritzm> uploaded php7.4 7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2+icu67u3 to component/icu67 (backports of latest security fixes to our PHP 7.4 build) [production]
14:23 <lucaswerkmeister-wmde@deploy2002> mhorsey, lucaswerkmeister-wmde: Backport for [[gerrit:1084765|Exclude affiliates from P&E dashboard integration for CampaignEvents Extension (T377252)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:12 <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1084765|Exclude affiliates from P&E dashboard integration for CampaignEvents Extension (T377252)]] [production]
14:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P70868 and previous config saved to /var/cache/conftool/dbconfig/20241104-141023-ladsgroup.json [production]
13:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T376905)', diff saved to https://phabricator.wikimedia.org/P70867 and previous config saved to /var/cache/conftool/dbconfig/20241104-135516-ladsgroup.json [production]
13:51 <marostegui> Start schema change on redacteddb1001:s8 T367856 (this will make replication in s8 lag for around 2-3 days) [production]
13:50 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Schema change T367856 [production]
13:50 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Schema change T367856 [production]
13:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1172 (T376905)', diff saved to https://phabricator.wikimedia.org/P70866 and previous config saved to /var/cache/conftool/dbconfig/20241104-134605-ladsgroup.json [production]
13:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
13:45 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
13:40 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:40 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70865 and previous config saved to /var/cache/conftool/dbconfig/20241104-134021-ladsgroup.json [production]
13:25 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1039.eqiad.wmnet to cluster eqiad and group B [production]
13:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P70864 and previous config saved to /var/cache/conftool/dbconfig/20241104-132513-ladsgroup.json [production]
13:24 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1039.eqiad.wmnet to cluster eqiad and group B [production]
13:11 <Dreamy_Jazz> Started slow MediaModeration scan for commonswiki to be scanning as close to upload as possible - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
13:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P70862 and previous config saved to /var/cache/conftool/dbconfig/20241104-131006-ladsgroup.json [production]
13:06 <Dreamy_Jazz> Started MediaModeration scan on all wikis other than s4 (commonswiki + testcommonswiki) - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
12:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70861 and previous config saved to /var/cache/conftool/dbconfig/20241104-125459-ladsgroup.json [production]
12:49 <XioNoX> deploy "Add temporary LVS community for liberica test" - T378453 [production]
12:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70860 and previous config saved to /var/cache/conftool/dbconfig/20241104-124533-ladsgroup.json [production]
12:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
12:45 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
12:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
12:44 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
12:35 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]