1-50 of 10000 results (101ms)
2026-05-25 ยง
15:03 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2228 (T426633)', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json [production]
14:53 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json [production]
14:42 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json [production]
14:33 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet [production]
14:32 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2228 (T426633)', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json [production]
14:32 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet [production]
14:32 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet [production]
14:31 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet [production]
14:25 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db2228 (T426633)', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json [production]
14:25 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance [production]
14:25 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2223 (T426633)', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json [production]
14:15 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json [production]
14:12 <jiji@cumin1003> START - Cookbook sre.dns.netbox [production]
14:06 <sukhe> curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: T424049 [production]
14:05 <sukhe> sukhe@lvs2013:~$ sudo systemctl restart pybal.service: T424049 [production]
14:05 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json [production]
14:03 <sukhe> sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) T424049"' [production]
14:02 <sukhe> sukhe@lvs2014:~$ sudo systemctl restart pybal.service": T424049 [production]
14:02 <sukhe> sukhe@lvs2014:~$ sudo systemctl restart pybal.service [production]
14:00 <sukhe> sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) T424049"' [production]
13:59 <jiji@cumin1003> START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet [production]
13:57 <sukhe> sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) T424049": NOOP change, since service is codfw only [production]
13:54 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2223 (T426633)', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json [production]
13:52 <Msz2001> Everything deployed, UTC afternoon config+backport window done [production]
13:52 <mszwarc@deploy1003> Finished scap sync-world: Backport for [[gerrit:1293119|Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) [production]
13:51 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet [production]
13:51 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet [production]
13:50 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet [production]
13:50 <sukhe@cumin1003> cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet [production]
13:49 <kart_> Updated Recommendation API to 2026-05-21-044522-production [production]
13:48 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db2223 (T426633)', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json [production]
13:48 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance [production]
13:47 <mszwarc@deploy1003> vadymts1, mszwarc: Continuing with deployment [production]
13:47 <kartik@deploy1003> helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
13:47 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2211 (T426633)', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json [production]
13:45 <mszwarc@deploy1003> vadymts1, mszwarc: Backport for [[gerrit:1293119|Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:45 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot [production]
13:43 <mszwarc@deploy1003> Started scap sync-world: Backport for [[gerrit:1293119|Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] [production]
13:40 <sukhe@cumin1003> START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad [production]
13:39 <sukhe@cumin1003> START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad [production]
13:38 <sbisson@deploy1003> Finished scap sync-world: Backport for [[gerrit:1290813|Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) [production]
13:38 <sukhe@cumin1003> START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin [production]
13:38 <sukhe@cumin1003> START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin [production]
13:37 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json [production]
13:34 <sbisson@deploy1003> sbisson: Continuing with deployment [production]
13:33 <kartik@deploy1003> helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
13:32 <jiji@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet [production]
13:32 <jiji@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:32 <jiji@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" [production]
13:31 <sbisson@deploy1003> sbisson: Backport for [[gerrit:1290813|Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]