2151-2200 of 10000 results (119ms)
2024-09-10 ยง
14:35 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=restbase2025.codfw.wmnet [production]
14:32 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 2%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68805 and previous config saved to /var/cache/conftool/dbconfig/20240910-143211-arnaudb.json [production]
14:31 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P68804 and previous config saved to /var/cache/conftool/dbconfig/20240910-143156-ladsgroup.json [production]
14:29 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:28 <sukhe> [end] rolling restart of {pdns-recursor,haproxy}.service on A:dnsbox [production]
14:25 <jayme> restoring leadership for partitions assigned to broker id 2002 on kafka-main-codfw - T363210 [production]
14:17 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 1%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68802 and previous config saved to /var/cache/conftool/dbconfig/20240910-141705-arnaudb.json [production]
14:16 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P68801 and previous config saved to /var/cache/conftool/dbconfig/20240910-141649-ladsgroup.json [production]
14:16 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet [production]
14:16 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles zhwiki | tee T363538-zhwiki-cleanupTitles [production]
14:15 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes zhwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-zhwiki-namespaceDupes [production]
14:15 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet [production]
14:15 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2013.codfw.wmnet [production]
14:15 <arnaudb@cumin1002> START - Cookbook sre.mysql.clone of db2137.codfw.wmnet onto db2237.codfw.wmnet [production]
14:15 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host parse2013.codfw.wmnet [production]
14:15 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2012.codfw.wmnet [production]
14:14 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles thwiki | tee T363538-thwiki-cleanupTitles [production]
14:14 <sukhe> sudo cumin "A:cp" 'disable-puppet "merging CRs 1065286"' [production]
14:14 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host parse2012.codfw.wmnet [production]
14:14 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2011.codfw.wmnet [production]
14:14 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes thwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-thwiki-namespaceDupes [production]
14:13 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles slwiki | tee T363538-slwiki-cleanupTitles [production]
14:13 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes slwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-slwiki-namespaceDupes [production]
14:11 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles simplewiki | tee T363538-simplewiki-cleanupTitles [production]
14:11 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes simplewiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-simplewiki-namespaceDupes [production]
14:11 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host parse2011.codfw.wmnet [production]
14:11 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2443.codfw.wmnet [production]
14:11 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-main2002.codfw.wmnet [production]
14:11 <jayme@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:10 <jayme@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002" [production]
14:10 <jayme@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002" [production]
14:10 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host mw2443.codfw.wmnet [production]
14:10 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2442.codfw.wmnet [production]
14:10 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles mswiki | tee T363538-mswiki-cleanupTitles [production]
14:10 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes mswiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-mswiki-namespaceDupes [production]
14:09 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host mw2442.codfw.wmnet [production]
14:09 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2440.codfw.wmnet [production]
14:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Cloning db2137 in db2237 for T373579', diff saved to https://phabricator.wikimedia.org/P68792 and previous config saved to /var/cache/conftool/dbconfig/20240910-140918-arnaudb.json [production]
14:09 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host mw2440.codfw.wmnet [production]
14:09 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2058.codfw.wmnet [production]
14:09 <arnaudb@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2227.codfw.wmnet onto db2127.codfw.wmnet [production]
14:08 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2058.codfw.wmnet [production]
14:08 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2041.codfw.wmnet [production]
14:08 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579 [production]
14:08 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579 [production]
14:08 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579 [production]
14:07 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2041.codfw.wmnet [production]
14:07 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2040.codfw.wmnet [production]
14:07 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579 [production]
14:07 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2040.codfw.wmnet [production]