2301-2350 of 10000 results (126ms)
2024-09-04 ยง
10:08 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:07 <cgoubert@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2325 to wikikube-worker2079 - cgoubert@cumin1002" [production]
10:06 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P68635 and previous config saved to /var/cache/conftool/dbconfig/20240904-100641-arnaudb.json [production]
10:04 <cgoubert@cumin1002> START - Cookbook sre.dns.netbox [production]
10:04 <cgoubert@cumin1002> START - Cookbook sre.hosts.rename from mw2325 to wikikube-worker2079 [production]
10:03 <aqu> Deployed refinery-source using jenkins [production]
10:02 <cgoubert@cumin1002> END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2325.codfw.wmnet [production]
10:01 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host mw2325.codfw.wmnet [production]
09:58 <aqu> Deployed refinery using scap, then deployed onto hdfs [production]
09:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T367781)', diff saved to https://phabricator.wikimedia.org/P68633 and previous config saved to /var/cache/conftool/dbconfig/20240904-095134-arnaudb.json [production]
09:50 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2080.codfw.wmnet [production]
09:50 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2080.codfw.wmnet [production]
09:48 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1181 (T367781)', diff saved to https://phabricator.wikimedia.org/P68632 and previous config saved to /var/cache/conftool/dbconfig/20240904-094824-arnaudb.json [production]
09:48 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
09:48 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
09:48 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1170 (T367781)', diff saved to https://phabricator.wikimedia.org/P68631 and previous config saved to /var/cache/conftool/dbconfig/20240904-094802-arnaudb.json [production]
09:41 <aqu@deploy1003> Finished deploy [analytics/refinery@07fd127] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@07fd1275] (duration: 04m 18s) [production]
09:37 <aqu@deploy1003> Started deploy [analytics/refinery@07fd127] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@07fd1275] [production]
09:34 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-text_drmrs for 9.2.5-1wm2 [production]
09:32 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P68630 and previous config saved to /var/cache/conftool/dbconfig/20240904-093255-arnaudb.json [production]
09:23 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:23 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for drmrs switch irb interface IPs - cmooney@cumin1002" [production]
09:23 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for drmrs switch irb interface IPs - cmooney@cumin1002" [production]
09:19 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 7:00:00 on db2198.codfw.wmnet with reason: Maintenance [production]
09:19 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 7:00:00 on db2198.codfw.wmnet with reason: Maintenance [production]
09:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2195 (T367856)', diff saved to https://phabricator.wikimedia.org/P68629 and previous config saved to /var/cache/conftool/dbconfig/20240904-091937-marostegui.json [production]
09:18 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
09:17 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P68628 and previous config saved to /var/cache/conftool/dbconfig/20240904-091748-arnaudb.json [production]
09:17 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:17 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:10 <isaranto@deploy1003> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . [production]
09:09 <isaranto@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . [production]
09:04 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P68627 and previous config saved to /var/cache/conftool/dbconfig/20240904-090429-marostegui.json [production]
09:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1170 (T367781)', diff saved to https://phabricator.wikimedia.org/P68626 and previous config saved to /var/cache/conftool/dbconfig/20240904-090240-arnaudb.json [production]
08:59 <dcausse@deploy1003> Finished scap sync-world: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]] (duration: 06m 48s) [production]
08:57 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:57 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
08:55 <dcausse@deploy1003> dcausse: Continuing with sync [production]
08:54 <dcausse@deploy1003> dcausse: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:52 <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]] [production]
08:49 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P68624 and previous config saved to /var/cache/conftool/dbconfig/20240904-084922-marostegui.json [production]
08:48 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2080.codfw.wmnet [production]
08:47 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2080.codfw.wmnet [production]
08:44 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2214 (T367781)', diff saved to https://phabricator.wikimedia.org/P68623 and previous config saved to /var/cache/conftool/dbconfig/20240904-084435-arnaudb.json [production]
08:36 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1170 (T367781)', diff saved to https://phabricator.wikimedia.org/P68622 and previous config saved to /var/cache/conftool/dbconfig/20240904-083622-arnaudb.json [production]
08:36 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
08:36 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
08:34 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2195 (T367856)', diff saved to https://phabricator.wikimedia.org/P68621 and previous config saved to /var/cache/conftool/dbconfig/20240904-083415-marostegui.json [production]
08:33 <vgutierrez@cumin1002> START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-text_drmrs for 9.2.5-1wm2 [production]
08:32 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_drmrs for 9.2.5-1wm2 [production]