6351-6400 of 10000 results (118ms)
2024-05-30 §
01:15 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2220.codfw.wmnet with reason: Maintenance [production]
01:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2208 (T364299)', diff saved to https://phabricator.wikimedia.org/P63617 and previous config saved to /var/cache/conftool/dbconfig/20240530-011454-marostegui.json [production]
01:03 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T352010)', diff saved to https://phabricator.wikimedia.org/P63616 and previous config saved to /var/cache/conftool/dbconfig/20240530-010302-ladsgroup.json [production]
00:59 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P63615 and previous config saved to /var/cache/conftool/dbconfig/20240530-005946-marostegui.json [production]
00:57 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:57 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-c3-codfw - pt1979@cumin2002" [production]
00:56 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-c3-codfw - pt1979@cumin2002" [production]
00:54 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
00:54 <pt1979@cumin2002> START - Cookbook sre.network.provision for device lsw1-c3-codfw.mgmt.codfw.wmnet [production]
00:53 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:53 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-c2-codfw - pt1979@cumin2002" [production]
00:52 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-c2-codfw - pt1979@cumin2002" [production]
00:50 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
00:50 <pt1979@cumin2002> START - Cookbook sre.network.provision for device lsw1-c2-codfw.mgmt.codfw.wmnet [production]
00:44 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P63614 and previous config saved to /var/cache/conftool/dbconfig/20240530-004438-marostegui.json [production]
00:41 <pt1979@cumin2002> END (PASS) - Cookbook sre.network.provision (exit_code=0) for device lsw1-c1-codfw.mgmt.codfw.wmnet [production]
00:37 <logmsgbot> @deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
00:37 <logmsgbot> @deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
00:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2208 (T364299)', diff saved to https://phabricator.wikimedia.org/P63613 and previous config saved to /var/cache/conftool/dbconfig/20240530-002930-marostegui.json [production]
00:09 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:09 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-c1-codfw - pt1979@cumin2002" [production]
00:08 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-c1-codfw - pt1979@cumin2002" [production]
00:06 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
00:06 <pt1979@cumin2002> START - Cookbook sre.network.provision for device lsw1-c1-codfw.mgmt.codfw.wmnet [production]
2024-05-29 §
23:43 <eileen> * civicrm upgraded from 0e3c277e to 44900b8c [production]
23:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2208 (T364299)', diff saved to https://phabricator.wikimedia.org/P63612 and previous config saved to /var/cache/conftool/dbconfig/20240529-232924-marostegui.json [production]
23:29 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2208.codfw.wmnet with reason: Maintenance [production]
23:29 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2208.codfw.wmnet with reason: Maintenance [production]
22:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
22:38 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
22:16 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
22:06 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage [production]
22:05 <jclark@cumin1002> END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage [production]
22:05 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
22:04 <jclark@cumin1002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
22:03 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage [production]
22:02 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage [production]
22:01 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS bullseye [production]
22:00 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
21:47 <cdanis@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
21:47 <cdanis@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
21:43 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2198.codfw.wmnet with reason: Maintenance [production]
21:43 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2198.codfw.wmnet with reason: Maintenance [production]
21:43 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2182 (T364299)', diff saved to https://phabricator.wikimedia.org/P63611 and previous config saved to /var/cache/conftool/dbconfig/20240529-214338-marostegui.json [production]
21:42 <cdanis@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
21:41 <cdanis@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
21:21 <jsn@deploy1002> Sync cancelled. [production]
21:21 <jsn@deploy1002> jsn: Backport for [[gerrit:1037132|Revert "feature(Popups): Conditional User Defaults Implementation"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:19 <jsn@deploy1002> Started scap: Backport for [[gerrit:1037132|Revert "feature(Popups): Conditional User Defaults Implementation"]] [production]
21:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P63609 and previous config saved to /var/cache/conftool/dbconfig/20240529-211321-marostegui.json [production]