2851-2900 of 10000 results (82ms)
2024-02-21 §
03:41 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2211 to codfw - jhancock@cumin2002" [production]
03:39 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
03:37 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2210.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:36 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
03:36 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2210 to codfw - jhancock@cumin2002" [production]
03:35 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2210 to codfw - jhancock@cumin2002" [production]
03:33 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
03:31 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2209.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:30 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
03:30 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2209 to codfw - jhancock@cumin2002" [production]
03:29 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2209 to codfw - jhancock@cumin2002" [production]
03:29 <rzl@deploy2002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
03:28 <rzl@deploy2002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
03:27 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
03:26 <rzl@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
03:26 <rzl@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
03:25 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2208.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:21 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2206.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:15 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2207.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:03 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2208.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:01 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2207.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:01 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2208.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:01 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2207.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:00 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2208.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:00 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2207.mgmt.codfw.wmnet with reboot policy FORCED [production]
03:00 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2206.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:59 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
02:59 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2206 to codfw - jhancock@cumin2002" [production]
02:58 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2206 to codfw - jhancock@cumin2002" [production]
02:56 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
02:49 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2204.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:39 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2205.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:29 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2204.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:28 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2205.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:23 <rzl@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
02:22 <rzl@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
02:20 <rzl@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
02:20 <rzl@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
02:11 <rzl@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
02:10 <rzl@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
00:27 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
00:27 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
00:00 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
2024-02-20 §
23:59 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
23:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance [production]
23:38 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance [production]
23:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235 (T355609)', diff saved to https://phabricator.wikimedia.org/P57433 and previous config saved to /var/cache/conftool/dbconfig/20240220-233832-marostegui.json [production]
23:25 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
23:24 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
23:24 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]