6801-6850 of 10000 results (116ms)
2024-03-22 §
16:32 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudbackup2003 to codfw - jhancock@cumin2002" [production]
16:29 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
16:01 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:01 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old asw-b-codfw entries - cmooney@cumin1002" [production]
16:00 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old asw-b-codfw entries - cmooney@cumin1002" [production]
15:52 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:52 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:34 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:34 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:30 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:30 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:26 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:25 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:15 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:15 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:05 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
15:04 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Remove asw-b-codfw from synced hiera data - cmooney@cumin1002 - T360776" [production]
15:04 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.23 refs T354441 [production]
14:52 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Remove asw-b-codfw from synced hiera data - cmooney@cumin1002 - T360776" [production]
14:40 <eoghan@cumin1002> START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org [production]
14:37 <eoghan@cumin1002> END (FAIL) - Cookbook sre.gitlab.failover (exit_code=93) Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org [production]
14:37 <eoghan@cumin1002> START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org [production]
14:35 <eoghan@cumin1002> END (FAIL) - Cookbook sre.gitlab.failover (exit_code=93) Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org [production]
14:35 <eoghan@cumin1002> START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org [production]
14:35 <eoghan@cumin1002> END (ERROR) - Cookbook sre.gitlab.failover (exit_code=93) Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org [production]
14:20 <urandom> restarting Cassandra decommission of restbase1024-{b,c} — T360548 [production]
14:11 <topranks> disabling LAG from asw-b-codfw to ssw-aX-codfw T360776 [production]
14:07 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on asw-b-codfw with reason: prepping to decom switch stack [production]
14:07 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on asw-b-codfw with reason: prepping to decom switch stack [production]
13:31 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:31 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:29 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:29 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:28 <brouberol@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
13:28 <brouberol@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
13:23 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:23 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:17 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:17 <elukey> `elukey@cumin1002:~$ sudo cumin 'stat100[4,5,8,9]*' 'kill `pgrep -u kcv-wikimf`'` to unblock puppet on various stat nodes [production]
13:17 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:07 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:07 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:06 <brouberol@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
13:06 <brouberol@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
12:44 <eoghan@cumin1002> START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org [production]
12:35 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:35 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
12:17 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:17 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
12:03 <reedy@deploy1002> Synchronized php-1.42.0-wmf.23/includes/htmlform/fields/HTMLHiddenField.php: T360717 (duration: 13m 06s) [production]