2551-2600 of 10000 results (56ms)
2022-05-06 §
07:13 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host puppetboard2002.codfw.wmnet [production]
07:11 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]
07:06 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet [production]
01:51 <dzahn@cumin2002> conftool action : set/pooled=no; selector: dc=eqiad,name=mw1415.eqiad.wmnet [production]
01:50 <dzahn@cumin2002> conftool action : set/pooled=no; selector: dc=codfw,name=mw1415.eqiad.wmnet [production]
00:46 <rook@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudvirt1016.eqiad.wmnet [production]
00:46 <rook@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1016.eqiad.wmnet [production]
2022-05-05 §
22:06 <razzi@cumin1001> END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka main-eqiad cluster: Reboot kafka nodes [production]
22:01 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:00 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:00 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:58 <hoo@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:734722|Add missing termbox codes from Wikibase (T277836)]] (duration: 00m 48s) [production]
21:56 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:35 <brennen@deploy1002> Synchronized php-1.39.0-wmf.10/includes/user: Backport: [[gerrit:789332|Suppress "named" group when TempUser system is disabled (T307675)]] (duration: 00m 48s) [production]
21:33 <brennen@deploy1002> scap failed: average error rate on 7/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details) [production]
21:26 <brennen@deploy1002> Finished scap: Resuming previously interrupted sync-world (duration: 03m 47s) [production]
21:25 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel [production]
21:24 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel [production]
21:22 <brennen@deploy1002> Started scap: Resuming previously interrupted sync-world [production]
21:21 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx1001.wikimedia.org with reason: new kernel [production]
21:21 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on mx1001.wikimedia.org with reason: new kernel [production]
21:21 <jhathaway> reboot mx1001 [production]
21:18 <dduvall@deploy1002> helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply [production]
21:18 <dduvall@deploy1002> helmfile [eqiad] START helmfile.d/services/blubberoid: apply [production]
21:18 <dduvall@deploy1002> helmfile [codfw] DONE helmfile.d/services/blubberoid: apply [production]
21:17 <dduvall@deploy1002> helmfile [codfw] START helmfile.d/services/blubberoid: apply [production]
21:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:12 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:11 <dduvall@deploy1002> helmfile [staging] DONE helmfile.d/services/blubberoid: apply [production]
21:11 <dduvall@deploy1002> helmfile [staging] START helmfile.d/services/blubberoid: apply [production]
21:08 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx2001.wikimedia.org with reason: new kernel [production]
21:08 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on mx2001.wikimedia.org with reason: new kernel [production]
21:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:05 <jhathaway> reboot mx2001 for kernel update [production]
21:05 <brennen@deploy1002> Synchronized php-1.39.0-wmf.10/includes/user: Backport: Revert: [[gerrit:789332|Suppress "named" group when TempUser system is disabled (T307675)]] (duration: 00m 50s) [production]
21:03 <brennen@deploy1002> sync-world aborted: Backport: Revert: [[gerrit:789333|Add messages for the "named" user group (T307675)]] and Backport: [[gerrit:789332|Suppress "named" group when TempUser system is disabled (T307675)]] (duration: 11m 53s) [production]
20:53 <brennen> sync of last patch ongoing, otherwise closing UTC late backport and config window [production]
20:51 <brennen@deploy1002> Started scap: Backport: Revert: [[gerrit:789333|Add messages for the "named" user group (T307675)]] and Backport: [[gerrit:789332|Suppress "named" group when TempUser system is disabled (T307675)]] [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:28 <thcipriani@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:789630|urwiki: allow "sysop" to add/remove "eliminator" (T307029)]] (duration: 00m 49s) [production]
20:22 <thcipriani@deploy1002> backport aborted: (duration: 00m 41s) [production]
20:20 <thcipriani@deploy1002> backport aborted: (duration: 00m 02s) [production]
20:10 <razzi@cumin1001> START - Cookbook sre.kafka.reboot-workers for Kafka main-eqiad cluster: Reboot kafka nodes [production]
18:55 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:54 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:54 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]