3401-3450 of 10000 results (88ms)
2023-09-20 ยง
23:44 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
19:26 <bearloga@deploy2002> Finished deploy [airflow-dags/analytics_product@80496b8]: (no justification provided) (duration: 00m 05s) [production]
19:26 <bearloga@deploy2002> Started deploy [airflow-dags/analytics_product@80496b8]: (no justification provided) [production]
19:25 <bearloga@deploy2002> Finished deploy [airflow-dags/analytics_product@80496b8]: (no justification provided) (duration: 00m 09s) [production]
19:24 <bearloga@deploy2002> Started deploy [airflow-dags/analytics_product@80496b8]: (no justification provided) [production]
18:21 <brennen@deploy2002> Synchronized php: group1 wikis to 1.41.0-wmf.27 refs T345888 (duration: 07m 17s) [production]
18:14 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.27 refs T345888 [production]
18:02 <brennen> train 1.41.0-wmf.27 (T345888): no current blockers, logs clean, rolling to group1 [production]
16:29 <elukey@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
16:28 <elukey@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
16:26 <elukey@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
16:26 <klausman> pushing revert of ORES TTL change [production]
16:24 <elukey@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
15:30 <cgoubert@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
15:30 <cgoubert@deploy2002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
15:29 <cgoubert@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
15:29 <cgoubert@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
15:29 <cgoubert@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
15:29 <cgoubert@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
15:09 <moritzm> added Taavi and Effie (new key) to pwstore [production]
15:08 <jgiannelos@deploy2002> helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [production]
15:08 <jgiannelos@deploy2002> helmfile [codfw] START helmfile.d/services/wikifeeds: apply [production]
15:06 <brouberol@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons. [production]
15:05 <jgiannelos@deploy2002> helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [production]
15:05 <jgiannelos@deploy2002> helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [production]
15:04 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
15:03 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw [production]
15:03 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw [production]
15:02 <jgiannelos@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [production]
15:02 <jgiannelos@deploy2002> helmfile [staging] START helmfile.d/services/wikifeeds: apply [production]
14:59 <jgiannelos@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [production]
14:58 <jgiannelos@deploy2002> helmfile [staging] START helmfile.d/services/wikifeeds: apply [production]
14:45 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:45 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new cloud-private records - cmooney@cumin1001" [production]
14:44 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new cloud-private records - cmooney@cumin1001" [production]
14:41 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
14:35 <kamila_> update maintenance.eqiad.wmnet to point to mwmaint2002 [production]
14:26 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic2044.codfw.wmnet for high load - bking@cumin1001 [production]
14:26 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2044.codfw.wmnet for high load - bking@cumin1001 [production]
14:25 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic2044 for high load - bking@cumin1001 [production]
14:25 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2044 for high load - bking@cumin1001 [production]
14:16 <kamila@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) [production]
14:10 <kamila@cumin1001> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters [production]
14:09 <kamila@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: MediaWiki - T346474 (duration: 12m 54s) [production]
14:07 <kamila_> Phase 9.5 Update DNS records for new database masters - T346474 [production]
14:06 <kamila@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) [production]
14:06 <kamila@cumin1001> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl [production]
14:06 <kamila@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
14:04 <marostegui> Testing [production]
14:03 <kamila@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]