2751-2800 of 10000 results (95ms)
2023-11-08 §
16:33 <ebernhardson@deploy2002> Started deploy [airflow-dags/search@869cca4]: Set group ownership of processed sparql queries [production]
16:31 <btullis@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [production]
16:24 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: dse_k8s::master [production]
16:23 <btullis@cumin1001> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad [production]
16:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1014.eqiad.wmnet with OS bookworm [production]
16:11 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: dse_k8s::master [production]
16:09 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: dse_k8s::worker [production]
16:04 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1014.eqiad.wmnet with reason: host reimage [production]
16:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc1014.eqiad.wmnet with reason: host reimage [production]
15:57 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: dse_k8s::worker [production]
15:57 <btullis@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
15:51 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2001.codfw.wmnet [production]
15:48 <bvibber> brion running requeueTranscodes.php on mwmaint2002 to continue backfill for iOS-compatible low-res video (throttled) [production]
15:43 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host kubestage2001.codfw.wmnet [production]
15:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestagemaster2001.codfw.wmnet [production]
15:41 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/editor-analytics: apply [production]
15:41 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/editor-analytics: apply [production]
15:33 <bvibber> brion running requeueTranscodes.php to batch-remove old low-res VP9 WebM transcodes (should be low impact) [production]
15:32 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host kubestagemaster2001.codfw.wmnet [production]
15:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2014.codfw.wmnet with reason: host reimage [production]
15:27 <bking@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
15:27 <bking@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
15:26 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
15:26 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
15:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc2014.codfw.wmnet with reason: host reimage [production]
15:18 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: kubernetes::staging::master [production]
15:08 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host pc2014.codfw.wmnet with OS bookworm [production]
15:07 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: kubernetes::staging::master [production]
15:04 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:972716|Revert "ProductionServices.php: Promote pc2014 to pc2 master"]] (duration: 06m 51s) [production]
14:59 <marostegui@deploy2002> marostegui: Continuing with sync [production]
14:59 <marostegui@deploy2002> marostegui: Backport for [[gerrit:972716|Revert "ProductionServices.php: Promote pc2014 to pc2 master"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:59 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: kubernetes::staging::worker [production]
14:58 <marostegui@deploy2002> Started scap: Backport for [[gerrit:972716|Revert "ProductionServices.php: Promote pc2014 to pc2 master"]] [production]
14:53 <bking@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:53 <bking@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
14:51 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: kubernetes::staging::worker [production]
14:51 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:972831|ProductionServices.php: Promote pc2014 to pc2 master]] (duration: 08m 41s) [production]
14:46 <marostegui@deploy2002> marostegui: Continuing with sync [production]
14:44 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: analytics_cluster::zookeeper [production]
14:44 <marostegui@deploy2002> marostegui: Backport for [[gerrit:972831|ProductionServices.php: Promote pc2014 to pc2 master]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:42 <marostegui@deploy2002> Started scap: Backport for [[gerrit:972831|ProductionServices.php: Promote pc2014 to pc2 master]] [production]
14:41 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc[2012,2014].codfw.wmnet,pc1012.eqiad.wmnet with reason: Upgrade [production]
14:41 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc[2012,2014].codfw.wmnet,pc1012.eqiad.wmnet with reason: Upgrade [production]
14:40 <taavi@deploy2002> Finished scap: Backport for [[gerrit:971517|[bnwikisource] Change the wordmark (T350482)]], [[gerrit:971518|[plwiki] Add 'abusefilter-log-private' flag to sysops (T350509)]] (duration: 07m 45s) [production]
14:35 <_joe_> Running puppet on cp-text to pick up the increase in traffic to mw on k8s [production]
14:35 <taavi@deploy2002> taavi and superpes: Continuing with sync [production]
14:34 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: analytics_cluster::zookeeper [production]
14:34 <jayme@deploy2002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
14:34 <taavi@deploy2002> taavi and superpes: Backport for [[gerrit:971517|[bnwikisource] Change the wordmark (T350482)]], [[gerrit:971518|[plwiki] Add 'abusefilter-log-private' flag to sysops (T350509)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:33 <jayme@deploy2002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]