1751-1800 of 10000 results (38ms)
2021-10-12 ยง
16:56 <bd808@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
16:55 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2009.codfw.wmnet [production]
16:53 <moritzm> failed over ganeti master for test cluster to ganeti2025 [production]
16:50 <bd808@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
16:48 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts testvm2009.codfw.wmnet [production]
16:32 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:30 <volans@cumin2002> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) for hosts testvm2009.codfw.wmnet [production]
16:30 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts testvm2009.codfw.wmnet [production]
16:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:26 <volans@cumin2002> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) for new host testvm2009.codfw.wmnet [production]
16:26 <dancy@deploy1002> Synchronized php-1.38.0-wmf.4/includes: Backport: [[gerrit:730226|Pre-format comments for non-local files too (T292570)]] (duration: 01m 15s) [production]
16:17 <volans@cumin2002> START - Cookbook sre.ganeti.makevm for new host testvm2009.codfw.wmnet [production]
16:16 <volans@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts testvm2009.codfw.wmnet [production]
16:11 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:10 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts testvm2009.codfw.wmnet [production]
16:09 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2009.codfw.wmnet [production]
16:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:06 <dancy@deploy1002> Synchronized php-1.38.0-wmf.4/extensions/SecurePoll/includes/Hooks/HookRunner.php: Backport: [[gerrit:730231|Fix wrong var being passed (T289950 T293102)]] (duration: 00m 57s) [production]
16:00 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts testvm2009.codfw.wmnet [production]
15:59 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:58 <dancy@deploy1002> Synchronized php-1.38.0-wmf.3/extensions/SecurePoll/includes/Hooks/HookRunner.php: Backport: [[gerrit:730230|Fix wrong var being passed (T289950 T293102)]] (duration: 02m 13s) [production]
15:57 <volans@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2009.codfw.wmnet [production]
15:57 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:51 <jgiannelos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
15:49 <volans@cumin2002> START - Cookbook sre.ganeti.makevm for new host testvm2009.codfw.wmnet [production]
15:48 <volans@cumin2002> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) for new host testvm2009.codfw.wmnet [production]
15:48 <volans@cumin2002> START - Cookbook sre.ganeti.makevm for new host testvm2009.codfw.wmnet [production]
15:41 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for analytics1069.eqiad.wmnet [production]
15:41 <btullis@cumin1001> START - Cookbook sre.hosts.remove-downtime for analytics1069.eqiad.wmnet [production]
15:02 <volans@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:50 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
13:49 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
13:40 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2006.codfw.wmnet [production]
13:25 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet [production]
13:21 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:14 <godog> add 50G to prometheus/k8s in eqiad [production]
13:13 <otto@deploy1002> Synchronized wmf-config/CommonSettings.php: Enable x_client_ip_forwarding_enabled for eventgate-analytics and eventgate-analytics-external - T288853 (duration: 00m 56s) [production]
13:11 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on analytics1069.eqiad.wmnet with reason: draining flea power T291732 [production]
13:11 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on analytics1069.eqiad.wmnet with reason: draining flea power T291732 [production]
13:05 <volans> upgraed spicerack to 1.0.5 on cumin hosts [production]
12:25 <volans> uploaded spicerack_1.0.5 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
12:15 <elukey> `kafka topics --alter --topic codfw.mediawiki.job.cirrusSearchElasticaWrite --partitions 5` - T288825 [production]
12:15 <elukey> `kafka topics --alter --topic eqiad.mediawiki.job.cirrusSearchElasticaWrite --partitions 5` - T288825 [production]
12:10 <elukey> `kafka topics --alter --topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5` - T288825 [production]
12:09 <elukey> `kafka topics --alter --topic eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5` - T288825 [production]
11:58 <elukey> `kafka topics --alter --topic codfw.resource-purge --partitions 5` on kafka-main2001 - T288825 [production]
11:49 <elukey> `kafka topics --alter --topic eqiad.resource-purge --partitions 5` on kafka-main2001 - T288825 [production]
11:46 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet [production]
11:44 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]