4751-4800 of 10000 results (43ms)
2021-09-01 §
16:23 <mforns@deploy1002> Started deploy [analytics/refinery@ff15071] (thin): Fix for cassandra3 loading THIN [analytics/refinery@ff15071] [production]
16:22 <mforns@deploy1002> Finished deploy [analytics/refinery@ff15071]: Fix for cassandra3 loading [analytics/refinery@ff15071] (duration: 26m 58s) [production]
16:06 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: REIMAGE [production]
16:04 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1065.eqiad.wmnet with reason: REIMAGE [production]
16:02 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1064.eqiad.wmnet with reason: REIMAGE [production]
16:01 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: REIMAGE [production]
16:01 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1065.eqiad.wmnet with reason: REIMAGE [production]
16:00 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1064.eqiad.wmnet with reason: REIMAGE [production]
15:55 <mforns@deploy1002> Started deploy [analytics/refinery@ff15071]: Fix for cassandra3 loading [analytics/refinery@ff15071] [production]
15:35 <jiji@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:08 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
14:08 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:04 <godog> move simone-this-dot from wmf to nda ldap group - T289783 [production]
13:51 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2009.codfw.wmnet [production]
13:49 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:48 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.20/includes/resourceloader: Id7c258841d7816 (duration: 01m 06s) [production]
13:46 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.21/includes/resourceloader: Id7c258841d7816 (duration: 01m 49s) [production]
13:45 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host rdb2009.codfw.wmnet [production]
13:45 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:38 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:16 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
13:05 <mutante> planet1002 - temp removing feed from ad.huikeshoven - seems to cause corrupt state file (T289984) [production]
13:01 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
12:48 <godog> s/webperf/navtiming/ [production]
12:47 <godog> bounce webperf on webperf2001 - T290138 [production]
12:41 <mutante> planet1002 - rm /etc/rawdog/en/feeds/39a7970f.state (corrupt) T289984 [production]
12:38 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
11:19 <Krinkle> effie restarted php-fpm on parse2007.codfw.wmnet, ref T290120. [production]
10:20 <jbond> start filtering more puppet facts G:715461 - T263578 [production]
09:23 <marostegui> Drop flaggedrevs_stats and flaggedrevs_stats2 from dewiki T289050 [production]
07:45 <ema> deploy Varnish SLO dashboard with grr apply slo_dashboards.jsonnet T289036 [production]
07:05 <XioNoX> pfw NAT and ACLs changes - T290077 [production]
06:29 <elukey@cumin1001> END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for sodium.wikimedia.org: Renew puppet certificate - elukey@cumin1001 [production]
06:28 <elukey@cumin1001> START - Cookbook sre.puppet.renew-cert for sodium.wikimedia.org: Renew puppet certificate - elukey@cumin1001 [production]
05:25 <effie> depool mw2251 mw2255 parse2001 for tests - T280497 [production]
04:41 <marostegui> Optimize idwiki.flaggedtemplates T290057 [production]
04:23 <marostegui> Optimize arwiki.flaggedtemplates T290057 [production]
04:16 <eileen> civicrm revision changed from 7da3eba4f9 to 30cd9c1d90, config revision is 5f004d94d7 [production]
00:53 <eileen> civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7 [production]
2021-08-31 §
23:41 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:38 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:38 <eileen> civicrm revision changed from 718aa9cad3 to e567b4c289, config revision is 7a24870bc7 [production]
23:33 <dpifke@deploy1002> Synchronized wmf-config/profiler.php: Revert excimer-k8s pipelines T288165 (duration: 01m 14s) [production]
23:31 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:25 <dpifke@deploy1002> scap failed: average error rate on 3/6 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
23:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:15 <mforns> failed deployment of refinery (v0.1.17) to an-test-coord1001.eqiad.wmnet (scap error) [production]
23:15 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]