1-50 of 10000 results (28ms)
2020-08-20 ยง
22:31 <eileen> civicrm revision changed from 27d5900f7d to ce28723709, config revision is 706cf3c898 [production]
22:20 <eileen> civicrm revision is 27d5900f7d, config revision is 706cf3c898 [production]
22:20 <mutante> permanently shut down tungsten.eqiad.wmnet T260395 T158837 T180761 T224549 [production]
22:18 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
22:17 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
21:35 <ejegg> updated fundraising CiviCRM from 958a79f660 to 27d5900f7d [production]
20:53 <cdanis> repool eqsin [production]
20:37 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
20:36 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
20:34 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
20:25 <cdanis> cdanis@cr2-eqsin> request vmhost reboot [production]
20:17 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
20:16 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
20:13 <cdanis> cdanis@cr2-eqsin> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-18.2R3-S5.3.tgz [production]
20:11 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
20:02 <cdanis> depool eqsin for router upgrade [production]
19:57 <ayounsi@cumin1001> END (ERROR) - Cookbook sre.network.prepare-upgrade (exit_code=97) [production]
19:37 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
19:34 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
19:34 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
19:24 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
19:17 <bstorm@cumin1001> END (FAIL) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=99) [production]
19:17 <bstorm@cumin1001> START - Cookbook wmcs.wikireplicas.add_wiki [production]
19:17 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.5 refs T257973 [production]
19:08 <mutante> restarted apache on cont2001 for integration.wikimedia.org docroot change [production]
19:07 <mutante> switching document root of integration.wikimedia.org to scap (T149924) [production]
19:02 <twentyafterfour> 1.36.0-wmf.5 has no known blockers and logspam is cleaned up, time to roll group2 wikis to wmf.5 [production]
18:42 <bstorm@cumin1001> END (FAIL) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=99) [production]
18:42 <bstorm@cumin1001> START - Cookbook wmcs.wikireplicas.add_wiki [production]
18:19 <mutante> ores1004 - starting failed celery-ores-worker [production]
18:18 <mutante> testreduce1001 - rt_client and vd_client now properly stopped by puppet T257906 [production]
17:29 <shdubsh> restart elasticsearch on logstash1012 (not 1020) -- high gc runtimes [production]
17:28 <shdubsh> restart elasticsearch on logstash1020 -- high gc runtimes [production]
17:23 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
17:23 <ayounsi@cumin1001> END (ERROR) - Cookbook sre.network.prepare-upgrade (exit_code=97) [production]
17:23 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
17:22 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99) [production]
17:22 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
16:48 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:48 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:46 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:45 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
16:43 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
16:40 <_joe_> restarted apache2 on icinga1001 [production]
16:13 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:11 <shdubsh> restart elasticsearch on logstash1011 -- long gc runs [production]
16:10 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:08 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:02 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
14:55 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]