1951-2000 of 10000 results (37ms)
2021-09-09 ยง
16:33 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:23 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:10 <volans@cumin1001> END (FAIL) - Cookbook sre.experimental.reimage (exit_code=99) for host sretest1001.eqiad.wmnet [production]
16:00 <volans@cumin1001> START - Cookbook sre.experimental.reimage for host sretest1001.eqiad.wmnet [production]
15:34 <volans@cumin1001> END (FAIL) - Cookbook sre.experimental.reimage (exit_code=99) for host sretest1001.eqiad.wmnet [production]
15:32 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:28 <dancy@deploy1002> Synchronized .pipeline/config.yaml: Config: [[gerrit:719610|pipeline: add comment redirecting to correct file]] (duration: 00m 59s) [production]
15:24 <volans@cumin1001> START - Cookbook sre.experimental.reimage for host sretest1001.eqiad.wmnet [production]
14:47 <mutante> planet - deleting all state and lock files for the "en" feeds (T285251 T289984) [production]
14:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx2002.wikimedia.org [production]
14:31 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host mx2002.wikimedia.org [production]
14:25 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
14:25 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
14:19 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
14:19 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
14:11 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps1007.eqiad.wmnet [production]
13:48 <jmm@cumin2002> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host mx2002.wikimedia.org [production]
13:16 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:11 <mutante> planet1002 - re-enabling disabled puppet [production]
13:06 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
13:06 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
13:05 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
13:05 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
13:03 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
13:03 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
13:00 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:56 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:49 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=maps1007.eqiad.wmnet [production]
10:48 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on maps1007.eqiad.wmnet with reason: Resyncing from master [production]
10:48 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on maps1007.eqiad.wmnet with reason: Resyncing from master [production]
10:48 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps1007.eqiad.wmnet [production]
10:48 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps1006.eqiad.wmnet [production]
10:47 <topranks> Removing peering to old IPs of AS139931 (BSCCL) at Equinix Singapore (cr3-eqsin). [production]
10:45 <topranks> Removing peering to AS24218 at Equinix Singapore (cr3-eqsin) - network no longer uses this ASN. [production]
10:22 <volans> upgrading spicerack on cumin1001 [production]
10:20 <volans@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mc1027.eqiad.wmnet [production]
10:10 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts mc1027.eqiad.wmnet [production]
09:56 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host mx2002.wikimedia.org [production]
09:47 <volans@cumin2002> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) for hosts mc1027.eqiad.wmnet [production]
09:46 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts mc1027.eqiad.wmnet [production]
09:37 <godog> swift eqiad add ms-be10[64-67] with initial weight - T290546 [production]
09:19 <filippo@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=swift-ro,name=eqiad [production]
09:19 <filippo@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad [production]
09:15 <volans> rebooting sretest1001 to test ipmi reboot via spicerack [production]
09:15 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on sretest1001.eqiad.wmnet with reason: testing reboot via ipmi [production]
09:15 <volans@cumin2002> START - Cookbook sre.hosts.downtime for 0:20:00 on sretest1001.eqiad.wmnet with reason: testing reboot via ipmi [production]
09:13 <btullis@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - btullis@cumin1001 [production]
09:09 <btullis@cumin1001> START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - btullis@cumin1001 [production]
08:59 <godog> move swift traffic fully to codfw to rebalance eqiad - T287539 [production]