4151-4200 of 10000 results (35ms)
2021-09-09 ยง
17:13 <jelto@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
17:13 <jelto@cumin1001> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
17:13 <jelto@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) [production]
17:12 <jelto@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
17:12 <jelto@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
17:12 <jelto@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2021-09-09 17:12:27.974410 [production]
17:12 <jelto@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
17:08 <jelto@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
17:07 <jelto@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
17:07 <jelto@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
17:04 <jelto@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
17:04 <jelto@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
16:58 <jelto@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
16:58 <jelto@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
16:58 <jelto@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
16:57 <jelto> start cookbook sre.switchdc.mediawiki eqiad codfw --live-test this will generate some additional SAL logs here [production]
16:41 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:36 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:33 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:23 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:10 <volans@cumin1001> END (FAIL) - Cookbook sre.experimental.reimage (exit_code=99) for host sretest1001.eqiad.wmnet [production]
16:00 <volans@cumin1001> START - Cookbook sre.experimental.reimage for host sretest1001.eqiad.wmnet [production]
15:34 <volans@cumin1001> END (FAIL) - Cookbook sre.experimental.reimage (exit_code=99) for host sretest1001.eqiad.wmnet [production]
15:32 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:28 <dancy@deploy1002> Synchronized .pipeline/config.yaml: Config: [[gerrit:719610|pipeline: add comment redirecting to correct file]] (duration: 00m 59s) [production]
15:24 <volans@cumin1001> START - Cookbook sre.experimental.reimage for host sretest1001.eqiad.wmnet [production]
14:47 <mutante> planet - deleting all state and lock files for the "en" feeds (T285251 T289984) [production]
14:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx2002.wikimedia.org [production]
14:31 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host mx2002.wikimedia.org [production]
14:25 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
14:25 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
14:19 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
14:19 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
14:11 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps1007.eqiad.wmnet [production]
13:48 <jmm@cumin2002> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host mx2002.wikimedia.org [production]
13:16 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:11 <mutante> planet1002 - re-enabling disabled puppet [production]
13:06 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
13:06 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
13:05 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
13:05 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
13:03 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muehlenhoff out of all services on: 2 hosts [production]
13:03 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Muehlenhoff out of all services on: 2 hosts [production]
13:00 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:56 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:49 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=maps1007.eqiad.wmnet [production]
10:48 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on maps1007.eqiad.wmnet with reason: Resyncing from master [production]
10:48 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on maps1007.eqiad.wmnet with reason: Resyncing from master [production]
10:48 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps1007.eqiad.wmnet [production]