551-600 of 10000 results (21ms)
2020-09-08 §
06:47 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
06:31 <marostegui> Deploy schema change on s5 eqiad master - T253276 [production]
06:23 <elukey> roll restart of Hadoop master daemons on an-master100[1,2] to pick up new opejdk settings [production]
06:14 <marostegui> Stop MySQL on db1106 for PDU maintenance T261452 [production]
05:34 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
05:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-09-07 §
23:35 <Reedy> Deployed patch for T262213 [production]
21:19 <reedy@deploy1001> Synchronized private/PrivateSettings.php: Remove old mitigation (duration: 00m 55s) [production]
18:04 <urbanecm@deploy1001> Synchronized private/PrivateSettings.php: Update T250887 mitigations (duration: 00m 56s) [production]
16:12 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:10 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:38 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P12511 and previous config saved to /var/cache/conftool/dbconfig/20200907-153857-kormat.json [production]
15:32 <kormat@cumin1001> dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P12510 and previous config saved to /var/cache/conftool/dbconfig/20200907-153206-kormat.json [production]
15:32 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:32 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:21 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P12509 and previous config saved to /var/cache/conftool/dbconfig/20200907-152117-kormat.json [production]
15:17 <kormat@cumin1001> dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P12508 and previous config saved to /var/cache/conftool/dbconfig/20200907-151718-kormat.json [production]
15:17 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:17 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:14 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
15:12 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
15:09 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P12507 and previous config saved to /var/cache/conftool/dbconfig/20200907-150901-kormat.json [production]
15:06 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
15:04 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
15:03 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:03 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:03 <moritzm> rebooting poolcounter1004/1005 [production]
15:03 <kormat@cumin1001> dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P12506 and previous config saved to /var/cache/conftool/dbconfig/20200907-150310-kormat.json [production]
15:03 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:03 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:38 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:38 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1133 from dbctl T253217', diff saved to https://phabricator.wikimedia.org/P12504 and previous config saved to /var/cache/conftool/dbconfig/20200907-143507-marostegui.json [production]
14:27 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:25 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:48 <_joe_> restarting pybal in codfw to pick up the new mobileapps TLS endpoint [production]
13:44 <_joe_> restarting pybal in eqiad to pick up the new mobileapps TLS endpoint [production]
13:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:28 <hashar@deploy1001> Finished deploy [integration/docroot@e4e3af9]: Support published documents outside of the git checkout # T149924 (duration: 00m 05s) [production]
13:27 <hashar@deploy1001> Started deploy [integration/docroot@e4e3af9]: Support published documents outside of the git checkout # T149924 [production]
13:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:25 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:22 <hashar@deploy1001> Finished deploy [integration/docroot@11ab4a0]: (no justification provided) (duration: 00m 10s) [production]
13:22 <hashar@deploy1001> Started deploy [integration/docroot@11ab4a0]: (no justification provided) [production]
13:14 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]