451-500 of 10000 results (28ms)
2020-09-07 ยง
14:25 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:48 <_joe_> restarting pybal in codfw to pick up the new mobileapps TLS endpoint [production]
13:44 <_joe_> restarting pybal in eqiad to pick up the new mobileapps TLS endpoint [production]
13:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:28 <hashar@deploy1001> Finished deploy [integration/docroot@e4e3af9]: Support published documents outside of the git checkout # T149924 (duration: 00m 05s) [production]
13:27 <hashar@deploy1001> Started deploy [integration/docroot@e4e3af9]: Support published documents outside of the git checkout # T149924 [production]
13:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:25 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:22 <hashar@deploy1001> Finished deploy [integration/docroot@11ab4a0]: (no justification provided) (duration: 00m 10s) [production]
13:22 <hashar@deploy1001> Started deploy [integration/docroot@11ab4a0]: (no justification provided) [production]
13:14 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
13:04 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
12:59 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
12:43 <kormat@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) [production]
12:42 <kormat@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
12:29 <marostegui> Upgrade and reboot db2094 and db2095 (sanitarium hosts in codfw) [production]
12:18 <gehel> restarting elasticsearch on elastic2029 (high GC) [production]
12:01 <volans> restart uwsgi on debmonitor1002 to test db reconnection [production]
11:58 <marostegui> Reboot pc1008 for upgrade [production]
11:36 <Urbanecm> EU B&C done [production]
11:30 <urbanecm@deploy1001> Synchronized docroot/noc/index.html: bbfe2ce61014f616d89bc0c21a380c15777b62e3: noc: Remove link to outdated blog (T259978) (duration: 00m 57s) [production]
11:27 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: ff9f1042529bd332effc0fcd18db70f417c2e939: Update help URL (T256623) (duration: 00m 56s) [production]
11:12 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7b512d3a27c4c33949389cbbe7823cc534fbff9a: [hewiktionary] Enable wikilove (T262181) (duration: 00m 57s) [production]
11:07 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 35224f43f1c461d42da5c963bb60d28fbe1992ee: [eswiki] Create an `abusefilter` user group (T262174; 2/2) (duration: 00m 57s) [production]
11:06 <urbanecm@deploy1001> Synchronized wmf-config/abusefilter.php: 35224f43f1c461d42da5c963bb60d28fbe1992ee: [eswiki] Create an `abusefilter` user group (T262174; 1/2) (duration: 01m 20s) [production]
11:02 <Urbanecm> [urbanecm@mwmaint2001 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hewiktionary wikilove # T262181 [production]
11:01 <marostegui> Reboot pc1007 for upgrade [production]
10:37 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:35 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:02 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:00 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:36 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
09:30 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
09:12 <dcausse@deploy1001> Finished deploy [wdqs/wdqs@c96b49e]: deploy wdqs-0.3.47 to wdqs1009 (test server) (duration: 00m 33s) [production]
09:11 <dcausse@deploy1001> Started deploy [wdqs/wdqs@c96b49e]: deploy wdqs-0.3.47 to wdqs1009 (test server) [production]
09:10 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
09:09 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:06 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:02 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
08:53 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
08:49 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
08:29 <jayme@deploy2001> helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:19 <marostegui> Upgrade and restart pc1010 [production]
08:18 <jayme@deploy2001> helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:10 <jayme@deploy2001> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:03 <marostegui> Compress InnoDB on s8 eqiad master (db1109) - T232446 [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 after MCR schema change', diff saved to https://phabricator.wikimedia.org/P12501 and previous config saved to /var/cache/conftool/dbconfig/20200907-051157-marostegui.json [production]