4601-4650 of 10000 results (27ms)
2020-04-15 §
06:43 <marostegui> Deploy schema change on labtestwiki - T250057 [production]
06:43 <XioNoX> re-set asw2-c-eqiad's licenses [production]
06:42 <marostegui> Deploy schema change on labswiki - T250057 [production]
06:32 <XioNoX> set uRPF log action back to log infra wide - T244147 [production]
06:04 <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[5006,5012] - T249335 [production]
05:49 <moritzm> installing git security updates [production]
05:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:22 <kart_> Update cxserver to 2020-04-13-094138-production (T239459, T249469) [production]
05:21 <marostegui> Remove db1114 from tendril and zarcillo T250224 [production]
05:17 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
05:13 <kartik@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
05:11 <kartik@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
05:07 <marostegui> Remove db1114 from tendril - T250224 [production]
2020-04-14 §
23:24 <AndyRussG> re-enabled thank-you, onimailing and new recurring charge jobs [production]
22:59 <AndyRussG> disabled thank-you and omnimailing jobs [production]
22:59 <AndyRussG> fundraising civicrm revision changed from 59e712ce8e to 18d7567cd7 [production]
21:36 <addshore> pool wdqs1006, it is caught up [production]
21:03 <addshore> depool wdqs1006 to give it a chance to catch up on lag [production]
20:34 <cdanis@cumin1001> dbctl commit (dc=all): 'tweak db1111 weight yet again', diff saved to https://phabricator.wikimedia.org/P10979 and previous config saved to /var/cache/conftool/dbconfig/20200414-203426-cdanis.json [production]
20:18 <James_F> Adding Create-Signed-Tag right to wikimedia-ui-base group for wikimedia-ui-base repo [production]
20:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Change s8 weights', diff saved to https://phabricator.wikimedia.org/P10978 and previous config saved to /var/cache/conftool/dbconfig/20200414-201412-marostegui.json [production]
19:58 <marostegui@cumin1001> dbctl commit (dc=all): 'reduce db1126 weight due to cpu issues', diff saved to https://phabricator.wikimedia.org/P10977 and previous config saved to /var/cache/conftool/dbconfig/20200414-195855-marostegui.json [production]
19:57 <cdanis@cumin1001> dbctl commit (dc=all): '+db1111, -db1126', diff saved to https://phabricator.wikimedia.org/P10976 and previous config saved to /var/cache/conftool/dbconfig/20200414-195734-cdanis.json [production]
19:51 <cdanis@cumin1001> dbctl commit (dc=all): 'more weight to db1104', diff saved to https://phabricator.wikimedia.org/P10975 and previous config saved to /var/cache/conftool/dbconfig/20200414-195100-cdanis.json [production]
19:47 <cdanis@cumin1001> dbctl commit (dc=all): '+weight on db1104@s8', diff saved to https://phabricator.wikimedia.org/P10974 and previous config saved to /var/cache/conftool/dbconfig/20200414-194710-cdanis.json [production]
19:26 <jforrester@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.28 [production]
19:22 <ebernhardson@deploy1001> Finished scap: wmf-config/PoolCounterSettings.php cirrus: increase pool counter size for traffic shift to codfw (duration: 21m 55s) [production]
19:00 <ebernhardson@deploy1001> Started scap: wmf-config/PoolCounterSettings.php cirrus: increase pool counter size for traffic shift to codfw [production]
18:41 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
18:38 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:59 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:57 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:34 <jforrester@deploy1001> Finished scap: Testwikis to php-1.35.0-wmf.28 and rebuild i18n cache for T247775 (duration: 42m 37s) [production]
17:26 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@baf0a4b]: Rollback removing k8s rules, again (duration: 00m 56s) [production]
17:25 <ppchelko@deploy1001> Started deploy [changeprop/deploy@baf0a4b]: Rollback removing k8s rules, again [production]
17:23 <ppchelko@deploy1001> deploy aborted: Rollback removing k8s rules, again (duration: 00m 05s) [production]
17:23 <ppchelko@deploy1001> Started deploy [changeprop/deploy@354ae2d]: Rollback removing k8s rules, again [production]
17:12 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@354ae2d]: Remove rules enabled in k8s T248677 attempt 2 (duration: 00m 25s) [production]
17:12 <ppchelko@deploy1001> Started deploy [changeprop/deploy@354ae2d]: Remove rules enabled in k8s T248677 attempt 2 [production]
17:08 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
17:07 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
17:05 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
16:52 <jforrester@deploy1001> Started scap: Testwikis to php-1.35.0-wmf.28 and rebuild i18n cache for T247775 [production]
16:49 <jforrester@deploy1001> sync aborted: testwikis wikis to 1.35.0-wmf.28 (duration: 00m 05s) [production]
16:49 <jforrester@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.28 [production]
16:38 <akosiaris> stop all ganeti components (VMs are fine) on all ganeti2* hosts for key/cert rollover [production]
16:37 <jforrester@deploy1001> Pruned MediaWiki: 1.35.0-wmf.25 (duration: 17m 20s) [production]
16:20 <James_F> Scap cleaning 1.35.0-wmf.25 T247775 [production]
16:07 <ariel@deploy1001> Finished deploy [dumps/dumps@90cbab0]: fix listing of input files for 7z recompression, retry (duration: 00m 04s) [production]