1401-1450 of 10000 results (21ms)
2020-04-15 §
09:43 <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight some more T250224', diff saved to https://phabricator.wikimedia.org/P10988 and previous config saved to /var/cache/conftool/dbconfig/20200415-094305-kormat.json [production]
09:08 <elukey> restart druid brokers on druid100[4-6] - stuck after datasource deletion [production]
09:07 <vgutierrez> repool cp1081 [production]
08:54 <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight T250224', diff saved to https://phabricator.wikimedia.org/P10986 and previous config saved to /var/cache/conftool/dbconfig/20200415-085432-kormat.json [production]
08:54 <vgutierrez> depool cp1081 for debugging purposes [production]
08:46 <XioNoX> reset edac counters on scb1001 [production]
08:43 <dcausse> errata: elastic (search cluster) reindexing commonswiki_content on cloudelastic (T246882) [production]
08:42 <dcausse> elastic (search cluster) reindex commmonswiki_content on cloudelastic (T246882) [production]
08:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1114 on s8 with low weight T250224', diff saved to https://phabricator.wikimedia.org/P10985 and previous config saved to /var/cache/conftool/dbconfig/20200415-081421-marostegui.json [production]
07:59 <marostegui> Deploy schema change on s7 codfw master - T250057 [production]
07:35 <elukey> restart cloudelastic-chi on cloudelastic1002 to apply new jvm settings - T231517 [production]
06:55 <mutante> install1003 moving /srv/autoinstall to /root, running puppet, leaving a README file to point out it moved to apt1001 [production]
06:47 <marostegui> Deploy schema change on s6 codfw with replication - T250057 [production]
06:43 <marostegui> Deploy schema change on labtestwiki - T250057 [production]
06:43 <XioNoX> re-set asw2-c-eqiad's licenses [production]
06:42 <marostegui> Deploy schema change on labswiki - T250057 [production]
06:32 <XioNoX> set uRPF log action back to log infra wide - T244147 [production]
06:04 <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[5006,5012] - T249335 [production]
05:49 <moritzm> installing git security updates [production]
05:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:22 <kart_> Update cxserver to 2020-04-13-094138-production (T239459, T249469) [production]
05:21 <marostegui> Remove db1114 from tendril and zarcillo T250224 [production]
05:17 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
05:13 <kartik@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
05:11 <kartik@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
05:07 <marostegui> Remove db1114 from tendril - T250224 [production]
2020-04-14 §
23:24 <AndyRussG> re-enabled thank-you, onimailing and new recurring charge jobs [production]
22:59 <AndyRussG> disabled thank-you and omnimailing jobs [production]
22:59 <AndyRussG> fundraising civicrm revision changed from 59e712ce8e to 18d7567cd7 [production]
21:36 <addshore> pool wdqs1006, it is caught up [production]
21:03 <addshore> depool wdqs1006 to give it a chance to catch up on lag [production]
20:34 <cdanis@cumin1001> dbctl commit (dc=all): 'tweak db1111 weight yet again', diff saved to https://phabricator.wikimedia.org/P10979 and previous config saved to /var/cache/conftool/dbconfig/20200414-203426-cdanis.json [production]
20:18 <James_F> Adding Create-Signed-Tag right to wikimedia-ui-base group for wikimedia-ui-base repo [production]
20:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Change s8 weights', diff saved to https://phabricator.wikimedia.org/P10978 and previous config saved to /var/cache/conftool/dbconfig/20200414-201412-marostegui.json [production]
19:58 <marostegui@cumin1001> dbctl commit (dc=all): 'reduce db1126 weight due to cpu issues', diff saved to https://phabricator.wikimedia.org/P10977 and previous config saved to /var/cache/conftool/dbconfig/20200414-195855-marostegui.json [production]
19:57 <cdanis@cumin1001> dbctl commit (dc=all): '+db1111, -db1126', diff saved to https://phabricator.wikimedia.org/P10976 and previous config saved to /var/cache/conftool/dbconfig/20200414-195734-cdanis.json [production]
19:51 <cdanis@cumin1001> dbctl commit (dc=all): 'more weight to db1104', diff saved to https://phabricator.wikimedia.org/P10975 and previous config saved to /var/cache/conftool/dbconfig/20200414-195100-cdanis.json [production]
19:47 <cdanis@cumin1001> dbctl commit (dc=all): '+weight on db1104@s8', diff saved to https://phabricator.wikimedia.org/P10974 and previous config saved to /var/cache/conftool/dbconfig/20200414-194710-cdanis.json [production]
19:26 <jforrester@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.28 [production]
19:22 <ebernhardson@deploy1001> Finished scap: wmf-config/PoolCounterSettings.php cirrus: increase pool counter size for traffic shift to codfw (duration: 21m 55s) [production]
19:00 <ebernhardson@deploy1001> Started scap: wmf-config/PoolCounterSettings.php cirrus: increase pool counter size for traffic shift to codfw [production]
18:41 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
18:38 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:59 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:57 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:34 <jforrester@deploy1001> Finished scap: Testwikis to php-1.35.0-wmf.28 and rebuild i18n cache for T247775 (duration: 42m 37s) [production]
17:26 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@baf0a4b]: Rollback removing k8s rules, again (duration: 00m 56s) [production]
17:25 <ppchelko@deploy1001> Started deploy [changeprop/deploy@baf0a4b]: Rollback removing k8s rules, again [production]
17:23 <ppchelko@deploy1001> deploy aborted: Rollback removing k8s rules, again (duration: 00m 05s) [production]