4501-4550 of 10000 results (33ms)
2020-04-15 §
13:23 <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight to 100% of target, and reduce db1104 slightly T250224', diff saved to https://phabricator.wikimedia.org/P10990 and previous config saved to /var/cache/conftool/dbconfig/20200415-132310-kormat.json [production]
13:10 <hashar> contint2001: starting zuul-merger process # T224591 [production]
12:49 <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight to 50% of target T250224', diff saved to https://phabricator.wikimedia.org/P10989 and previous config saved to /var/cache/conftool/dbconfig/20200415-124931-kormat.json [production]
12:41 <vgutierrez> rolling upgrade to ATS 8.0.7-rc0-1wm2 in ulsfo and eqsin - T249335 [production]
12:03 <mutante> puppetmaster1001: revoking ganeti01.svc.eqiad.wmnet and ganeti01.svc.codfw.wmnet certificates. adding eqiad and codfw to cergen .yaml file, recreating ganeti certs [production]
11:27 <awight@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:588701|Deploy Welcome Survey to Serbian Wikipedia and French Wiktionary (T249956)]] (double-sync) (duration: 01m 03s) [production]
11:26 <awight@deploy1001> sync-file aborted: SWAT: [[gerrit:588701|Deploy Welcome Survey to Serbian Wikipedia and French Wiktionary (T249956)]] (double-sync) (duration: 00m 02s) [production]
11:23 <awight> EU SWAT complete [production]
11:22 <awight@deploy1001> Synchronized php-1.35.0-wmf.28/extensions/TwoColConflict: SWAT: [[gerrit:588966|Flatten exit logging (T248601)]] (duration: 01m 09s) [production]
11:09 <awight@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:588701|Deploy Welcome Survey to Serbian Wikipedia and French Wiktionary (T249956)]] (duration: 01m 24s) [production]
10:57 <marostegui> Deploy schema change on s8 codfw master - T250057 [production]
10:25 <ema> cp3050: varnish-frontend-restart to clear mbox lag and see how long it takes to show up T249583 [production]
10:02 <ema> upload purged 0.5 to buster-wikimedia T249583 [production]
09:50 <jynus@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:48 <jynus@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:48 <vgutierrez> disable KA between ats-tls and varnish-fe for POST requests on eqiad - T250258 [production]
09:45 <godog> force-run curator from logstash1008 - T250133 [production]
09:43 <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight some more T250224', diff saved to https://phabricator.wikimedia.org/P10988 and previous config saved to /var/cache/conftool/dbconfig/20200415-094305-kormat.json [production]
09:08 <elukey> restart druid brokers on druid100[4-6] - stuck after datasource deletion [production]
09:07 <vgutierrez> repool cp1081 [production]
08:54 <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight T250224', diff saved to https://phabricator.wikimedia.org/P10986 and previous config saved to /var/cache/conftool/dbconfig/20200415-085432-kormat.json [production]
08:54 <vgutierrez> depool cp1081 for debugging purposes [production]
08:46 <XioNoX> reset edac counters on scb1001 [production]
08:43 <dcausse> errata: elastic (search cluster) reindexing commonswiki_content on cloudelastic (T246882) [production]
08:42 <dcausse> elastic (search cluster) reindex commmonswiki_content on cloudelastic (T246882) [production]
08:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1114 on s8 with low weight T250224', diff saved to https://phabricator.wikimedia.org/P10985 and previous config saved to /var/cache/conftool/dbconfig/20200415-081421-marostegui.json [production]
07:59 <marostegui> Deploy schema change on s7 codfw master - T250057 [production]
07:35 <elukey> restart cloudelastic-chi on cloudelastic1002 to apply new jvm settings - T231517 [production]
06:55 <mutante> install1003 moving /srv/autoinstall to /root, running puppet, leaving a README file to point out it moved to apt1001 [production]
06:47 <marostegui> Deploy schema change on s6 codfw with replication - T250057 [production]
06:43 <marostegui> Deploy schema change on labtestwiki - T250057 [production]
06:43 <XioNoX> re-set asw2-c-eqiad's licenses [production]
06:42 <marostegui> Deploy schema change on labswiki - T250057 [production]
06:32 <XioNoX> set uRPF log action back to log infra wide - T244147 [production]
06:04 <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[5006,5012] - T249335 [production]
05:49 <moritzm> installing git security updates [production]
05:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:22 <kart_> Update cxserver to 2020-04-13-094138-production (T239459, T249469) [production]
05:21 <marostegui> Remove db1114 from tendril and zarcillo T250224 [production]
05:17 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
05:13 <kartik@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
05:11 <kartik@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
05:07 <marostegui> Remove db1114 from tendril - T250224 [production]
2020-04-14 §
23:24 <AndyRussG> re-enabled thank-you, onimailing and new recurring charge jobs [production]
22:59 <AndyRussG> disabled thank-you and omnimailing jobs [production]
22:59 <AndyRussG> fundraising civicrm revision changed from 59e712ce8e to 18d7567cd7 [production]
21:36 <addshore> pool wdqs1006, it is caught up [production]
21:03 <addshore> depool wdqs1006 to give it a chance to catch up on lag [production]
20:34 <cdanis@cumin1001> dbctl commit (dc=all): 'tweak db1111 weight yet again', diff saved to https://phabricator.wikimedia.org/P10979 and previous config saved to /var/cache/conftool/dbconfig/20200414-203426-cdanis.json [production]