401-450 of 10000 results (25ms)
2020-11-30 ยง
14:23 <marostegui> Deploy schema change on s3 codfw, lag will show up on s3 codfw T268004 [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1092', diff saved to https://phabricator.wikimedia.org/P13480 and previous config saved to /var/cache/conftool/dbconfig/20201130-141953-marostegui.json [production]
14:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1087 (re)pooling @ 75%: After cloning clouddb1016:3318', diff saved to https://phabricator.wikimedia.org/P13479 and previous config saved to /var/cache/conftool/dbconfig/20201130-141729-root.json [production]
14:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P13478 and previous config saved to /var/cache/conftool/dbconfig/20201130-141146-marostegui.json [production]
14:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1087 (re)pooling @ 50%: After cloning clouddb1016:3318', diff saved to https://phabricator.wikimedia.org/P13477 and previous config saved to /var/cache/conftool/dbconfig/20201130-140226-root.json [production]
13:58 <ema> varnish 6.0.7-1wm1 uploaded to apt.wikimedia.org component/varnish6 T268736 [production]
13:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P13475 and previous config saved to /var/cache/conftool/dbconfig/20201130-134841-marostegui.json [production]
13:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1087 (re)pooling @ 25%: After cloning clouddb1016:3318', diff saved to https://phabricator.wikimedia.org/P13474 and previous config saved to /var/cache/conftool/dbconfig/20201130-134722-root.json [production]
13:23 <jbond42> update zeromq on jessie hosts [production]
13:21 <dcausse> depooling wdqs1004 (lag) [production]
13:18 <moritzm> CAS enabled for racktables [production]
13:16 <gilles@deploy1001> Synchronized debug.json: T268167 Add mwdebug1003 to list of debug servers (duration: 00m 56s) [production]
12:50 <Urbanecm> EU B&C window done [production]
12:46 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 3476644e4c27dd28339f7b10c8871be2e9455394: Grant enwikibooks reviewers suppressredirect and raise move rate limit to 100/60 (T268849; 2nd attempt) (duration: 00m 56s) [production]
12:43 <hnowlan@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Redeploy to fix gelf traffic (duration: 00m 24s) [production]
12:43 <hnowlan@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: Redeploy to fix gelf traffic [production]
12:41 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: dc=eqiad,cluster=maps,service=kartotherian,name=maps1004.eqiad.wmnet [production]
12:40 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 5585fd79119a4e077705789d1c1928c9e9efa956: Enable RelatedArticles on ptwikinews (T268945) (duration: 00m 57s) [production]
12:38 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: ba6d0f8fd2a443e5c913a292365063f01f2d076b: Grant enwikibooks reviewers suppressredirect and raise move rate limit to 100/60 (T268849) (duration: 00m 57s) [production]
12:37 <hnowlan@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Newer codfw maps hosts (duration: 02m 05s) [production]
12:37 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 9942d68c914a56073d2d192434ba24ff8cb921ba: Assign patrolmarks right to autoconfirmed users on itwiki (T268734) (duration: 00m 57s) [production]
12:35 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=eqiad,cluster=maps,service=kartotherian,name=maps1005.eqiad.wmnet [production]
12:35 <hnowlan@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: Newer codfw maps hosts [production]
12:34 <hnowlan@deploy1001> Finished deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts (duration: 00m 24s) [production]
12:34 <hnowlan@deploy1001> Started deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts [production]
12:34 <hnowlan@deploy1001> Finished deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts (duration: 00m 51s) [production]
12:33 <hnowlan@deploy1001> Started deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts [production]
12:32 <Lucas_WMDE> Deployed patch for T260349 [production]
12:27 <hnowlan@deploy1001> Finished deploy [tilerator/deploy@97575e4]: New eqiad maps hosts (duration: 00m 03s) [production]
12:27 <hnowlan@deploy1001> Started deploy [tilerator/deploy@97575e4]: New eqiad maps hosts [production]
12:24 <hnowlan@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: New eqiad maps hosts (duration: 00m 03s) [production]
12:24 <hnowlan@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: New eqiad maps hosts [production]
12:21 <hnowlan@puppetmaster1001> conftool action : set/pooled=no:weight=0; selector: dc=eqiad,cluster=maps,service=kartotherian,name=maps1005.eqiad.wmnet [production]
12:12 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 2922abe7b810f1b53b446af783dc9d51e6585225: Remove wgContentTranslationRESTBase config (T266213) (duration: 00m 57s) [production]
11:43 <marostegui> Sanitize clouddb1016:3318 - T267090 [production]
11:38 <ema> A:cp upgrade fifo-log-demux to 0.6.2 T268883 [production]
11:36 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:644196| Bumping portals to master (T128546)]] (duration: 00m 57s) [production]
11:35 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:644196| Bumping portals to master (T128546)]] (duration: 01m 01s) [production]
11:32 <ariel@deploy1001> Finished deploy [dumps/dumps@e8c6267]: allow page content fixup script to write output files to arbitrary dir (duration: 00m 04s) [production]
11:32 <ariel@deploy1001> Started deploy [dumps/dumps@e8c6267]: allow page content fixup script to write output files to arbitrary dir [production]
11:28 <ema> upload fifo-log-demux 0.6.2 to buster-wikimedia T268883 [production]
11:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13473 and previous config saved to /var/cache/conftool/dbconfig/20201130-111321-root.json [production]
11:00 <hnowlan> bootstrapping maps1005 cassandra [production]
10:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13472 and previous config saved to /var/cache/conftool/dbconfig/20201130-105818-root.json [production]
10:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13471 and previous config saved to /var/cache/conftool/dbconfig/20201130-104314-root.json [production]
10:29 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
10:29 <marostegui> Compare data between clouddb1014:3312 clouddb1018:3312 labsdb1012 T267090 [production]
10:29 <marostegui> Compare data between clouddb1012:3312 clouddb1018:3312 labsdb1012 T267090 [production]
10:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13470 and previous config saved to /var/cache/conftool/dbconfig/20201130-102811-root.json [production]
10:24 <akosiaris> applying https://gerrit.wikimedia.org/r/q/topic:%22k8s_config%22 series of patches [production]