1001-1050 of 10000 results (66ms)
2020-01-08 ยง
16:25 <_joe_> running puppet on deploy1001 to remove my hot-patch to scap.cfg [production]
16:20 <ema> rolling ats-be restart on !text@eqiad, !text@esams to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/562849/ [production]
16:00 <bblack> re-pooling esams text traffic in DNS [production]
15:45 <ema> cumin -s10 -b1 'A:cp-text_eqiad' 'run-puppet-agent -q ; ats-backend-restart' [production]
15:40 <vgutierrez> restarting ats-tls on esams text nodes [production]
15:37 <ema> cumin -s10 -b1 'A:cp-text_esams' 'run-puppet-agent -q ; ats-backend-restart' [production]
15:37 <bblack> authdns-update to depool esams [production]
15:26 <otto@deploy1001> Synchronized wmf-config/ProductionServices.php: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 34s) [production]
15:23 <otto@deploy1001> sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 03m 56s) [production]
15:19 <otto@deploy1001> sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 33s) [production]
15:12 <otto@deploy1001> Scap failed!: 4/11 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
15:11 <otto@deploy1001> sync-file aborted: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 00s) [production]
15:10 <otto@deploy1001> Synchronized wmf-config/ProductionServices.php: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 10s) [production]
15:02 <XioNoX> Routinator 0.6.4 looking good on rpki2001, upgrading rpki1001 - T242197 [production]
15:00 <ottomata> deploying change to use new TLS port for eventgate-analytics - T242224 [production]
14:35 <ema> repool cp4028 after successful X-Analytics-TLS patch test T237993 [production]
14:23 <ema> depool cp4028 to test X-Analytics-TLS patch T237993 [production]
14:07 <XioNoX> add routinator 0.6.4 to reprepro stretch-wikimedia - T242197 [production]
14:00 <ariel@deploy1001> Finished deploy [dumps/dumps@dbd0ecd]: don't regenerate existing 7z files on rerun of the 7z recompression job (duration: 00m 05s) [production]
14:00 <ariel@deploy1001> Started deploy [dumps/dumps@dbd0ecd]: don't regenerate existing 7z files on rerun of the 7z recompression job [production]
12:46 <_joe_> deleting releng/composer-php55:0.1.0 from the docker registry [production]
12:36 <Lucas_WMDE> EU SWAT done [production]
12:34 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:510875|Update Skolt Sami language name (T223544)]] (duration: 01m 06s) [production]
12:30 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.35.0-wmf.11/extensions/Cite: SWAT: [[gerrit:561169|Fix handling of `<references responsive="" />` (T241303)]] (duration: 01m 06s) [production]
12:17 <tarrow@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562777|Enable tainted references on test.wikidata.org (T239621)]] (duration: 01m 19s) [production]
12:08 <kart_> Updated cxserver to 2020-01-06-070550-production (T233405) [production]
12:04 <kartik@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
12:01 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
12:00 <kartik@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
11:47 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: name=kubernetes2001.* [production]
11:45 <akosiaris@cumin1001> conftool action : set/weight=10; selector: service=echostore [production]
11:44 <vgutierrez> uploaded varnish 5.1.3-1wm12 to apt.wikimedia.org (buster) - T242093 [production]
11:44 <akosiaris@cumin1001> conftool action : set/weight=10; selector: name=kubernetes1001.* [production]
11:44 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: name=kubernetes1001.* [production]
11:07 <moritzm> test failover of Ganeti master in eqsin T228099 [production]
11:00 <moritzm> drain ganeti5003 to test new Ganeti setup in eqsin T228099 [production]
10:53 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:53 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:41 <moritzm> rebooting netflow5001 to pick up microcode [production]
10:08 <moritzm> enabling spec-ctr, ssbd. md-clear passthrough for new eqsin cluster T228099 [production]
09:27 <moritzm> installing urldownloader1002 T241979 [production]
09:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1085', diff saved to https://phabricator.wikimedia.org/P10088 and previous config saved to /var/cache/conftool/dbconfig/20200108-091124-marostegui.json [production]
09:00 <moritzm> installing urldownloader1001 T241979 [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P10087 and previous config saved to /var/cache/conftool/dbconfig/20200108-082930-marostegui.json [production]
08:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P10086 and previous config saved to /var/cache/conftool/dbconfig/20200108-082050-marostegui.json [production]
08:09 <marostegui> Upgrade db1085 [production]
08:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1085', diff saved to https://phabricator.wikimedia.org/P10085 and previous config saved to /var/cache/conftool/dbconfig/20200108-080853-marostegui.json [production]
08:07 <marostegui> Deploy schema change on s1 codfw, there will be lag on s1 codfw - T234052 [production]
07:57 <marostegui> Deploy schema change on clouddb2001-dev.labtestwiki - T234052 [production]
07:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1079', diff saved to https://phabricator.wikimedia.org/P10084 and previous config saved to /var/cache/conftool/dbconfig/20200108-072017-marostegui.json [production]