3451-3500 of 10000 results (74ms)
2020-03-10 ยง
14:00 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: switch echotore to use envoy (duration: 00m 57s) [production]
13:52 <marostegui> Stop mysql on db2121 for reimage to buster T246604 [production]
13:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2121 for reimage to buster - T246604', diff saved to https://phabricator.wikimedia.org/P10676 and previous config saved to /var/cache/conftool/dbconfig/20200310-134648-marostegui.json [production]
13:45 <akosiaris@cumin1001> conftool action : set/pooled=inactive; selector: dc=eqiad,service=eventstreams,name=kubernetes.* [production]
13:44 <akosiaris@cumin1001> conftool action : set/pooled=no; selector: dc=eqiad,service=eventstreams,name=kubernetes.* [production]
13:41 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Mediawiki client side error logging on hawwiki (take 2) - T246030 (duration: 00m 57s) [production]
13:40 <akosiaris> bump eventstreams on scb1003 to force users to reconnect, hoping more connections will make it to kubernetes hosts [production]
13:35 <akosiaris> pool all kubernetes hosts in eqiad for eventstreams. weight=2 which means ~20% of requests are going to be served by kubernetes [production]
13:34 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,service=eventstreams,name=kubernetes.* [production]
13:34 <akosiaris@cumin1001> conftool action : set/weight=2; selector: dc=eqiad,service=eventstreams,name=kubernetes.* [production]
13:31 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Mediawiki client side error logging on hawwiki - T246030 (duration: 00m 58s) [production]
13:29 <akosiaris> T202360 upload apertium-oci-fra_0.3.0-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia main [production]
13:25 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
13:23 <gehel@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
13:23 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
13:22 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
13:19 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:17 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
13:17 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:16 <vgutierrez> upgrade ATS on ulsfo to 8.0.6-1wm2 - T245616 [production]
13:16 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:15 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:13 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:10 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:05 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:02 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:01 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:00 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:56 <vgutierrez> upload trafficserver 8.0.6-1wm2 to apt.wm.o (buster) - T245616 [production]
11:41 <Lucas_WMDE> EU SWAT done [production]
11:40 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.35.0-wmf.22/extensions/EventLogging/: SWAT: [[gerrit:578317|Make BackgroundQueue more aware of page unload flow (T246382, T244874)]] (duration: 00m 58s) [production]
11:30 <oblivian@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'echostore' for release 'production' . [production]
11:27 <oblivian@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'echostore' for release 'production' . [production]
11:26 <marostegui> Restart mysqld exporter on db2125 to see if the collection errors decrease from 30 T247290 [production]
11:21 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.35.0-wmf.22/extensions/DiscussionTools/: SWAT: [[gerrit:578364|controller: apply ve.fixBase to the parsed Parsoid response (T245781)]] (duration: 00m 59s) [production]
09:38 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
09:37 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'citoid' for release 'production' . [production]
09:36 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
09:34 <marostegui> es5 deployment window finished T246072 [production]
09:31 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
09:30 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
09:29 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
09:27 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Enable es5 as new writable external store section - T246072 (duration: 00m 57s) [production]
09:26 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Enable es5 as new writable external store section - T246072 (duration: 00m 58s) [production]
09:25 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Enable es5 as new writable external store section - T246072 (duration: 00m 59s) [production]
09:21 <akosiaris> update blubberoid, cxserver, citoid to push the TLS resources changes T244843 [production]
09:21 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
09:21 <akosiaris> update blubberoid, cxserver, citoid to push the TLS resources changes [production]
09:20 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . [production]
09:19 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]