4351-4400 of 10000 results (32ms)
2020-05-05 ยง
15:47 <cstone> SmashPig revision changed from 8c30ed7fe5 to cd1a49da5f [production]
15:38 <kormat@cumin1001> dbctl commit (dc=all): 'Repool es1024 to 100% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11153 and previous config saved to /var/cache/conftool/dbconfig/20200505-153843-kormat.json [production]
15:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:03 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:00 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:58 <hnowlan@deploy1001> Finished deploy [changeprop/deploy@6c65779]: Enabling on_transclusion_update on k8s, disabling on scb (duration: 01m 31s) [production]
14:56 <hnowlan@deploy1001> Started deploy [changeprop/deploy@6c65779]: Enabling on_transclusion_update on k8s, disabling on scb [production]
14:44 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
14:43 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
14:31 <kormat@cumin1001> dbctl commit (dc=all): 'Repool es1024 to 75% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11149 and previous config saved to /var/cache/conftool/dbconfig/20200505-143158-kormat.json [production]
13:46 <akosiaris> deploy cxserver chart 0.0.15 to staging, codfw, eqiad. T219921 [production]
13:45 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
13:41 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
13:41 <hashar> Updated Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler to have it defined in JJB # T97513 [production]
13:36 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
13:18 <vgutierrez> upgrade ATS to version 8.1 () on cp4026, cp4032, cp5006 and cp5011 [production]
13:15 <kormat@cumin1001> dbctl commit (dc=all): 'Repool es1024 to 50% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11147 and previous config saved to /var/cache/conftool/dbconfig/20200505-131520-kormat.json [production]
12:52 <kormat@cumin1001> dbctl commit (dc=all): 'Repool es1024 at 25% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11145 and previous config saved to /var/cache/conftool/dbconfig/20200505-125254-kormat.json [production]
12:37 <XioNoX> push pfw policy - T251769 [production]
12:07 <jbond42> updating cas login page [production]
12:07 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:05 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:03 <moritzm> rolling restart of apache on puppetboard* to pick up OpenLDAP update [production]
11:47 <moritzm> rolling restart of apache on kibana hosts [production]
11:41 <mutante> LDAP - added eamedia to wmf group (T251358) [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 T248086', diff saved to https://phabricator.wikimedia.org/P11144 and previous config saved to /var/cache/conftool/dbconfig/20200505-113152-marostegui.json [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 T248086', diff saved to https://phabricator.wikimedia.org/P11143 and previous config saved to /var/cache/conftool/dbconfig/20200505-113100-marostegui.json [production]
11:30 <marostegui> Drop T248086_wb_terms table on labsdb hosts - T248086 [production]
11:26 <moritzm> rolling restart of apache/FPM on mw1261-mw1265 [production]
11:22 <kart_> EU SWAT done. [production]
11:09 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|592479|Adjust ContentTranslation MT threshold for Chinese WP to 70% (T246383)]] (duration: 01m 01s) [production]
11:01 <moritzm> installing remaining openldap security updates (client-side libs, tools) [production]
11:00 <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1024 for reimaging, add es1023 (master) for reading in the meantime T250666', diff saved to https://phabricator.wikimedia.org/P11141 and previous config saved to /var/cache/conftool/dbconfig/20200505-110031-kormat.json [production]
10:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1126 T248086', diff saved to https://phabricator.wikimedia.org/P11140 and previous config saved to /var/cache/conftool/dbconfig/20200505-104540-marostegui.json [production]
10:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1126 T248086', diff saved to https://phabricator.wikimedia.org/P11139 and previous config saved to /var/cache/conftool/dbconfig/20200505-104441-marostegui.json [production]
10:33 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
10:23 <arturo> copy prometheus-rabbitmq-exporter v0.4 from stretch-wikimedia to buster-wikimedia in apt1001 (T251660) [production]
10:18 <arturo> copy prometheus-pdns-exporter v0.5.1 from stretch-wikimedia to buster-wikimedia in apt1001 (T251575) [production]
10:16 <mutante> temp disabling puppet on all ganeti hosts to carefully deploy change related to rapi cert location [production]
09:37 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
09:36 <moritzm> removing boron.eqiad.wmnet [production]
09:36 <jmm@cumin2001> START - Cookbook sre.hosts.decommission [production]
09:03 <gehel> restarting wdqs updater on all servers [production]
08:53 <moritzm> installing Java security updates on releases* [production]
08:44 <kormat> reimaging es1024 to buster T250666 [production]
08:27 <ema> cp2028 and cp2030 (both upload): varnish-fe restart to clear cache and evaluate 'exp' admission policy T144187 T249809 [production]
08:26 <moritzm> upgrading slapd on serpens/seaborgium [production]
08:19 <ema> cp2027 and cp2029 (both text): varnish-fe restart to clear cache and evaluate 'exp' admission policy T144187 T249809 [production]
08:08 <moritzm> installing Java security updates on notebook/stat hosts [production]