4001-4050 of 10000 results (34ms)
2020-09-28 §
08:06 <jayme> restarting pybal on lvs3006 for switching to conf1005 - T196487 [production]
08:02 <jayme> restarting pybal on lvs3007 for switching to conf1005 - T196487 [production]
08:02 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) [production]
07:58 <kormat@cumin1001> dbctl commit (dc=all): 'db2125 (re)pooling @ 50%: mobo replaced T260670', diff saved to https://phabricator.wikimedia.org/P12810 and previous config saved to /var/cache/conftool/dbconfig/20200928-075817-kormat.json [production]
07:54 <elukey@cumin1001> START - Cookbook sre.hadoop.stop-cluster [production]
07:43 <kormat@cumin1001> dbctl commit (dc=all): 'db2125 (re)pooling @ 25%: mobo replaced T260670', diff saved to https://phabricator.wikimedia.org/P12809 and previous config saved to /var/cache/conftool/dbconfig/20200928-074313-kormat.json [production]
07:29 <_joe_> restarting pybal on the LVS primaries [production]
07:24 <dcausse> T263970: forcing allocation of enwiki_general_1587198756 (chi@eqiad) [production]
07:18 <_joe_> restarting pybal on the backup LVS in eqiad, codfw to pick up the new wikifeeds endpoint [production]
07:17 <elukey@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) [production]
07:09 <elukey@cumin1001> START - Cookbook sre.presto.roll-restart-workers [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote es2028 as es1 master in codfw T261717', diff saved to https://phabricator.wikimedia.org/P12806 and previous config saved to /var/cache/conftool/dbconfig/20200928-065938-marostegui.json [production]
06:15 <marostegui> Set innodb_change_buffering = inserts; on db2089 (s5), db2106 (s4), db2108 (s2), db2085 (s1), db2085 (s8), db2087 (s7), db2087 (s6), db2109 (s3) T263443 [production]
05:55 <marostegui> Stop MySQL on es2013 before decommissioning it T263740 [production]
05:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es2013 from dbctl T263740', diff saved to https://phabricator.wikimedia.org/P12805 and previous config saved to /var/cache/conftool/dbconfig/20200928-055410-marostegui.json [production]
05:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2013 T263740', diff saved to https://phabricator.wikimedia.org/P12804 and previous config saved to /var/cache/conftool/dbconfig/20200928-054846-marostegui.json [production]
05:22 <marostegui> Decrease labsdb1011 weight [production]
2020-09-27 §
06:36 <elukey> powercycle analytics1048 [production]
2020-09-26 §
19:20 <chrisalbon> sudo service uwsgi-ores restart [production]
02:17 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
02:04 <cdanis@cumin2001> conftool action : set/pooled=false; selector: dnsdisc=ores,name=eqiad [production]
02:04 <cdanis@cumin2001> conftool action : set/pooled=true; selector: dnsdisc=ores,name=codfw [production]
01:56 <cdanis> ❌cdanis@cumin2001.codfw.wmnet ~ 🕙🍺 sudo cumin 'A:ores and A:codfw' 'systemctl restart celery-ores-worker.service uwsgi-ores.service ' [production]
01:48 <cdanis@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=ores,name=codfw [production]
01:48 <cdanis@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=ores,name=eqiad [production]
01:17 <cdanis> ❌cdanis@ores2001.codfw.wmnet ~ 🕤🍺 sudo systemctl restart uwsgi-ores.service [production]
01:11 <cdanis> ✔️ cdanis@ores2001.codfw.wmnet ~ 🕘🍺 sudo systemctl restart celery-ores-worker.service [production]
00:56 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
00:55 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
00:50 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
00:46 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
00:43 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
00:43 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
00:43 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
2020-09-25 §
23:03 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@a135388]: correct scap variable refernce in airflow_variables (duration: 26m 57s) [production]
22:36 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@a135388]: correct scap variable refernce in airflow_variables [production]
22:17 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@d1a619f]: increase airflow_variable debugging verbosity (duration: 10m 42s) [production]
22:12 <ejegg|food> updated fundraising CiviCRM from eb90dbcfd3 to 035ad1c351 [production]
22:06 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@d1a619f]: increase airflow_variable debugging verbosity [production]
21:23 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@d999f76]: adding debug info to deployment (duration: 11m 33s) [production]
21:11 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@d999f76]: adding debug info to deployment [production]
20:26 <effie> installing memcached 1.4.33-1+deb9u1 on mwdebug1001 [production]
19:34 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@303eaf3]: Enable icutoknorm in glent m0 and m1 (duration: 53m 58s) [production]
18:40 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@303eaf3]: Enable icutoknorm in glent m0 and m1 [production]
17:47 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/MobileFrontend/: Backport: [[gerrit:630065|Make all section `collapsible` during server side rendering (T263832)]] (duration: 00m 59s) [production]
17:37 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@ae3c936]: Deploy glent 0.2.3 (duration: 02m 01s) [production]
17:35 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@ae3c936]: Deploy glent 0.2.3 [production]
16:35 <dcausse@deploy1001> Finished deploy [wikimedia/discovery/analytics@94c8e6a]: fixed start data for wikidata ttl import (duration: 01m 10s) [production]
16:34 <dcausse@deploy1001> Started deploy [wikimedia/discovery/analytics@94c8e6a]: fixed start data for wikidata ttl import [production]
16:33 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: Promote 1.35.0 to stable in extensiondistributor (duration: 00m 57s) [production]