1701-1750 of 10000 results (18ms)
2020-09-26 §
01:48 <cdanis@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=ores,name=eqiad [production]
01:17 <cdanis> ❌cdanis@ores2001.codfw.wmnet ~ 🕤🍺 sudo systemctl restart uwsgi-ores.service [production]
01:11 <cdanis> ✔️ cdanis@ores2001.codfw.wmnet ~ 🕘🍺 sudo systemctl restart celery-ores-worker.service [production]
00:56 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
00:55 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
00:50 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
00:46 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
00:43 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
00:43 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
00:43 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
2020-09-25 §
23:03 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@a135388]: correct scap variable refernce in airflow_variables (duration: 26m 57s) [production]
22:36 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@a135388]: correct scap variable refernce in airflow_variables [production]
22:17 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@d1a619f]: increase airflow_variable debugging verbosity (duration: 10m 42s) [production]
22:12 <ejegg|food> updated fundraising CiviCRM from eb90dbcfd3 to 035ad1c351 [production]
22:06 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@d1a619f]: increase airflow_variable debugging verbosity [production]
21:23 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@d999f76]: adding debug info to deployment (duration: 11m 33s) [production]
21:11 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@d999f76]: adding debug info to deployment [production]
20:26 <effie> installing memcached 1.4.33-1+deb9u1 on mwdebug1001 [production]
19:34 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@303eaf3]: Enable icutoknorm in glent m0 and m1 (duration: 53m 58s) [production]
18:40 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@303eaf3]: Enable icutoknorm in glent m0 and m1 [production]
17:47 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/MobileFrontend/: Backport: [[gerrit:630065|Make all section `collapsible` during server side rendering (T263832)]] (duration: 00m 59s) [production]
17:37 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@ae3c936]: Deploy glent 0.2.3 (duration: 02m 01s) [production]
17:35 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@ae3c936]: Deploy glent 0.2.3 [production]
16:35 <dcausse@deploy1001> Finished deploy [wikimedia/discovery/analytics@94c8e6a]: fixed start data for wikidata ttl import (duration: 01m 10s) [production]
16:34 <dcausse@deploy1001> Started deploy [wikimedia/discovery/analytics@94c8e6a]: fixed start data for wikidata ttl import [production]
16:33 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: Promote 1.35.0 to stable in extensiondistributor (duration: 00m 57s) [production]
16:29 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:23 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
15:23 <jynus> fixing enwikivoyage ipblocks inconsistency cluster-wide T263842 [production]
14:54 <elukey> install linux-image-4.19-amd64 on an-worker1096 + reboot [production]
12:41 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:41 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:21 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:21 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:13 <kormat@cumin1001> dbctl commit (dc=all): 'Add db2113 to various groups T263842', diff saved to https://phabricator.wikimedia.org/P12797 and previous config saved to /var/cache/conftool/dbconfig/20200925-121332-kormat.json [production]
11:25 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:23 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:10 <moritzm> reimaging sretest1001 to validate puppetised sources.list with a new installation T158562 [production]
10:42 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:40 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:28 <moritzm> reimaging sretest1002 to validate puppetised sources.list with a new installation T158562 [production]
09:58 <moritzm> restarting archiva to pick up Java security update [production]
09:22 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:22 <ema> upload@eqsin: rolling varnish upgrade to 6.0.6-1wm1 T263557 [production]
09:20 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:02 <ema> text@eqsin: rolling varnish upgrade to 6.0.6-1wm1 T263557 [production]
06:50 <elukey> shutdown ganeti5002 (mistakenly powercycled it without seeing T261130) [production]
06:40 <elukey> powercycle ganeti5002 (no instances running on it, mgmt console shows no tty usable) [production]
06:34 <elukey> reboot stat1004 to pick up kernel settings [production]
03:10 <ejegg> updated payments-wiki from f89c594e12 to b2eb456ed1 [production]