|
2019-11-27
ยง
|
| 21:10 |
<bblack> |
restarting acme-chief service on acmechief1001 (daemon appears to be stuck on a lock and nonfunctional for days...) |
[production] |
| 20:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:40 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:32 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:29 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:14 |
<cstone> |
payments-wiki revision changed from 2eb54fd6ef to 06a8c3cdff |
[production] |
| 19:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1119 for schema change', diff saved to https://phabricator.wikimedia.org/P9773 and previous config saved to /var/cache/conftool/dbconfig/20191127-193528-marostegui.json |
[production] |
| 19:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1080 after schema change', diff saved to https://phabricator.wikimedia.org/P9772 and previous config saved to /var/cache/conftool/dbconfig/20191127-193227-marostegui.json |
[production] |
| 19:32 |
<ebernhardson@deploy1001> |
Finished deploy [search/airflow@45b7790]: Allow airflow virtualenv to import system site packages to facilitate libmysqlclient (duration: 00m 45s) |
[production] |
| 19:31 |
<ebernhardson@deploy1001> |
Started deploy [search/airflow@45b7790]: Allow airflow virtualenv to import system site packages to facilitate libmysqlclient |
[production] |
| 19:27 |
<mutante> |
an-airflow1001 - apt-get install python3-mysqldb - start airflow-webserver |
[production] |
| 19:24 |
<ebernhardson@deploy1001> |
Finished deploy [search/airflow@f3bad9d]: revert adding mysqlclient python package (duration: 00m 42s) |
[production] |
| 19:23 |
<ebernhardson@deploy1001> |
Started deploy [search/airflow@f3bad9d]: revert adding mysqlclient python package |
[production] |
| 19:08 |
<ebernhardson@deploy1001> |
Finished deploy [search/airflow@57f4caa]: Install mysqlclient to airflow instance (duration: 00m 40s) |
[production] |
| 19:08 |
<ebernhardson@deploy1001> |
Started deploy [search/airflow@57f4caa]: Install mysqlclient to airflow instance |
[production] |
| 19:00 |
<mutante> |
an-airflow1001: cd /etc/ ; chown airflow airflow; systemctl start airflow-webserver to let airflow write unittests.cfg (it tries to write this on first start and did not have permissions to do so) T236180 |
[production] |
| 18:58 |
<mutante> |
an-airflow1001: cd /etc/ ; chown airflow airflow; systemctl start airflow-webserver to let airflow write unittests.cfg |
[production] |
| 18:57 |
<eileen> |
process-control config revision is b95355c0c0 - repair omnirecipient job off |
[production] |
| 16:57 |
<andrewbogott> |
disabling puppet on clouvirt* and cloudcontrol* while merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552894/ |
[production] |
| 16:50 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external |
[production] |
| 16:32 |
<cdanis@deploy1001> |
Synchronized wmf-config/PoolCounterSettings.php: dd4c76d3d SpecialContributions: max concurrency 3 (instead of 10) T234450 (duration: 01m 17s) |
[production] |
| 16:22 |
<ejegg> |
shifted daily silverpop export start time one hour earlier |
[production] |
| 16:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1080 for schema change', diff saved to https://phabricator.wikimedia.org/P9768 and previous config saved to /var/cache/conftool/dbconfig/20191127-161525-marostegui.json |
[production] |
| 16:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1089 after schema change', diff saved to https://phabricator.wikimedia.org/P9767 and previous config saved to /var/cache/conftool/dbconfig/20191127-161450-marostegui.json |
[production] |
| 16:06 |
<ema> |
cp3050: set proxy.config.http.server_session_sharing.match to "ip" T238494 |
[production] |
| 15:57 |
<_joe_> |
restarting pybal on lvs1015 |
[production] |
| 15:56 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 15:56 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:55 |
<_joe_> |
restarting pybal on lvs1016 |
[production] |
| 15:52 |
<jynus> |
disabling puppet on dbprov1001 to test bacula restore T238048 |
[production] |
| 15:47 |
<papaul> |
testing redundancy power on scs-a1-codfw |
[production] |
| 15:47 |
<_joe_> |
restarting pybal on lvs2003 |
[production] |
| 15:44 |
<_joe_> |
restarting pybal again on lvs2006 |
[production] |
| 15:42 |
<jynus> |
migrate db entries of archive Media to backup1001 T238048 |
[production] |
| 15:37 |
<marostegui> |
Logging retroactively for the record: drop user 'nova'@'%' from m5 - T239170 |
[production] |
| 15:30 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 15:30 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:29 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 15:29 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:29 |
<marostegui> |
Add grants for dump (10.192.0.114,10.192.16.96) for nova_cell0_eqiad database on db1117:3325 and db2078:3325 - T239170 |
[production] |
| 15:27 |
<marostegui> |
Add grants for dump (10.64.0.95,10.64.16.31) for nova_cell0_eqiad database on db1117:3325 and db2078:3325 - T239170 |
[production] |
| 15:25 |
<_joe_> |
restarting lvs2006 for addition of eventgate-logging-external,blubberoid-https |
[production] |
| 15:24 |
<moritzm> |
installing freetype bugfix updates from Buster 10.2 point release |
[production] |
| 15:21 |
<oblivian@cumin1001> |
conftool action : set/weight=10:pooled=yes; selector: service=eventgate-logging-external |
[production] |
| 15:14 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 15:14 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:11 |
<moritzm> |
downgrading trapperkeeper-webserver-jetty9-clojure packages on puppetdb hosts to the version shipped in Buster 10.2 |
[production] |
| 15:06 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 15:06 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:04 |
<ema> |
cp-ats: rolling ats-{tls,backend} restart to enable lua reload T233274 |
[production] |