251-300 of 10000 results (69ms)
2019-11-27 ยง
22:31 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:21 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:18 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:56 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:54 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:35 <mutante> mw2215 scap pull [production]
21:30 <mutante> mw2215 rebooting [production]
21:10 <bblack> restarting acme-chief service on acmechief1001 (daemon appears to be stuck on a lock and nonfunctional for days...) [production]
20:43 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:32 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:29 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:14 <cstone> payments-wiki revision changed from 2eb54fd6ef to 06a8c3cdff [production]
19:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1119 for schema change', diff saved to https://phabricator.wikimedia.org/P9773 and previous config saved to /var/cache/conftool/dbconfig/20191127-193528-marostegui.json [production]
19:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1080 after schema change', diff saved to https://phabricator.wikimedia.org/P9772 and previous config saved to /var/cache/conftool/dbconfig/20191127-193227-marostegui.json [production]
19:32 <ebernhardson@deploy1001> Finished deploy [search/airflow@45b7790]: Allow airflow virtualenv to import system site packages to facilitate libmysqlclient (duration: 00m 45s) [production]
19:31 <ebernhardson@deploy1001> Started deploy [search/airflow@45b7790]: Allow airflow virtualenv to import system site packages to facilitate libmysqlclient [production]
19:27 <mutante> an-airflow1001 - apt-get install python3-mysqldb - start airflow-webserver [production]
19:24 <ebernhardson@deploy1001> Finished deploy [search/airflow@f3bad9d]: revert adding mysqlclient python package (duration: 00m 42s) [production]
19:23 <ebernhardson@deploy1001> Started deploy [search/airflow@f3bad9d]: revert adding mysqlclient python package [production]
19:08 <ebernhardson@deploy1001> Finished deploy [search/airflow@57f4caa]: Install mysqlclient to airflow instance (duration: 00m 40s) [production]
19:08 <ebernhardson@deploy1001> Started deploy [search/airflow@57f4caa]: Install mysqlclient to airflow instance [production]
19:00 <mutante> an-airflow1001: cd /etc/ ; chown airflow airflow; systemctl start airflow-webserver to let airflow write unittests.cfg (it tries to write this on first start and did not have permissions to do so) T236180 [production]
18:58 <mutante> an-airflow1001: cd /etc/ ; chown airflow airflow; systemctl start airflow-webserver to let airflow write unittests.cfg [production]
18:57 <eileen> process-control config revision is b95355c0c0 - repair omnirecipient job off [production]
16:57 <andrewbogott> disabling puppet on clouvirt* and cloudcontrol* while merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552894/ [production]
16:50 <oblivian@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external [production]
16:32 <cdanis@deploy1001> Synchronized wmf-config/PoolCounterSettings.php: dd4c76d3d SpecialContributions: max concurrency 3 (instead of 10) T234450 (duration: 01m 17s) [production]
16:22 <ejegg> shifted daily silverpop export start time one hour earlier [production]
16:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1080 for schema change', diff saved to https://phabricator.wikimedia.org/P9768 and previous config saved to /var/cache/conftool/dbconfig/20191127-161525-marostegui.json [production]
16:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1089 after schema change', diff saved to https://phabricator.wikimedia.org/P9767 and previous config saved to /var/cache/conftool/dbconfig/20191127-161450-marostegui.json [production]
16:06 <ema> cp3050: set proxy.config.http.server_session_sharing.match to "ip" T238494 [production]
15:57 <_joe_> restarting pybal on lvs1015 [production]
15:56 <jiji@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:56 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:55 <_joe_> restarting pybal on lvs1016 [production]
15:52 <jynus> disabling puppet on dbprov1001 to test bacula restore T238048 [production]
15:47 <papaul> testing redundancy power on scs-a1-codfw [production]
15:47 <_joe_> restarting pybal on lvs2003 [production]
15:44 <_joe_> restarting pybal again on lvs2006 [production]
15:42 <jynus> migrate db entries of archive Media to backup1001 T238048 [production]
15:37 <marostegui> Logging retroactively for the record: drop user 'nova'@'%' from m5 - T239170 [production]
15:30 <jiji@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:30 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:29 <jiji@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:29 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:29 <marostegui> Add grants for dump (10.192.0.114,10.192.16.96) for nova_cell0_eqiad database on db1117:3325 and db2078:3325 - T239170 [production]
15:27 <marostegui> Add grants for dump (10.64.0.95,10.64.16.31) for nova_cell0_eqiad database on db1117:3325 and db2078:3325 - T239170 [production]
15:25 <_joe_> restarting lvs2006 for addition of eventgate-logging-external,blubberoid-https [production]
15:24 <moritzm> installing freetype bugfix updates from Buster 10.2 point release [production]