|
2020-06-04
ยง
|
| 09:05 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:05 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:05 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:05 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:05 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:05 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:05 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:04 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:04 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:03 |
<moritzm> |
deploying Java security updates on elastic search nodes |
[production] |
| 09:03 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:00 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:59 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:59 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:59 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:58 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:50 |
<marostegui> |
Repool labsdb1009 after running maintain-views T252219 |
[production] |
| 08:47 |
<wm-bot> |
<jeanfred> Hacking lighttpd config for removing spurious URL parameters |
[tools.wudele] |
| 08:42 |
<moritzm> |
restarting archiva to pick up Java security updates |
[production] |
| 08:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 to clone db1091 on s1 T253217', diff saved to https://phabricator.wikimedia.org/P11392 and previous config saved to /var/cache/conftool/dbconfig/20200604-081545-marostegui.json |
[production] |
| 08:14 |
<marostegui> |
Run sudo /usr/local/sbin/maintain-views --all-databases --replace-all on labsdb1009 - T252219 |
[production] |
| 07:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:45 |
<marostegui> |
Depool labsdb1009 - T252219 |
[production] |
| 07:45 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:35 |
<RhinosF1> |
tools.zppixbot-test@tools-sgebastion-07:~/public_html/new-web/public_html/Font-Awesome$ git submodule init && git submodule update |
[tools.zppixbot-test] |
| 07:33 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=10; selector: dc=eqiad,cluster=labweb,service=labweb-ssl |
[production] |
| 07:32 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=eqiad,cluster=cloudceph,service=cloudceph |
[production] |
| 07:31 |
<elukey> |
stop netflow hive2druid timers to do some experiments |
[analytics] |
| 06:52 |
<mutante> |
mwmaint1002 started mediawiki_job_cirrus_build_completion_indices_eqiad.service |
[production] |
| 06:13 |
<elukey> |
kill application_1589903254658_75731 (druid indexation for netflow still running since 12h ago) |
[analytics] |
| 06:06 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=10; selector: name=logstash200.* |
[production] |
| 06:05 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=10; selector: name=logstash100.* |
[production] |
| 06:04 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=10; selector: cluster=eventschemas,service=eventschemas |
[production] |
| 06:02 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=10; selector: dc=codfw,cluster=elasticsearch,service=elasticsearch.* |
[production] |
| 06:01 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=10; selector: dc=codfw,cluster=elasticsearch,service=elasticsearch |
[production] |
| 05:59 |
<_joe_> |
fixing weights of cp2040 T245594 |
[production] |
| 05:36 |
<elukey> |
restart druid middlemanager on druid1002 - strange protobuf warnings, netflow hive2druid indexation job stuck for hours |
[analytics] |
| 05:31 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:28 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:13 |
<elukey> |
reimage druid1003 to Buster |
[analytics] |