|
2019-10-23
ยง
|
| 09:23 |
<arturo> |
cloudvirt1026 reboot ended OK |
[admin] |
| 09:13 |
<arturo> |
9 tools-sgeexec nodes and 6 other related VMs are down because hypervisor is rebooting |
[tools] |
| 09:12 |
<arturo> |
rebooting cloudvirt1026 for kernel upgrade |
[admin] |
| 09:12 |
<hashar> |
Pooling back integration-agent-docker-1006 , went offline for unknown reason? |
[releng] |
| 09:09 |
<arturo> |
cloudvirt1025 reboot ended OK |
[admin] |
| 09:09 |
<hashar> |
CI Jenkins: downgrading AnsiColor plugin from 0.6.2 to 0.5.3 # T236222 |
[releng] |
| 09:03 |
<arturo> |
tools-sgebastion-08 is down because hypervisor is rebooting |
[tools] |
| 09:03 |
<arturo> |
paws-master-01/03 and a couple of other servers are down because hypervisor is rebooting |
[paws] |
| 09:00 |
<moritzm> |
rebooting logstash2021 for some firmware tests |
[production] |
| 09:00 |
<arturo> |
rebooting cloudvirt1025 for kernel upgrade |
[admin] |
| 08:59 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:59 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:54 |
<moritzm> |
installing systemd bugfix update on mw canaries |
[production] |
| 08:51 |
<arturo> |
icinga downtime cloudvirt1025/1026 for reboots |
[admin] |
| 08:50 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:50 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:42 |
<godog> |
roll restart rsyslog on cirrus and wqds hosts to pick up changes to logback topic partitions |
[production] |
| 08:33 |
<hashar> |
contint1001: backing up Zuul logs to /var/log/zuul/backup-T236114 so we get gerrit activities traces for T236114 |
[releng] |
| 08:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2091:3312 after table compression', diff saved to https://phabricator.wikimedia.org/P9452 and previous config saved to /var/cache/conftool/dbconfig/20191023-082826-marostegui.json |
[production] |
| 08:23 |
<godog> |
roll restart logstash in codfw/eqiad to pick up new kafka partitions |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change weights to x100 on s8 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9451 and previous config saved to /var/cache/conftool/dbconfig/20191023-082246-marostegui.json |
[production] |
| 08:11 |
<godog> |
kafka-logging eqiad set 12 partitions for ^mwlog- ^logback- and eqiad.client.error topics |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change weights to x100 on s8 codfw - T231018', diff saved to https://phabricator.wikimedia.org/P9450 and previous config saved to /var/cache/conftool/dbconfig/20191023-080857-marostegui.json |
[production] |
| 07:55 |
<godog> |
kafka-logging delete unused topic syslog-notice |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change weights to x100 on s7 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9449 and previous config saved to /var/cache/conftool/dbconfig/20191023-075106-marostegui.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change weights to x100 on s7 codfw - T231018', diff saved to https://phabricator.wikimedia.org/P9448 and previous config saved to /var/cache/conftool/dbconfig/20191023-074828-marostegui.json |
[production] |
| 07:46 |
<XioNoX> |
powering down cr2-esams for relocation (for real this time) |
[production] |
| 07:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change weights to x100 on s6 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9447 and previous config saved to /var/cache/conftool/dbconfig/20191023-073831-marostegui.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change weights to x100 on s6 codfw - T231018', diff saved to https://phabricator.wikimedia.org/P9446 and previous config saved to /var/cache/conftool/dbconfig/20191023-073556-marostegui.json |
[production] |
| 07:30 |
<XioNoX> |
powering down cr2-esams for relocation |
[production] |
| 07:28 |
<hashar> |
logstash: refreshing index fields for logstash-* indices (via https://logstash.wikimedia.org/app/kibana#/management/kibana/indices/logstash-* ) # T234564 |
[production] |
| 07:05 |
<XioNoX> |
redirect ns2 to eqiad - T235805 |
[production] |
| 07:04 |
<marostegui> |
Enable slow query log 1/10 on db1089 (enwiki) T223151 |
[production] |
| 07:02 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:02 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:59 |
<XioNoX> |
depool esams - T235805 |
[production] |
| 06:57 |
<effie> |
Depooling mw1317 |
[production] |
| 06:54 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:54 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:46 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:46 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:38 |
<marostegui> |
Compress tables on db1097:3315 T235599 |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3315 for compression T235599', diff saved to https://phabricator.wikimedia.org/P9445 and previous config saved to /var/cache/conftool/dbconfig/20191023-063800-marostegui.json |
[production] |
| 05:29 |
<ema@puppetmaster1001> |
conftool action : set/pooled=false; selector: dnsdisc=kibana,name=codfw |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9444 and previous config saved to /var/cache/conftool/dbconfig/20191023-052940-marostegui.json |
[production] |
| 05:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9443 and previous config saved to /var/cache/conftool/dbconfig/20191023-050812-marostegui.json |
[production] |
| 04:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9442 and previous config saved to /var/cache/conftool/dbconfig/20191023-045722-marostegui.json |
[production] |
| 04:49 |
<vgutierrez> |
repool cp5007 - T234887 |
[production] |
| 04:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9441 and previous config saved to /var/cache/conftool/dbconfig/20191023-044833-marostegui.json |
[production] |
| 04:36 |
<MaxSem> |
Fixed a page title via namespaceDupes.php on pswiki |
[production] |