3351-3400 of 10000 results (89ms)
2019-10-23 ยง
09:23 <arturo> cloudvirt1026 reboot ended OK [admin]
09:13 <arturo> 9 tools-sgeexec nodes and 6 other related VMs are down because hypervisor is rebooting [tools]
09:12 <arturo> rebooting cloudvirt1026 for kernel upgrade [admin]
09:12 <hashar> Pooling back integration-agent-docker-1006 , went offline for unknown reason? [releng]
09:09 <arturo> cloudvirt1025 reboot ended OK [admin]
09:09 <hashar> CI Jenkins: downgrading AnsiColor plugin from 0.6.2 to 0.5.3 # T236222 [releng]
09:03 <arturo> tools-sgebastion-08 is down because hypervisor is rebooting [tools]
09:03 <arturo> paws-master-01/03 and a couple of other servers are down because hypervisor is rebooting [paws]
09:00 <moritzm> rebooting logstash2021 for some firmware tests [production]
09:00 <arturo> rebooting cloudvirt1025 for kernel upgrade [admin]
08:59 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:59 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:54 <moritzm> installing systemd bugfix update on mw canaries [production]
08:51 <arturo> icinga downtime cloudvirt1025/1026 for reboots [admin]
08:50 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:50 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:42 <godog> roll restart rsyslog on cirrus and wqds hosts to pick up changes to logback topic partitions [production]
08:33 <hashar> contint1001: backing up Zuul logs to /var/log/zuul/backup-T236114 so we get gerrit activities traces for T236114 [releng]
08:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2091:3312 after table compression', diff saved to https://phabricator.wikimedia.org/P9452 and previous config saved to /var/cache/conftool/dbconfig/20191023-082826-marostegui.json [production]
08:23 <godog> roll restart logstash in codfw/eqiad to pick up new kafka partitions [production]
08:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s8 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9451 and previous config saved to /var/cache/conftool/dbconfig/20191023-082246-marostegui.json [production]
08:11 <godog> kafka-logging eqiad set 12 partitions for ^mwlog- ^logback- and eqiad.client.error topics [production]
08:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s8 codfw - T231018', diff saved to https://phabricator.wikimedia.org/P9450 and previous config saved to /var/cache/conftool/dbconfig/20191023-080857-marostegui.json [production]
07:55 <godog> kafka-logging delete unused topic syslog-notice [production]
07:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s7 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9449 and previous config saved to /var/cache/conftool/dbconfig/20191023-075106-marostegui.json [production]
07:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s7 codfw - T231018', diff saved to https://phabricator.wikimedia.org/P9448 and previous config saved to /var/cache/conftool/dbconfig/20191023-074828-marostegui.json [production]
07:46 <XioNoX> powering down cr2-esams for relocation (for real this time) [production]
07:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s6 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9447 and previous config saved to /var/cache/conftool/dbconfig/20191023-073831-marostegui.json [production]
07:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s6 codfw - T231018', diff saved to https://phabricator.wikimedia.org/P9446 and previous config saved to /var/cache/conftool/dbconfig/20191023-073556-marostegui.json [production]
07:30 <XioNoX> powering down cr2-esams for relocation [production]
07:28 <hashar> logstash: refreshing index fields for logstash-* indices (via https://logstash.wikimedia.org/app/kibana#/management/kibana/indices/logstash-* ) # T234564 [production]
07:05 <XioNoX> redirect ns2 to eqiad - T235805 [production]
07:04 <marostegui> Enable slow query log 1/10 on db1089 (enwiki) T223151 [production]
07:02 <ayounsi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:02 <ayounsi@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:59 <XioNoX> depool esams - T235805 [production]
06:57 <effie> Depooling mw1317 [production]
06:54 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
06:54 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
06:46 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:38 <marostegui> Compress tables on db1097:3315 T235599 [production]
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 for compression T235599', diff saved to https://phabricator.wikimedia.org/P9445 and previous config saved to /var/cache/conftool/dbconfig/20191023-063800-marostegui.json [production]
05:29 <ema@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=kibana,name=codfw [production]
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9444 and previous config saved to /var/cache/conftool/dbconfig/20191023-052940-marostegui.json [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9443 and previous config saved to /var/cache/conftool/dbconfig/20191023-050812-marostegui.json [production]
04:57 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9442 and previous config saved to /var/cache/conftool/dbconfig/20191023-045722-marostegui.json [production]
04:49 <vgutierrez> repool cp5007 - T234887 [production]
04:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9441 and previous config saved to /var/cache/conftool/dbconfig/20191023-044833-marostegui.json [production]
04:36 <MaxSem> Fixed a page title via namespaceDupes.php on pswiki [production]