|
2020-06-16
§
|
| 15:48 |
<arturo> |
change DNS record k8s.svc.paws.eqiad1.wikimedia.cloud to point to the haproxy VIP port address 172.16.1.171 (T195217) |
[paws] |
| 15:47 |
<arturo> |
associate floating IP 185.15.56.57 with haproxy VIP port (T295217) |
[paws] |
| 15:46 |
<wm-bot> |
<zppixbot> auto-update@website: Synced website repo in 50.s |
[tools.zppixbot] |
| 15:45 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels (duration: 00m 41s) |
[production] |
| 15:44 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels |
[production] |
| 15:43 |
<arturo> |
allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 1b40be58-7182-41aa-95ce-797f94f83d66` (T295217) |
[paws] |
| 15:43 |
<arturo> |
allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 9ccc43d9-1a8a-4287-afda-67e8bab27a9f` (T295217) |
[paws] |
| 15:38 |
<arturo> |
created by hand neutron port 9c0a9a13-e409-49de-9ba3-bc8ec4801dbf `paws-haproxy-vip` (T295217) |
[admin] |
| 15:37 |
<arturo> |
`aborrero@cloudcontrol1004:~ 1 $ sudo wmcs-openstack --os-project-id=paws port create --network 7425e328-560c-4f00-8e99-706f3fb90bb4 paws-haproxy-vip` (T295217) |
[paws] |
| 15:33 |
<milimetric> |
refinery deployed and synced to hdfs, with refinery-source at 0.0.126 |
[analytics] |
| 15:26 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62] (duration: 00m 08s) |
[production] |
| 15:25 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62] |
[production] |
| 15:23 |
<arturo> |
live-hacking paws-puppetmaster-01 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/605944 for T195217 |
[paws] |
| 15:23 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62] (duration: 07m 56s) |
[production] |
| 15:20 |
<elukey> |
reboot kafka-jumbo1007 for kernel upgrades |
[analytics] |
| 15:20 |
<elukey> |
reboot kafka-jumbo1007 for kernel upgrades |
[production] |
| 15:15 |
<moritzm> |
upgrading intel-microcode on jessie hosts |
[production] |
| 15:15 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62] |
[production] |
| 15:13 |
<elukey> |
re-enabling timers on launcher after maintenance |
[analytics] |
| 15:10 |
<arturo> |
merging a patch with changes to the template for keepalived (used in the elastic cluster) https://gerrit.wikimedia.org/r/c/operations/puppet/+/605898 |
[tools] |
| 15:06 |
<elukey> |
reboot an-coord1001 for kernel upgrades |
[production] |
| 15:06 |
<elukey> |
reboot an-coord1001 for kernel upgrades |
[analytics] |
| 14:49 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 14:49 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:48 |
<MacFan4000> |
restarting |
[tools.zppixbot-test] |
| 14:45 |
<moritzm> |
rebooting scandium for kernel security update |
[production] |
| 14:45 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:43 |
<cdanis> |
repool eqiad T243080 |
[production] |
| 14:40 |
<papaul> |
power off ms-be2018 for BBU replacement |
[production] |
| 14:33 |
<cdanis> |
eqiad router upgrades completed! 🎉 T243080 |
[production] |
| 14:33 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:31 |
<elukey> |
reboot druid100[7,8] for kernel upgrades |
[production] |
| 14:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:27 |
<elukey> |
stop timers on an-launcher1001, prep before rebooting an-coord1001 |
[analytics] |
| 14:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:23 |
<elukey> |
reboot druid100[7,8] for kernel upgrades |
[analytics] |
| 14:22 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P11541 and previous config saved to /var/cache/conftool/dbconfig/20200616-141540-marostegui.json |
[production] |
| 14:14 |
<cdanis> |
T243080 cdanis@re1.cr2-eqiad> request chassis routing-engine master switch |
[production] |
| 14:10 |
<moritzm> |
removing stray nginx packages from mw canaries (mw1261-mw1265 and mw1276-mw1283) T255565 |
[production] |
| 14:06 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:56 |
<cdanis> |
T243080 cdanis@re0.cr2-eqiad> request chassis routing-engine master switch |
[production] |
| 13:50 |
<cdanis> |
cr2-eqiad: rebooting RE1 [backup] with new junos version T243080 |
[production] |
| 13:39 |
<cdanis> |
cr2-eqiad: disable transit/peering BGP & bump fr MED T243080 |
[production] |
| 13:32 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Repool db2092 T254462', diff saved to https://phabricator.wikimedia.org/P11535 and previous config saved to /var/cache/conftool/dbconfig/20200616-133241-marostegui.json |
[production] |