|
2020-06-16
§
|
| 18:52 |
<qchris> |
Turning on puppet again on gerrit1002 to avoid having it lag too far behind. |
[production] |
| 18:48 |
<hashar> |
Reloading Zuul for Ibb8b8655f1837ced28783ef44f2b7b1e4a15e9ea |
[releng] |
| 18:32 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 18:18 |
<mutante> |
mw2293 - scap pull (because Icinga reports mismatched MW versions) |
[production] |
| 18:01 |
<crusnov@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 17:55 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 17:52 |
<crusnov@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 17:44 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@f4f5d7b]: airflow: adjust glent legal cutoff (duration: 01m 35s) |
[production] |
| 17:42 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@f4f5d7b]: airflow: adjust glent legal cutoff |
[production] |
| 17:32 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 17:25 |
<Krinkle> |
Fiddling on deploy01 with Task 230245 |
[releng] |
| 17:03 |
<herron> |
performing rolling reboots of kafka-main hosts for security updates T254990 |
[production] |
| 16:26 |
<hnowlan@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 16:26 |
<hnowlan> |
Updating changeprop to new container version with updated dependencies |
[production] |
| 16:07 |
<hnowlan@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 16:04 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 16:02 |
<elukey> |
reboot kafka-jumbo1008 for kernel upgrades |
[analytics] |
| 16:02 |
<elukey> |
reboot kafka-jumbo1008 for kernel upgrades |
[production] |
| 15:58 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 15:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1076', diff saved to https://phabricator.wikimedia.org/P11543 and previous config saved to /var/cache/conftool/dbconfig/20200616-154924-marostegui.json |
[production] |
| 15:48 |
<arturo> |
change DNS record k8s.svc.paws.eqiad1.wikimedia.cloud to point to the haproxy VIP port address 172.16.1.171 (T195217) |
[paws] |
| 15:47 |
<arturo> |
associate floating IP 185.15.56.57 with haproxy VIP port (T295217) |
[paws] |
| 15:46 |
<wm-bot> |
<zppixbot> auto-update@website: Synced website repo in 50.s |
[tools.zppixbot] |
| 15:45 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels (duration: 00m 41s) |
[production] |
| 15:44 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels |
[production] |
| 15:43 |
<arturo> |
allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 1b40be58-7182-41aa-95ce-797f94f83d66` (T295217) |
[paws] |
| 15:43 |
<arturo> |
allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 9ccc43d9-1a8a-4287-afda-67e8bab27a9f` (T295217) |
[paws] |
| 15:38 |
<arturo> |
created by hand neutron port 9c0a9a13-e409-49de-9ba3-bc8ec4801dbf `paws-haproxy-vip` (T295217) |
[admin] |
| 15:37 |
<arturo> |
`aborrero@cloudcontrol1004:~ 1 $ sudo wmcs-openstack --os-project-id=paws port create --network 7425e328-560c-4f00-8e99-706f3fb90bb4 paws-haproxy-vip` (T295217) |
[paws] |
| 15:33 |
<milimetric> |
refinery deployed and synced to hdfs, with refinery-source at 0.0.126 |
[analytics] |
| 15:26 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62] (duration: 00m 08s) |
[production] |
| 15:25 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62] |
[production] |
| 15:23 |
<arturo> |
live-hacking paws-puppetmaster-01 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/605944 for T195217 |
[paws] |
| 15:23 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62] (duration: 07m 56s) |
[production] |
| 15:20 |
<elukey> |
reboot kafka-jumbo1007 for kernel upgrades |
[analytics] |
| 15:20 |
<elukey> |
reboot kafka-jumbo1007 for kernel upgrades |
[production] |
| 15:15 |
<moritzm> |
upgrading intel-microcode on jessie hosts |
[production] |
| 15:15 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62] |
[production] |
| 15:13 |
<elukey> |
re-enabling timers on launcher after maintenance |
[analytics] |
| 15:10 |
<arturo> |
merging a patch with changes to the template for keepalived (used in the elastic cluster) https://gerrit.wikimedia.org/r/c/operations/puppet/+/605898 |
[tools] |
| 15:06 |
<elukey> |
reboot an-coord1001 for kernel upgrades |
[production] |
| 15:06 |
<elukey> |
reboot an-coord1001 for kernel upgrades |
[analytics] |
| 14:49 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 14:49 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:48 |
<MacFan4000> |
restarting |
[tools.zppixbot-test] |
| 14:45 |
<moritzm> |
rebooting scandium for kernel security update |
[production] |
| 14:45 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:43 |
<cdanis> |
repool eqiad T243080 |
[production] |
| 14:40 |
<papaul> |
power off ms-be2018 for BBU replacement |
[production] |
| 14:33 |
<cdanis> |
eqiad router upgrades completed! 🎉 T243080 |
[production] |