|
2020-06-16
ยง
|
| 19:58 |
<ottomata> |
evolving event.SearchSatisfaction Hive table using /analytics/legacy/searchsatisfaction/latest schema |
[analytics] |
| 19:45 |
<MacFan4000> |
restarting after reverting |
[tools.zppixbot-test] |
| 19:41 |
<ottomata> |
bumping Refine refinery jar version to 0.0.127 - T238230 |
[analytics] |
| 19:37 |
<MacFan4000> |
restarting for new pip module |
[tools.zppixbot-test] |
| 19:24 |
<brennen@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.37 (duration: 01m 04s) |
[production] |
| 19:23 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.37 |
[production] |
| 19:18 |
<otto@deploy1001> |
Started deploy [analytics/refinery@8b8ce6e]: deploying refinery source 0.0.127 for eventlogging -> eventgate migration - T249261 |
[production] |
| 19:17 |
<ottomata> |
deploying refinery source 0.0.127 for eventlogging -> eventgate migration - T249261 |
[analytics] |
| 19:15 |
<brennen@deploy1001> |
Synchronized php-1.35.0-wmf.37/skins/Vector/resources/skins.vector.styles/: [[gerrit:605975|Restore Watchlist star]] (duration: 01m 05s) |
[production] |
| 19:07 |
<Krinkle> |
Done fiddling with deploy01 |
[releng] |
| 19:03 |
<brennen> |
CORRECTION: holding _1.35.0-wmf.37_ deploy to group1 for a few minutes while merging & testing fix for T255574 |
[production] |
| 19:01 |
<brennen> |
holding 1.35.0-wmf.27 deploy to group1 for a few minutes while merging & testing fix for T255574 |
[production] |
| 18:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 18:52 |
<qchris> |
Turning on puppet again on gerrit1002 to avoid having it lag too far behind. |
[production] |
| 18:48 |
<hashar> |
Reloading Zuul for Ibb8b8655f1837ced28783ef44f2b7b1e4a15e9ea |
[releng] |
| 18:32 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 18:18 |
<mutante> |
mw2293 - scap pull (because Icinga reports mismatched MW versions) |
[production] |
| 18:01 |
<crusnov@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 17:55 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 17:52 |
<crusnov@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 17:44 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@f4f5d7b]: airflow: adjust glent legal cutoff (duration: 01m 35s) |
[production] |
| 17:42 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@f4f5d7b]: airflow: adjust glent legal cutoff |
[production] |
| 17:32 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 17:25 |
<Krinkle> |
Fiddling on deploy01 with Task 230245 |
[releng] |
| 17:03 |
<herron> |
performing rolling reboots of kafka-main hosts for security updates T254990 |
[production] |
| 16:26 |
<hnowlan@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 16:26 |
<hnowlan> |
Updating changeprop to new container version with updated dependencies |
[production] |
| 16:07 |
<hnowlan@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 16:04 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 16:02 |
<elukey> |
reboot kafka-jumbo1008 for kernel upgrades |
[analytics] |
| 16:02 |
<elukey> |
reboot kafka-jumbo1008 for kernel upgrades |
[production] |
| 15:58 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 15:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1076', diff saved to https://phabricator.wikimedia.org/P11543 and previous config saved to /var/cache/conftool/dbconfig/20200616-154924-marostegui.json |
[production] |
| 15:48 |
<arturo> |
change DNS record k8s.svc.paws.eqiad1.wikimedia.cloud to point to the haproxy VIP port address 172.16.1.171 (T195217) |
[paws] |
| 15:47 |
<arturo> |
associate floating IP 185.15.56.57 with haproxy VIP port (T295217) |
[paws] |
| 15:46 |
<wm-bot> |
<zppixbot> auto-update@website: Synced website repo in 50.s |
[tools.zppixbot] |
| 15:45 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels (duration: 00m 41s) |
[production] |
| 15:44 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels |
[production] |
| 15:43 |
<arturo> |
allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 1b40be58-7182-41aa-95ce-797f94f83d66` (T295217) |
[paws] |
| 15:43 |
<arturo> |
allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 9ccc43d9-1a8a-4287-afda-67e8bab27a9f` (T295217) |
[paws] |
| 15:38 |
<arturo> |
created by hand neutron port 9c0a9a13-e409-49de-9ba3-bc8ec4801dbf `paws-haproxy-vip` (T295217) |
[admin] |
| 15:37 |
<arturo> |
`aborrero@cloudcontrol1004:~ 1 $ sudo wmcs-openstack --os-project-id=paws port create --network 7425e328-560c-4f00-8e99-706f3fb90bb4 paws-haproxy-vip` (T295217) |
[paws] |
| 15:33 |
<milimetric> |
refinery deployed and synced to hdfs, with refinery-source at 0.0.126 |
[analytics] |
| 15:26 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62] (duration: 00m 08s) |
[production] |
| 15:25 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62] |
[production] |
| 15:23 |
<arturo> |
live-hacking paws-puppetmaster-01 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/605944 for T195217 |
[paws] |
| 15:23 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62] (duration: 07m 56s) |
[production] |
| 15:20 |
<elukey> |
reboot kafka-jumbo1007 for kernel upgrades |
[analytics] |
| 15:20 |
<elukey> |
reboot kafka-jumbo1007 for kernel upgrades |
[production] |
| 15:15 |
<moritzm> |
upgrading intel-microcode on jessie hosts |
[production] |