|
2020-06-30
§
|
| 23:20 |
<Reedy> |
Clean up bogus empty captchas from beta presumably caused by testing of T230245 |
[releng] |
| 23:00 |
<bstorm> |
added paws-public.wmflabs.org to the alt-names for acme-chief, which broke it until we hand off the zone to the paws project <sorry!> T195217 T255997 |
[paws] |
| 22:12 |
<RhinosF1> |
deploy web |
[tools.zppixbot] |
| 21:58 |
<RhinosF1> |
deploy web |
[tools.zppixbot] |
| 21:48 |
<crusnov@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 21:46 |
<crusnov@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 21:45 |
<crusnov@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 21:43 |
<crusnov@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 21:42 |
<crusnov@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 21:40 |
<crusnov@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 21:40 |
<crusnov@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 21:38 |
<crusnov@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 21:38 |
<crusnov@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) |
[production] |
| 21:38 |
<crusnov@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 20:22 |
<mutante> |
managed to let certbot get LE certs for gerrit.devtools.wmflabs.org and the floating IP |
[devtools] |
| 19:19 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: group 0 wikis to 1.35.0-wmf.39 # T254176 |
[production] |
| 18:31 |
<cdanis> |
T256790 ✔️ cdanis@netflow2001.codfw.wmnet ~ 🕝☕ sudo apt install valgrind |
[production] |
| 18:28 |
<joal> |
trying to release refinery-source to archiva from jenkins (second time) |
[analytics] |
| 18:27 |
<tgr> |
Morning deploys done |
[production] |
| 18:23 |
<tgr@deploy1001> |
Synchronized php-1.35.0-wmf.39/extensions/ElectronPdfService/src/ElectronPdfServiceHooks.php: Backport: [[gerrit:608485|Hotfix: "Undefined index: print" (T256761)]] (duration: 01m 05s) |
[production] |
| 18:11 |
<shdubsh> |
restart varnishmtail,atsmtail,ncredirmtail on ncredir,cp hosts in codfw and eqsin |
[production] |
| 18:05 |
<cdanis> |
installing libc6-dbg on netflow2001 T256790 |
[production] |
| 17:40 |
<mdholloway> |
mobileapps deployments on k8s failing with timeouts; filed T256786 |
[production] |
| 17:37 |
<cdanis> |
✔️ cdanis@netflow2001.codfw.wmnet ~ 🕜☕ sudo systemctl restart nfacctd |
[production] |
| 17:33 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 17:21 |
<bd808> |
Reset permissions on /data/project/welcomebots-bn/ to 2775 (drwxrwsr-x) |
[tools.welcomebots-bn] |
| 17:18 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 17:17 |
<papaul> |
uplugging msw-c3 power to relocate port on PDU |
[production] |
| 17:09 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@f9df1af]: Update mobileapps to 5c7611b9 (duration: 03m 33s) |
[production] |
| 17:05 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@f9df1af]: Update mobileapps to 5c7611b9 |
[production] |
| 16:57 |
<cdanis> |
T256444 restarted purged on cp2030 and repooling |
[production] |
| 16:48 |
<cdanis> |
T256444 ✔️ cdanis@cp2030.codfw.wmnet ~ 🕐☕ sudo depool |
[production] |
| 16:30 |
<joal> |
Release refinery-source v0.0.129 using jenkins |
[analytics] |
| 16:30 |
<joal> |
Deploy refien |
[analytics] |
| 16:05 |
<elukey> |
re-enable timers on an-launcher1002 after archiva maintenance |
[analytics] |
| 15:54 |
<otto@deploy1001> |
Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3 (duration: 00m 03s) |
[production] |
| 15:54 |
<otto@deploy1001> |
Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3 |
[production] |
| 15:40 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:35 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 15:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:30 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 15:24 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:23 |
<elukey> |
stop timers on an-launcher1002 to ease debugging for refinery deploy |
[analytics] |
| 15:20 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 15:16 |
<otto@deploy1001> |
Finished deploy [analytics/refinery@1112749]: roll back to 1112749 on an-launcher1002, git-fat not pulling artifacts (duration: 01m 21s) |
[production] |
| 15:14 |
<otto@deploy1001> |
Started deploy [analytics/refinery@1112749]: roll back to 1112749 on an-launcher1002, git-fat not pulling artifacts |
[production] |
| 15:14 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:10 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 15:10 |
<moritzm> |
rebooting mwdebug* hosts for kernel security update |
[production] |
| 15:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |