|
2021-09-03
ยง
|
| 10:54 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mc[1025-1026].eqiad.wmnet |
[production] |
| 10:47 |
<joal@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): Deploy latest code on AQS new servers - test after failures (duration: 00m 32s) |
[production] |
| 10:46 |
<joal@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): Deploy latest code on AQS new servers - test after failures |
[production] |
| 10:45 |
<joal@deploy1002> |
deploy aborted: Deploy latest code on AQS new servers - test after failures (duration: 00m 05s) |
[production] |
| 10:45 |
<joal@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-test): Deploy latest code on AQS new servers - test after failures |
[production] |
| 10:29 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 03s) |
[production] |
| 10:29 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 10:22 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 55s) |
[production] |
| 10:21 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 10:17 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 36s) |
[production] |
| 10:16 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 10:08 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 45s) |
[production] |
| 10:08 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 10:05 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 36s) |
[production] |
| 10:04 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 10:02 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 01m 25s) |
[production] |
| 10:01 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 10:00 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 01m 53s) |
[production] |
| 09:58 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 09:57 |
<hnowlan@deploy1002> |
Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 09s) |
[production] |
| 09:57 |
<hnowlan@deploy1002> |
Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts |
[production] |
| 09:32 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@4ff8979] (thin): Analytics hotfix deploy THIN [analytics/refinery@4ff8979] (duration: 00m 07s) |
[production] |
| 09:32 |
<joal@deploy1002> |
Started deploy [analytics/refinery@4ff8979] (thin): Analytics hotfix deploy THIN [analytics/refinery@4ff8979] |
[production] |
| 09:26 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@4ff8979]: Analytics hotfix deploy [analytics/refinery@4ff8979] (duration: 17m 36s) |
[production] |
| 09:25 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mc[1025-1026].eqiad.wmnet |
[production] |
| 09:15 |
<jelto@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:14 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1022.eqiad.wmnet |
[production] |
| 09:13 |
<jelto@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 09:09 |
<akosiaris@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mathoid' for release 'production' . |
[production] |
| 09:09 |
<jelto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:09 |
<joal@deploy1002> |
Started deploy [analytics/refinery@4ff8979]: Analytics hotfix deploy [analytics/refinery@4ff8979] |
[production] |
| 09:08 |
<jelto@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 09:06 |
<akosiaris@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mathoid' for release 'production' . |
[production] |
| 09:03 |
<jelto@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:03 |
<jelto@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 08:53 |
<jelto@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 08:52 |
<jelto@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 08:45 |
<ema> |
cp-eqsin: clean apt cache to free up some space T290305 |
[production] |
| 08:45 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mc1022.eqiad.wmnet |
[production] |
| 08:23 |
<akosiaris@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'staging' . |
[production] |
| 07:43 |
<legoktm> |
uploaded pygments 2.10.0+dfsg-1~wmf1 to apt.wm.o in component/pygments |
[production] |
| 07:42 |
<marostegui> |
Remove flaggedrevs_stats2 and flaggedrevs_stats from severak s3 wikis - T289050 |
[production] |
| 07:10 |
<godog> |
more weight to ms-be20[62-65] - T288458 |
[production] |
| 07:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 06:57 |
<marostegui@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 06:45 |
<elukey> |
run `apt-get clean` on cp5012 to free some space (94% of the root partition used) |
[production] |
| 06:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17203 and previous config saved to /var/cache/conftool/dbconfig/20210903-061204-root.json |
[production] |
| 06:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17202 and previous config saved to /var/cache/conftool/dbconfig/20210903-061138-root.json |
[production] |
| 05:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17201 and previous config saved to /var/cache/conftool/dbconfig/20210903-055700-root.json |
[production] |
| 05:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17200 and previous config saved to /var/cache/conftool/dbconfig/20210903-055635-root.json |
[production] |