2301-2350 of 10000 results (32ms)
2021-09-03 ยง
10:46 <joal@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): Deploy latest code on AQS new servers - test after failures [production]
10:45 <joal@deploy1002> deploy aborted: Deploy latest code on AQS new servers - test after failures (duration: 00m 05s) [production]
10:45 <joal@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-test): Deploy latest code on AQS new servers - test after failures [production]
10:29 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 03s) [production]
10:29 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
10:22 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 55s) [production]
10:21 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
10:17 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 36s) [production]
10:16 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
10:08 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 45s) [production]
10:08 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
10:05 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 36s) [production]
10:04 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
10:02 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 01m 25s) [production]
10:01 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
10:00 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 01m 53s) [production]
09:58 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
09:57 <hnowlan@deploy1002> Finished deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts (duration: 00m 09s) [production]
09:57 <hnowlan@deploy1002> Started deploy [analytics/aqs/deploy@d273fde] (aqs-next): deploying aqs to inactive aqs-next hosts [production]
09:32 <joal@deploy1002> Finished deploy [analytics/refinery@4ff8979] (thin): Analytics hotfix deploy THIN [analytics/refinery@4ff8979] (duration: 00m 07s) [production]
09:32 <joal@deploy1002> Started deploy [analytics/refinery@4ff8979] (thin): Analytics hotfix deploy THIN [analytics/refinery@4ff8979] [production]
09:26 <joal@deploy1002> Finished deploy [analytics/refinery@4ff8979]: Analytics hotfix deploy [analytics/refinery@4ff8979] (duration: 17m 36s) [production]
09:25 <jiji@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc[1025-1026].eqiad.wmnet [production]
09:15 <jelto@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
09:14 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1022.eqiad.wmnet [production]
09:13 <jelto@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
09:09 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mathoid' for release 'production' . [production]
09:09 <jelto@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
09:09 <joal@deploy1002> Started deploy [analytics/refinery@4ff8979]: Analytics hotfix deploy [analytics/refinery@4ff8979] [production]
09:08 <jelto@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
09:06 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mathoid' for release 'production' . [production]
09:03 <jelto@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:03 <jelto@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
08:53 <jelto@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
08:52 <jelto@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
08:45 <ema> cp-eqsin: clean apt cache to free up some space T290305 [production]
08:45 <jiji@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc1022.eqiad.wmnet [production]
08:23 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'staging' . [production]
07:43 <legoktm> uploaded pygments 2.10.0+dfsg-1~wmf1 to apt.wm.o in component/pygments [production]
07:42 <marostegui> Remove flaggedrevs_stats2 and flaggedrevs_stats from severak s3 wikis - T289050 [production]
07:10 <godog> more weight to ms-be20[62-65] - T288458 [production]
07:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:57 <marostegui@cumin1001> START - Cookbook sre.dns.netbox [production]
06:45 <elukey> run `apt-get clean` on cp5012 to free some space (94% of the root partition used) [production]
06:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17203 and previous config saved to /var/cache/conftool/dbconfig/20210903-061204-root.json [production]
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17202 and previous config saved to /var/cache/conftool/dbconfig/20210903-061138-root.json [production]
05:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17201 and previous config saved to /var/cache/conftool/dbconfig/20210903-055700-root.json [production]
05:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17200 and previous config saved to /var/cache/conftool/dbconfig/20210903-055635-root.json [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 50%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17199 and previous config saved to /var/cache/conftool/dbconfig/20210903-054157-root.json [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 50%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17198 and previous config saved to /var/cache/conftool/dbconfig/20210903-054131-root.json [production]