1251-1300 of 10000 results (39ms)
2020-07-23 §
10:24 <akosiaris@cumin1001> conftool action : set/pooled=no; selector: dc=eqiad,service=mobileapps,name=scb1001.* [production]
10:18 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'zotero' for release 'production' . [production]
10:14 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'zotero' for release 'production' . [production]
10:11 <akosiaris> poole kubernetes in mobileapps/eqiad. T218733 [production]
10:11 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,service=mobileapps,name=kubernetes.* [production]
10:06 <volans@deploy1001> Finished deploy [debmonitor/deploy@16d0c45]: Release v0.2.6 (duration: 00m 36s) [production]
10:06 <volans@deploy1001> Started deploy [debmonitor/deploy@16d0c45]: Release v0.2.6 [production]
10:05 <volans@deploy1001> Finished deploy [debmonitor/deploy@44aa1ee]: Release v0.2.6 (duration: 00m 14s) [production]
10:05 <volans@deploy1001> Started deploy [debmonitor/deploy@44aa1ee]: Release v0.2.6 [production]
10:04 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' . [production]
09:51 <akosiaris> prepare for pooling kubernetes mobileapps capacity in eqiad. T218733 [production]
09:51 <akosiaris@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,service=mobileapps,name=kubernetes.* [production]
09:46 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
09:40 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
09:38 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
09:27 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:27 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
09:25 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
09:24 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
09:20 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
09:19 <akosiaris> lower replica count back to 80 for mobileapps. T218733 [production]
09:19 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:19 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
09:02 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
08:59 <marostegui> transfer --type=xtrabackup from db1117:3322 to db1107 T257540 [production]
08:45 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:42 <godog> test librenms poller from netmon2001 [production]
08:41 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:40 <XioNoX> remove pim-rp IPs from last routers - T257573 [production]
08:40 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:39 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:29 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1107 from s1 T257540', diff saved to https://phabricator.wikimedia.org/P12025 and previous config saved to /var/cache/conftool/dbconfig/20200723-082647-marostegui.json [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 to move it to m2 T257540', diff saved to https://phabricator.wikimedia.org/P12024 and previous config saved to /var/cache/conftool/dbconfig/20200723-081650-marostegui.json [production]
05:29 <marostegui> Restore labsdb1009's original weight [production]
00:24 <legoktm@deploy1001> Synchronized php-1.35.0-wmf.41/includes/: T258664: Revert "Add a new type of database to the installer from extension" (2/2) (duration: 01m 08s) [production]
00:22 <legoktm@deploy1001> Synchronized php-1.35.0-wmf.41/includes/libs/rdbms/database/Database.php: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 05s) [production]
00:20 <legoktm@deploy1001> Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org) [production]
00:16 <legoktm@deploy1001> Synchronized php-1.36.0-wmf.1/includes/: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 09s) [production]
00:11 <legoktm@deploy1001> scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
2020-07-22 §
22:07 <cdanis> remove downtime on api.svc.codfw.wmnet T258614 [production]
19:26 <jhuneidi@deploy1001> Synchronized php: group1 wikis to 1.36.0-wmf.1 (duration: 01m 03s) [production]
19:25 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.1 [production]
19:15 <urbanecm@deploy1001> Finished scap: 9529cf8d2570bbf6dd1e919c966f5954e39dbd67: b66ec9143bd96cbf3a20b70f6aa3f2d6d7963bb5: OOUI backport; 93755a6a92923ae390e3a04b19421c8562568d2a: i18n changes for OAuth, removal of spam messages (duration: 42m 26s) [production]
19:14 <ejegg> updated payments-wiki from bf91f8adff to 31a3de1130 [production]
19:11 <mutante> mw2335 - mw2339 - scap pull [production]
18:39 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw233[5-9].codfw.wmnet [production]
18:38 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw233[6-9].codfw.wmnet [production]
18:36 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw233[6-9].codfw.wmnet [production]
18:33 <urbanecm@deploy1001> Started scap: 9529cf8d2570bbf6dd1e919c966f5954e39dbd67: b66ec9143bd96cbf3a20b70f6aa3f2d6d7963bb5: OOUI backport; 93755a6a92923ae390e3a04b19421c8562568d2a: i18n changes for OAuth, removal of spam messages [production]