4151-4200 of 10000 results (36ms)
2020-07-23 §
08:59 <marostegui> transfer --type=xtrabackup from db1117:3322 to db1107 T257540 [production]
08:45 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:42 <godog> test librenms poller from netmon2001 [production]
08:41 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:40 <XioNoX> remove pim-rp IPs from last routers - T257573 [production]
08:40 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:39 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:29 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1107 from s1 T257540', diff saved to https://phabricator.wikimedia.org/P12025 and previous config saved to /var/cache/conftool/dbconfig/20200723-082647-marostegui.json [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 to move it to m2 T257540', diff saved to https://phabricator.wikimedia.org/P12024 and previous config saved to /var/cache/conftool/dbconfig/20200723-081650-marostegui.json [production]
05:29 <marostegui> Restore labsdb1009's original weight [production]
00:24 <legoktm@deploy1001> Synchronized php-1.35.0-wmf.41/includes/: T258664: Revert "Add a new type of database to the installer from extension" (2/2) (duration: 01m 08s) [production]
00:22 <legoktm@deploy1001> Synchronized php-1.35.0-wmf.41/includes/libs/rdbms/database/Database.php: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 05s) [production]
00:20 <legoktm@deploy1001> Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org) [production]
00:16 <legoktm@deploy1001> Synchronized php-1.36.0-wmf.1/includes/: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 09s) [production]
00:11 <legoktm@deploy1001> scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
2020-07-22 §
22:07 <cdanis> remove downtime on api.svc.codfw.wmnet T258614 [production]
19:26 <jhuneidi@deploy1001> Synchronized php: group1 wikis to 1.36.0-wmf.1 (duration: 01m 03s) [production]
19:25 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.1 [production]
19:15 <urbanecm@deploy1001> Finished scap: 9529cf8d2570bbf6dd1e919c966f5954e39dbd67: b66ec9143bd96cbf3a20b70f6aa3f2d6d7963bb5: OOUI backport; 93755a6a92923ae390e3a04b19421c8562568d2a: i18n changes for OAuth, removal of spam messages (duration: 42m 26s) [production]
19:14 <ejegg> updated payments-wiki from bf91f8adff to 31a3de1130 [production]
19:11 <mutante> mw2335 - mw2339 - scap pull [production]
18:39 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw233[5-9].codfw.wmnet [production]
18:38 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw233[6-9].codfw.wmnet [production]
18:36 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw233[6-9].codfw.wmnet [production]
18:33 <urbanecm@deploy1001> Started scap: 9529cf8d2570bbf6dd1e919c966f5954e39dbd67: b66ec9143bd96cbf3a20b70f6aa3f2d6d7963bb5: OOUI backport; 93755a6a92923ae390e3a04b19421c8562568d2a: i18n changes for OAuth, removal of spam messages [production]
18:33 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2335.codfw.wmnet [production]
18:28 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw233[5-9].codfw.wmnet [production]
18:16 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2339.codfw.wmnet [production]
17:58 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2338.codfw.wmnet [production]
17:58 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2337.codfw.wmnet [production]
17:58 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2336.codfw.wmnet [production]
17:26 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2335.codfw.wmnet [production]
15:31 <moritzm> updated stretch installer image to Stretch 9.13 release T258407 [production]
15:27 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
15:27 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
14:52 <XioNoX> add accept-data and remove bogus v6 IP from ulsfo sandbox vlan [production]
14:43 <akosiaris@cumin1001> conftool action : set/pooled=no; selector: dc=codfw,service=mobileapps,name=scb.* [production]
14:43 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
14:43 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
14:35 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
14:35 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
14:12 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . [production]
14:12 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . [production]
14:06 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:04 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:54 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
13:54 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
13:50 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . [production]
13:49 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . [production]