151-200 of 10000 results (18ms)
2020-06-11 ยง
11:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2075 T254139', diff saved to https://phabricator.wikimedia.org/P11469 and previous config saved to /var/cache/conftool/dbconfig/20200611-115430-marostegui.json [production]
11:46 <marostegui> Deploy schema change on s6 codfw - T250066 [production]
11:44 <volans@deploy1001> Finished deploy [homer/deploy@df83901]: Release v0.2.3 (duration: 00m 25s) [production]
11:44 <volans@deploy1001> Started deploy [homer/deploy@df83901]: Release v0.2.3 [production]
11:36 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
11:36 <matthiasmullie> EU BACON done [production]
11:35 <mlitn@deploy1001> Synchronized php-1.35.0-wmf.36/extensions/GrowthExperiments: Help panel: Update guidance behavior rules (duration: 01m 06s) [production]
11:34 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
11:34 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
11:28 <kartik@deploy1001> Synchronized php-1.35.0-wmf.36/extensions/ContentTranslation/modules/tools/mw.cx.tools.IssueTrackingTool.js: Backport: [[gerrit|604587|IssueTrackingTool: Fix js error in getCurrentNodeId method (T254965)]] (duration: 01m 07s) [production]
11:08 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
11:04 <mlitn@deploy1001> Synchronized php-1.35.0-wmf.36/extensions/MachineVision: $aliases should be an array of strings, not AliasGroup objects (duration: 01m 07s) [production]
10:47 <moritzm> repooling mw1318,mw2139,mw2145,mw2147,mw2221,mw2219,mw2250,mw2350 (these were depooled, but seem all fine in Icinga and were probably just forgotten) [production]
10:41 <filippo@cumin1001> conftool action : set/pooled=yes; selector: cluster=thanos,service=thanos-swift [production]
10:40 <filippo@cumin1001> conftool action : set/pooled=yes; selector: cluster=thanos,service=thanos-query [production]
10:37 <moritzm> installing buster kernel security updates (no reboots yet, on hold for regression-free microcode update) [production]
10:32 <godog> roll-restart pybal in eqiad lvs low-traffic [production]
10:21 <mutante> restarting gerrit on gerrit-replica (gerrit2001) - java.lang.OutOfMemoryError: Java heap space [production]
10:21 <Urbanecm> Run scap pull at mwdebug1001 to revert temporary changes [production]
10:14 <Urbanecm> Applying temporary changes on mwdebug1001 [production]
09:58 <moritzm> upgrading netmon* to PHP 7.2.31 [production]
09:55 <marostegui> Upgrade es2025 [production]
09:54 <moritzm> upgrading mwmaint* to PHP 7.2.31 [production]
09:46 <moritzm> upgrading labweb* PHP 7.2.31 [production]
09:36 <elukey> switch piwik.wikimedia.org from matomo1001 to matomo1002 (new buster node) [production]
09:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:48 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . [production]
08:48 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . [production]
08:42 <moritzm> imported memcached 1.6.6-1~wmf10u1 [production]
08:39 <marostegui> Reimage es2024 to buster [production]
08:30 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:30 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:25 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:25 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:25 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:25 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:24 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:24 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:24 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
08:24 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:23 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
08:23 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
08:22 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:20 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:18 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:18 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:01 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
08:01 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
07:59 <moritzm> upgrading remaining job runners in eqiad to PHP 7.2.31 [production]