1801-1850 of 10000 results (31ms)
2021-06-28 ยง
22:38 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
22:38 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
22:32 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
22:32 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
22:32 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
22:31 <legoktm> starting DC switchover live test, which will "switch" us from codfw -> eqiad [production]
22:28 <eileen> civicrm revision changed from 9d1203fb28 to 927ab7cff7, config revision is 1739c53fcb [production]
22:09 <legoktm> live-hacked spicerack on cumin1001 to ignore x2, see https://phabricator.wikimedia.org/T285519#7182377 [production]
21:55 <Krinkle> krinkle@mwmaint1002: purgeParserCache.php --tag pc2, ref T282761 [production]
20:03 <cstone> payments-wiki revision is d9892207c1 [production]
19:48 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.11/maintenance/: I618bc1e8ca3008 (duration: 00m 56s) [production]
19:46 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.11/includes/libs/objectcache/: T282761 - I618bc1e8ca3008 (duration: 00m 56s) [production]
19:45 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.11/includes/objectcache/SqlBagOStuff.php: T282761 - I618bc1e8ca3008 (duration: 00m 59s) [production]
18:40 <ebernhardson@deploy1002> Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s) [production]
18:34 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/includes/WelcomeSurveyHooks.php: ecf1d6c47c6cc30d84161e023373c6a2c7287be8: Make it possible to force opt-in/opt-out to Growth features during account creation (T284119; T284800; 3/3) (duration: 00m 55s) [production]
18:32 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/includes/HelpPanelHooks.php: ecf1d6c47c6cc30d84161e023373c6a2c7287be8: Make it possible to force opt-in/opt-out to Growth features during account creation (T284119; T284800; 2/3) (duration: 00m 55s) [production]
18:31 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/includes/HomepageHooks.php: ecf1d6c47c6cc30d84161e023373c6a2c7287be8: Make it possible to force opt-in/opt-out to Growth features during account creation (T284119; T284800; 1/3) (duration: 00m 58s) [production]
18:24 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/VisualEditor/: 794a46c861dbf5ac05ec824d7591e507c1eefd16: Hotfix for broken "Extract show all to placeholder class" (T284636; T285571) (duration: 00m 57s) [production]
18:11 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 4ae0fdd3ad19105fc36bd1eb7102dea9c4a5178d: Enable DiscussionTools topicsubscription as beta feature on partner wikis (T274280) (duration: 00m 57s) [production]
18:10 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5b59184804b2ee0697ad20102eca0646aec4b105: Remove redundant wgDiscussionToolsEnable overrides (duration: 00m 56s) [production]
18:04 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1043c931b4b41d13410e91336f87b122b5447959: Growth: Enable community configuration at all Growth wikis (T285423) (duration: 00m 56s) [production]
16:35 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:15 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
15:44 <sukhe> Traffic: depool eqiad from user traffic [production]
15:30 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:30 <jayme@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=thanos-.*,name=eqiad [production]
15:26 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
15:09 <jayme@cumin1001> END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) [production]
15:08 <jayme@cumin1001> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
15:07 <gehel> restarting wdqs-updater on all wdqs hosts for new configuration [production]
14:54 <jayme@cumin1001> END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) [production]
14:53 <jayme@cumin1001> Switching services swift, proton, mathoid, restbase, swift-ro, eventstreams, search, shellbox, eventgate-analytics-external, wdqs-internal, kartotherian, api-gateway, termbox, mobileapps, similar-users, wikifeeds, apertium, restbase-async, eventgate-main, eventgate-logging-external, ores, sessionstore, linkrecommendation, echostore, push-notifications, citoid, zotero, eventgate-analytics, wdqs, eventstreams-i [production]
14:53 <jayme@cumin1001> START - Cookbook sre.switchdc.services.01-switch-dc [production]
14:37 <jayme@cumin1001> END (FAIL) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=99) [production]
14:36 <jayme@cumin1001> Switching services kartotherian, proton, wdqs-internal, wikifeeds, zotero, recommendation-api, swift-ro, linkrecommendation, mobileapps, citoid, eventgate-analytics, push-notifications, eventstreams-internal, mathoid, similar-users, schema, apertium, restbase-async, shellbox, termbox, wdqs, ores, eventgate-analytics-external, swift, helm-charts, restbase, cxserver, search, sessionstore, eventstreams, api-gate [production]
14:36 <jayme@cumin1001> START - Cookbook sre.switchdc.services.01-switch-dc [production]
14:35 <jayme@cumin1001> END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) [production]
14:29 <jayme@cumin1001> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
14:21 <effie> restarted mw[1322,1329,1333,1350,1351,1352,1353,1354,1366,1367,1368,1370,1372,1373] [production]
14:07 <effie> restarting busy php-fpm app servers [production]
13:07 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:701503|Remove $wmgWikibaseRepoForeignRepositories (T257260)]] (2/2, beta) (duration: 00m 57s) [production]
13:06 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:701503|Remove $wmgWikibaseRepoForeignRepositories (T257260)]] (1/2, prod) (duration: 00m 57s) [production]
12:59 <moritzm> installing intel-microcode security updates on buster [production]
12:29 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.11/includes/media/MediaHandler.php: Backport: [[gerrit:701718|media: Handle lack of 'metadata' key from getSizeAndMetadata gracefully (T285490)]] (duration: 00m 56s) [production]
12:24 <dcausse> repool wdqs1012 [production]
12:00 <Lucas_WMDE> EU backport+config window done [production]
11:50 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:701502|Stop setting Wikibase repo foreignRepositories (T257260)]] (duration: 00m 55s) [production]
11:40 <XioNoX> push "Port cloud-in4 to Capirca" to cr1/2-eqiad [production]
11:38 <XioNoX> push "Port cloud-in4 to Capirca" to cr1/2-codfw [production]
11:34 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: e4a088fbcf90a8c232c616ba5df9ad01cb5449e8: vector: Enable language switcher treatment A/B test on fawiki (T269093) (duration: 00m 55s) [production]