2851-2900 of 10000 results (82ms)
2020-03-18 ยง
19:20 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:15 <fdans@deploy1001> Finished deploy [analytics/refinery@549f6a4]: deploying analytics refinery (duration: 15m 02s) [production]
19:11 <hashar> 1.35.0-wmf.24 is on hold: too many blockers [production]
19:00 <fdans@deploy1001> Started deploy [analytics/refinery@549f6a4]: deploying analytics refinery [production]
18:32 <Lucas_WMDE> Morning SWAT done [production]
18:30 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
18:27 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:579018|Update linter whitelist w/ parsoid11's IP address (T246833)]] (beta-only) (duration: 01m 04s) [production]
18:20 <Lucas_WMDE> scap pull on mwdebug1001, attempting to fix mismatched wikiversions alert [production]
18:14 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:580373|Add beta configuration for Wikibase reference formatting (T247416)]] (duration: 01m 08s) [production]
18:13 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
18:13 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:580373|Add beta configuration for Wikibase reference formatting (T247416)]], take II (duration: 01m 07s) [production]
18:11 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
18:11 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:580373|Add beta configuration for Wikibase reference formatting (T247416)]] (duration: 01m 07s) [production]
16:43 <mutante> wtp1025 - Icinga alerted it's running out of disk - 'apt-get clean' lowered disk usage from 97% to 91% [production]
16:00 <hashar@deploy1001> Finished scap: testwiki to 1.35.0-wmf.24 and rebuild l10n cache - T233872 (duration: 61m 23s) [production]
14:58 <hashar@deploy1001> Started scap: testwiki to 1.35.0-wmf.24 and rebuild l10n cache - T233872 [production]
14:41 <vgutierrez> disable TLS session tickets in ulsfo - T245616 T170567 [production]
14:29 <godog> add debug to icinga2001 - T247538 [production]
14:28 <_joe_> restarted php-fpm on mw1283, was throwing SIGILL [production]
14:17 <marostegui> Rename wb_terms on codfw hosts: s8 (wikidatawiki - db2081), s3 (testwikidatawiki - db2109), s4 (commonswiki, testcommonswiki - db2106) T208425 [production]
14:06 <hashar@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.23 [production]
11:59 <hashar@deploy1001> Synchronized php-1.35.0-wmf.24/includes/objectcache/ObjectCache.php: objectcache: Restore keyspace for LocalServerCache service - T247562 (duration: 01m 07s) [production]
11:57 <hashar@deploy1001> Synchronized php-1.35.0-wmf.23/includes/objectcache/ObjectCache.php: objectcache: Restore keyspace for LocalServerCache service - T247562 (duration: 01m 10s) [production]
11:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Decrease db1087, vslow host weight in main, given that the CPU across s8 is now doing a lot better', diff saved to https://phabricator.wikimedia.org/P10715 and previous config saved to /var/cache/conftool/dbconfig/20200318-114259-marostegui.json [production]
11:17 <ema> upload atskafka 0.3 to buster-wikimedia T237993 [production]
11:16 <kart_> EU Mid-day SWAT done [production]
11:11 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|579893|Enable ContentTranslation as a default tool in Malay, Azerbaijani and Estonian WPs (T246622, T246628, T246629)]], take II (duration: 01m 07s) [production]
11:10 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|579893|Enable ContentTranslation as a default tool in Malay, Azerbaijani and Estonian WPs (T246622, T246628, T246629)]] (duration: 01m 07s) [production]
10:58 <_joe_> setting num_retries=0 on mw2224 for eventgate-analytics in envoy (T247484) [production]
10:58 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store (wb_terms table) in wikidata (T208425)]], take II (duration: 01m 06s) [production]
10:55 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store (wb_terms table) in wikidata (T208425)]] (duration: 01m 08s) [production]
10:52 <_joe_> setting num_retries=0, idle_timeout=5s on mw2223 for eventgate-analytics in envoy (T247484) [production]
10:48 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store in testwikidatawiki (T208425)]], take II (duration: 01m 07s) [production]
10:45 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store in testwikidatawiki (T208425)]] (duration: 01m 07s) [production]
10:33 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]], take II (duration: 01m 07s) [production]
10:31 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]] (duration: 01m 07s) [production]
10:14 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]], take II (duration: 01m 07s) [production]
10:12 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]] (duration: 01m 08s) [production]
09:43 <vgutierrez> enabling inbound TLSv1.3 in upload@ulsfo - T170567 [production]
09:18 <vgutierrez> enabling inbound TLSv1.3 in cp4026 - T170567 [production]
08:44 <marostegui> Start replication pc1008 from pc1010 to get some of the new keys so it is not fully empty - T247787 [production]
08:14 <vgutierrez> upgrade ATS to 8.0.6-1wm3 in ulsfo - T170567 [production]
07:55 <moritzm> installing remaining libxslt security updates [production]
07:40 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: eventgate-analytics to use envoy everywhere (duration: 01m 10s) [production]
07:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:31 <marostegui> Reboot pc1008 to try to get its RAID redone - T247787 [production]
00:31 <Amir1> foreachwikiindblist medium deleteEqualMessages.php --delete (T247562) [production]
00:10 <crusnov@deploy1001> Finished deploy [netbox/deploy@14256f9]: netbox 2.7.10 upgrade (duration: 02m 29s) [production]
00:08 <crusnov@deploy1001> Started deploy [netbox/deploy@14256f9]: netbox 2.7.10 upgrade [production]