3301-3350 of 10000 results (50ms)
2021-10-07 §
05:53 <ryankemper> [Elastic] `ryankemper@relforge1004:~$ sudo systemctl restart elasticsearch_6@relforge-eqiad-small-alpha.service && sudo systemctl restart elasticsearch_6@relforge-eqiad.service` [production]
05:48 <ryankemper> [Elastic] Performing rolling restarts of `relforge`. `relforge1003` is the master so I'll restart `relforge1004` first to minimize disruption [production]
03:00 <ejegg> updated payments-wiki from 23d0ffac66 to 6d3560d083 [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
02:28 <tstarling@deploy1002> Synchronized wmf-config/CommonSettings.php: enable Parsoid API everywhere (duration: 01m 04s) [production]
02:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:11 <mutante> [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync [production]
2021-10-06 §
23:57 <mutante> releases2002 - rm /srv/org/wikimedia/reprepro/conf/distributions - contains only jessie-mediawiki - see 725670 and EOL of MediaWiki 1.31 [production]
23:32 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:21 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:21 <jforrester@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:726955|Adding and use wordmark in ckbwiki (T288368)]] (duration: 01m 04s) [production]
23:20 <jforrester@deploy1002> Synchronized static/images/mobile/copyright/wikipedia-wordmark-ckb.svg: Config: [[gerrit:726955|Adding and use wordmark in ckbwiki (T288368)]] (duration: 01m 04s) [production]
23:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:16 <jforrester@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:726603|Enable NewUserMessage for ptwikivoyage (T290820)]] (duration: 01m 05s) [production]
22:30 <mutante> re-enabling puppet on mw*, an-worker* after deploying gerrit:726954. no issue this time [production]
22:23 <mutante> temp. disabling puppet on an-worker*, mw* [production]
20:50 <mutante> global puppet failure - revert is merged, puppet run will recover on next run everywhere. partially forcing with cumin, partially letting it recover naturally [production]
20:43 <mutante> [cumin1001:~] $ sudo cumin -b 5 -p 95 'mw2*' 'run-puppet-agent -q --failed-only' [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:05 <brennen@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.3 refs T281167 (duration: 01m 03s) [production]
19:04 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.3 refs T281167 [production]
19:01 <brennen> 1.38.0-wmf.3 train (T281167): still unblocked after triage meeting, rolling to group1 [production]
18:52 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:50 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:44 <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Revert disabling static mapframes on eswiki (duration: 01m 14s) [production]
18:36 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:34 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:31 <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: eswiki: Disable static mapframes (T291736) (duration: 01m 17s) [production]
18:25 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:22 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:22 <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: viwikibooks: Set $wgRestrictDisplayTitle to false (T289837) (duration: 01m 21s) [production]
17:16 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
17:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:53 <bd808@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
16:47 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.3 refs T281167 [production]
16:47 <bd808@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
16:43 <brennen> 1.38.0-wmf.3 train (T281167): unblocked, rolling to group0 [production]
16:41 <bd808@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
16:35 <brennen@deploy1002> Synchronized php-1.38.0-wmf.3/extensions/Scribunto/includes/engines/LuaCommon/LanguageLibrary.php: Backport: [[gerrit:726596|Replace deprecated ParserOptions::getUser with ::getUserIdentity (T292589)]] (duration: 01m 04s) [production]
16:35 <jynus> stopping db1127 for hw maintenance T292366 [production]
16:31 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: hw maintenance [production]
16:31 <jynus@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: hw maintenance [production]
16:28 <brennen@deploy1002> Synchronized php-1.38.0-wmf.2/extensions/Scribunto/includes/engines/LuaCommon/LanguageLibrary.php: Backport: [[gerrit:726597|Replace deprecated ParserOptions::getUser with ::getUserIdentity (T292589)]] (duration: 01m 10s) [production]
16:22 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:11 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:01 <volans@cumin2002> END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host sretest1002.eqiad.wmnet [production]