2351-2400 of 10000 results (56ms)
2018-12-06 §
18:17 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642) [production]
17:31 <moritzm> installing nodejs updates on proton* [production]
17:11 <moritzm> uploaded nodejs 6.11~dfsg-1+wmf5 for jessie-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it) [production]
16:27 <urandom> decommissioning cassandra-a, restbase2001 -- T210843 [production]
16:17 <gehel> shutting down elasticsearch on elastic2001-2024 (second try) - T211023 [production]
15:51 <moritzm> upgrading spamassassin on mx1001/fermium [production]
15:45 <fsero@deploy1001> scap-helm zotero finished [production]
15:45 <fsero@deploy1001> scap-helm zotero cluster codfw completed [production]
15:45 <fsero@deploy1001> scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw] [production]
15:42 <fsero> modifying zotero deploy CLUSTER=codfw scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero - T211322 [production]
15:42 <_joe_> disabling puppet fleet-wide for a change in the role() function [production]
15:38 <moritzm> uploaded nodejs 6.11~dfsg-1+wmf5 for stretch-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it) [production]
15:08 <gehel> restartign new elasticsearch masters on codfw - T211023 [production]
15:05 <gehel> upgrade nginx on wdqs servers [production]
14:59 <elukey@deploy1001> Finished deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 (duration: 00m 09s) [production]
14:59 <elukey@deploy1001> Started deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 [production]
14:46 <moritzm> uploaded nodejs 10.4.0~dfsg-1+wmf2 to apt.wikimedia.org/component/node10 (backports of recent security fixes) [production]
14:16 <moritzm> installing nginx security updates on mw in eqiad [production]
13:46 <moritzm> upgrading spamassassin on mx2001 [production]
12:56 <gehel> depooling and shutting down elasticsearch on elastic2001-2024 - T211023 [production]
12:55 <moritzm> installing nginx updates on mw in codfw [production]
11:59 <moritzm> installing nginx updates on mw canaries [production]
10:59 <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3315 (duration: 00m 47s) [production]
10:56 <volans> disable event handler on Icinga for ms-be2047 MD Raid and MegaRAID checks, it's spamming Phabricator - T209921 [production]
10:56 <banyek> repooling db1096 for schema change - T85757 [production]
10:36 <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3315 (duration: 00m 49s) [production]
10:29 <banyek> depooling db1096 for schema change - T85757 [production]
09:56 <dcausse> elastic@codfw cleanup: deleting wikidatawiki_content_1537469318 index (failed reindex probably) [production]
01:03 <tstarling@deploy1001> Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s) [production]
00:55 <tstarling@deploy1001> Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s) [production]
00:42 <tstarling@deploy1001> Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s) [production]
00:40 <tstarling@deploy1001> Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s) [production]
00:38 <tstarling@deploy1001> Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s) [production]
00:15 <mutante> MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults [production]
00:13 <mutante> re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug [production]
00:10 <urandom> bootstrapping cassandra-c, restbase2018 -- T210843 [production]
2018-12-05 §
23:50 <ejegg> updated payments-wiki from 20595cca97 to b99cd0816e [production]
23:40 <ejegg> re-enabled fundraising queue consumer jobs [production]
23:33 <ejegg> updated fundraising CiviCRM from e757753a46 to 3e5d74f17e [production]
23:32 <ejegg> turned off fundraising queue jobs for base queue consumer logic update [production]
22:43 <jijiki> restarting pdfreder on scb* hosts in eqiad [production]
21:44 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 02m 47s) [production]
21:41 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 [production]
21:41 <mdholloway> mobileapps deployment failed for group default03, rolling back and retrying [production]
21:39 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 18m 46s) [production]
21:39 <arlolra> Updated Parsoid to a6058e3 (T210647, T208360, T205333) [production]
21:37 <urandom> bootstrapping cassandra-b, restbase2018 -- T210843 [production]
21:23 <arlolra@deploy1001> Finished deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3 (duration: 11m 36s) [production]
21:20 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 [production]
21:12 <arlolra@deploy1001> Started deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3 [production]