551-600 of 10000 results (36ms)
2017-05-04 §
08:49 <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Remove db1040 from config files as it will be decommissioned - T164057 (duration: 00m 55s) [production]
08:23 <gehel> restart elasticsearch on relforge for JDK update [production]
07:59 <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Remove tempdb2001 from config files as it will be decommissioned - T161712 (duration: 01m 07s) [production]
07:58 <marostegui@naos> Synchronized wmf-config/db-codfw.php: Remove tempdb2001 from config files as it will be decommissioned - T161712 (duration: 01m 25s) [production]
07:25 <_joe_> restarted cp3043 backend varnish at 7:13 UTC while trying to debug issues [production]
06:58 <moritzm> installing freetype security updates [production]
06:26 <marostegui@naos> Synchronized wmf-config/db-codfw.php: Depool tempdb2001, no longer needed - T161712 (duration: 01m 08s) [production]
06:17 <marostegui> Stop MySQL on tempdb2001 to take a backup and prepare to decomission - T161712 [production]
06:10 <marostegui> Deploy alter table on wikidatawiki.wb_terms - db2066 - T162539 T163548 [production]
06:10 <marostegui@naos> Synchronized wmf-config/db-codfw.php: Depool db2066 - T162539 T163548 (duration: 01m 25s) [production]
06:09 <Dereckson> CentralAuth: Removed MediaWiki 2FA for Alexsh (T164265) [production]
06:03 <marostegui> Deploy alter table on wikidatawiki.wb_terms - dbstore2002 - T162539 T163548 [production]
02:31 <l10nupdate@naos> ResourceLoader cache refresh completed at Thu May 4 02:31:22 UTC 2017 (duration 5m 21s) [production]
02:26 <l10nupdate@naos> scap sync-l10n completed (1.29.0-wmf.21) (duration: 08m 02s) [production]
02:08 <mobrovac@naos> Finished deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3a (duration: 08m 37s) [production]
02:00 <urandom> T160759: lowering tombstone threshold to 1000 on all eqiad nodes [production]
01:59 <mobrovac@naos> Started deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3a [production]
01:58 <mobrovac@naos> Finished deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3 (duration: 03m 29s) [production]
01:54 <mobrovac@naos> Started deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3 [production]
01:51 <mobrovac@naos> Finished deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki (duration: 03m 28s) [production]
01:47 <mobrovac@naos> Started deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki [production]
01:47 <mobrovac@naos> Finished deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki (duration: 04m 12s) [production]
01:42 <mobrovac@naos> Started deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki [production]
01:22 <urandom_> T160759: lowering tombstone_threshold on restbase1013 & restbase1014 [production]
01:09 <urandom_> T160759: starting restbase1012-a [production]
2017-05-03 §
22:59 <RainbowSprinkles> gerrit: Quick restart to pick up logging config change [production]
22:47 <ejegg> updated fundraising tools from 20afe9d20a44a0dbee58f4f82e65ba5689c60de1 to f2522cdabf1741a60b7b60ac8f7ead7afd50b054 [production]
22:23 <ejegg> updated fundraising tools from a1e9342e093a85032255fc1d9904db7df13680b7 to 20afe9d20a44a0dbee58f4f82e65ba5689c60de1 [production]
21:06 <demon@naos> Synchronized README: No-op, forcing co-master sync (duration: 02m 28s) [production]
20:35 <mutante> mw1167 - same as mw1166 (jobrunners) - there was a hhvm[12547]: Fatal error: unknown exception followed by mysql slow query, SELECT MASTER_TID_WAIT... | systemctl restart hhvm recovers it [production]
20:30 <mutante> mw1166 - restart hhvm service (Fatal error: request has exceeded memory limit) [production]
20:13 <urandom> T160759: restoring default tombstone thresholds, restbase10{3,4,6} [production]
19:57 <mutante> mw1287 - also restarting hhvm (with systemctl restart) [production]
19:56 <mutante> mw1287 - restarted crashed apache (proxy_fcgi:error) [production]
19:48 <demon@naos> Finished scap: Cleaning up some unused branches, no-op (duration: 15m 13s) [production]
19:33 <demon@naos> Started scap: Cleaning up some unused branches, no-op [production]
19:32 <demon@naos> Pruned MediaWiki: 1.29.0-wmf.18 (duration: 00m 19s) [production]
19:30 <demon@naos> Pruned MediaWiki: 1.29.0-wmf.20 [keeping static files] (duration: 00m 44s) [production]
19:27 <ppchelko@naos> Finished deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 attempt #2 - checks timeout (duration: 01m 39s) [production]
19:26 <ppchelko@naos> Started deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 attempt #2 - checks timeout [production]
19:25 <ppchelko@naos> Finished deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 (duration: 07m 39s) [production]
19:18 <ppchelko@naos> Started deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 [production]
18:48 <papaul> db2084 - signing puppet certs, salt-key, initial run [production]
18:48 <urandom> T160759: reducing tombstone threshold to 1000, restbase1014 [production]
18:46 <urandom> T160759: reducing tombstone threshold to 1000, restbase1016 [production]
18:39 <urandom> T160759: reducing tombstone threshold to 1000, restbase1013 [production]
18:35 <urandom> restarting restbase1016-c [production]
18:34 <urandom> restarting restbase1013-b [production]
18:00 <bblack> restart cp2005 backend (lag) [production]
17:33 <moritzm> uploaded openjdk-8 u131 to apt.wikimedia.org [production]