1301-1350 of 10000 results (31ms)
2016-03-03 ยง
14:34 <gehel> elastic1026.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
13:47 <gehel> elastic1025.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
12:48 <gehel> elastic1024.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
12:22 <godog> temporary repool ms-fe1004, apply https://gerrit.wikimedia.org/r/#/c/273431 to test T128081 [production]
12:12 <elukey@tin> Synchronized wmf-config/jobqueue-eqiad.php: Revert - Remove rdb1003 from the Redis JobQueue pool for maintenance (duration: 00m 28s) [production]
12:07 <elukey@tin> Synchronized wmf-config/jobqueue-eqiad.php: Remove rdb1003 from the Redis JobQueue pool for maintenance (duration: 00m 32s) [production]
12:02 <gehel> elastic1023.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
11:55 <_joe_> disabled notifications from the redises IPSEC checks while replication is disabled [production]
11:52 <volans> Migrating data es2006->es2015 and es2008->es2017->es2019 T127330 [production]
11:40 <moritzm> upgrading cp1008 to openssl 1.0.2g [production]
11:21 <volans@tin> Synchronized wmf-config/db-codfw.php: Depool es2005,es2008 to migrate data to es2015,es2017 T127330 (duration: 00m 53s) [production]
10:54 <godog> replicate swift unsharded -deleted containers eqiad -> codfw T128096 [production]
10:48 <volans> Changing local replica topology for shard es3 in codfw for T127330 [production]
10:36 <volans> Changing local replica topology for shard es2 in codfw for T127330 [production]
10:21 <_joe_> rolling restart of strongswan on eqiad failing servers [production]
10:17 <_joe_> restarted strongswan on mc1011 [production]
10:07 <gehel> elastic1022.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
09:57 <volans> Added es2014,es2016,es2018 to tendril [ T127330 ] [production]
09:46 <jynus> schema change finished on all hosts (except delayed slaves) [production]
09:21 <_joe_> puppet re-enabled everywhere, now troubleshooting ipsec issues [production]
08:59 <moritzm> repooled scb1002 [production]
08:49 <moritzm> repooled scb1001, depooling scb1002 for nodejs upgrade [production]
08:35 <_joe_> disabled puppet across the main redises fleet in order to merge https://gerrit.wikimedia.org/r/271261 safely [production]
08:33 <moritzm> depooling scb1001 for nodejs upgrade [production]
08:27 <jynus> altering heartbeat table on all production servers [production]
06:39 <ebernhardson> upgrade elastic1021.eqiad.wmnet to elasticsearch 1.7.5 [production]
05:47 <ebernhardson> upgrade elastic1020.eqiad.wmnet to elasticsearch 1.7.5 [production]
05:02 <ebernhardson> upgrade elastic1019.eqiad.wmnet to elasticseach 1.7.5 [production]
04:05 <bblack> disabling puppet on caches for a bit, JIC [production]
03:51 <ebernhardson> upgrade elastic1018.eqiad.wmnet to elasticsearch 1.7.5 [production]
03:13 <l10nupdate@tin> ResourceLoader cache refresh completed at Thu Mar 3 03:13:23 UTC 2016 (duration 8m 38s) [production]
03:04 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.15) (duration: 18m 46s) [production]
03:03 <ebernhardson> upgrade elastic1017.eqiad.wmnet to elasticsearch 1.7.5 [production]
02:28 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.14) (duration: 13m 44s) [production]
02:18 <ebernhardson> upgrade elastic1016.eqiad.wmnet to elasticserach 1.7.5 [production]
02:03 <bd808> Events flowing into logstash elasticsearch cluster again after forcing allocation of missing shard replica [production]
01:59 <twentyafterfour> puppet ran on iridium, no errors. :) [production]
01:54 <bd808> Deleted logstash-2016.02.03 index to free disk space [production]
01:51 <bd808> New index not being created due to low disk watermark exceeded on logstash1006 [production]
01:49 <bd808> Logstash elasticsearch cluster not responsive; investigating [production]
01:48 <ebernhardson> upgrade elastic1015.eqiad.wmnet to elasticsearch 1.7.5 [production]
01:44 <twentyafterfour> phabricator is back online [production]
01:21 <twentyafterfour> manually installed scap package on iridium, will fix in puppet immediately after maintenance is finished [production]
01:18 <mutante> elastic1013 "dpkg reports broken packages " [production]
01:17 <twentyafterfour> puppet says "Provider scap3 is not functional on this host" [production]
01:16 <twentyafterfour> testing puppet on iridium [production]
01:07 <mutante> iridium - stop apache [production]
01:00 <ebernhardson> upgrade elastic1014.eqiad.wmnet to elasticsearch 1.7.5 [production]
00:50 <twentyafterfour> Phabricator will be going down for maintenance around 01:00 UTC (Approximately 10 minutes from now) [production]
00:36 <hoo@tin> Synchronized php-1.27.0-wmf.15/extensions/TemplateData/: Change default format to null instead of 'inline' (duration: 01m 02s) [production]