|
2016-03-03
§
|
| 02:03 |
<bd808> |
Events flowing into logstash elasticsearch cluster again after forcing allocation of missing shard replica |
[production] |
| 01:59 |
<twentyafterfour> |
puppet ran on iridium, no errors. :) |
[production] |
| 01:54 |
<bd808> |
Deleted logstash-2016.02.03 index to free disk space |
[production] |
| 01:51 |
<bd808> |
New index not being created due to low disk watermark exceeded on logstash1006 |
[production] |
| 01:49 |
<bd808> |
Logstash elasticsearch cluster not responsive; investigating |
[production] |
| 01:48 |
<ebernhardson> |
upgrade elastic1015.eqiad.wmnet to elasticsearch 1.7.5 |
[production] |
| 01:44 |
<twentyafterfour> |
phabricator is back online |
[production] |
| 01:21 |
<twentyafterfour> |
manually installed scap package on iridium, will fix in puppet immediately after maintenance is finished |
[production] |
| 01:18 |
<mutante> |
elastic1013 "dpkg reports broken packages " |
[production] |
| 01:17 |
<twentyafterfour> |
puppet says "Provider scap3 is not functional on this host" |
[production] |
| 01:16 |
<twentyafterfour> |
testing puppet on iridium |
[production] |
| 01:07 |
<mutante> |
iridium - stop apache |
[production] |
| 01:00 |
<ebernhardson> |
upgrade elastic1014.eqiad.wmnet to elasticsearch 1.7.5 |
[production] |
| 00:50 |
<twentyafterfour> |
Phabricator will be going down for maintenance around 01:00 UTC (Approximately 10 minutes from now) |
[production] |
| 00:36 |
<hoo@tin> |
Synchronized php-1.27.0-wmf.15/extensions/TemplateData/: Change default format to null instead of 'inline' (duration: 01m 02s) |
[production] |
| 00:30 |
<bd808> |
Ran sync-common on mw1025 |
[production] |
| 00:23 |
<hoo> |
Ran sync-common on mw1025, because it apparently didn't pick up recent changes |
[production] |
| 00:20 |
<ebernhardson> |
upgrade elastic1013.eqiad.wmnet to elasticsearch 1.7.5 |
[production] |
| 00:19 |
<hoo> |
Restarted hhvm on mw1025 because of "Cannot access property on non-object in /srv/mediawiki/php-1.27.0-wmf.14/includes/filerepo/LocalRepo.php" |
[production] |
| 00:14 |
<mutante> |
wikitech: delete /a/backup/public/foo and ./bar cruft |
[production] |
| 00:10 |
<hoo@tin> |
Synchronized wmf-config/Wikibase-production.php: Set $wgWikimediaBadgesCommonsCategoryProperty to null on commons (T128661) (duration: 01m 09s) |
[production] |
| 00:00 |
<andrewbogott> |
restarting pdns on labservices1001 |
[production] |
|
2016-03-02
§
|
| 23:51 |
<chasemp> |
ran puppet on elastic1012 manually which started a mystery stopped (crashed?) elastic search |
[production] |
| 22:49 |
<krinkle@tin> |
Synchronized php-1.27.0-wmf.15/includes/api/ApiMain.php: Fix PHP Notice (duration: 01m 17s) |
[production] |
| 22:28 |
<urandom> |
enabling brotli compression on local_group_wikipedia_T_parsoid_html.data in staging, and forcing rewrite of corresponding tables on xenon : T125906 |
[production] |
| 21:12 |
<urandom> |
forcing a major compaction on {local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ,local_group_wikipedia_T_parsoid_html}.data, xenon.eqiad.wmnet : T125906 |
[production] |
| 20:53 |
<bblack> |
repooling cp1048, seems unlikely to recrash (rare kernel bug) |
[production] |
| 20:45 |
<bblack> |
cp1048: depooled in confd, too |
[production] |
| 20:45 |
<bblack> |
cp1048: unresponsive console, powercycled |
[production] |
| 20:35 |
<gehel> |
elastic1011.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 20:11 |
<demon@tin> |
Finished scap: group1 to wmf.15 (duration: 08m 41s) |
[production] |
| 20:02 |
<demon@tin> |
Started scap: group1 to wmf.15 |
[production] |
| 19:21 |
<mobrovac> |
restbase rolling restart for https://gerrit.wikimedia.org/r/274456 T127387 |
[production] |
| 19:07 |
<volans> |
Data transfer completed, started MySQL and replica on es2014,es2016,es2018 [ T127330 ] |
[production] |
| 18:58 |
<gehel> |
elastic1010.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 18:44 |
<apergos> |
rolled back all changes for dataset1001, running with same old precise OS, grrrrr |
[production] |
| 18:06 |
<apergos> |
still slugging away at pxe book with these broadcom netxtreme II nics (dataset1001) |
[production] |
| 17:55 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Repooling external storage DBs in codfw after data was copied: T127330 (duration: 01m 06s) |
[production] |
| 17:44 |
<godog> |
bounce statsdlb on graphite1001 to add 3x statsite instances T105679 |
[production] |
| 17:35 |
<jynus> |
disabling puppet on db1009 (m5-master) to test heartbeat changes |
[production] |
| 17:20 |
<gehel> |
elastic1009.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 16:54 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enabling ShortURL for bnwikisource [[gerrit:273936]] (duration: 01m 04s) |
[production] |
| 16:39 |
<mobrovac> |
restbase deploy end of fb66dbf |
[production] |
| 16:34 |
<gehel> |
elastic1008.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 16:31 |
<mobrovac> |
restbase deploy continue of fb66dbf for the rest of the nodes |
[production] |
| 16:30 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.15/extensions/ContentTranslation/includes/TranslationStorageManager.php: SWAT: Use correct timestamp for updates [[gerrit:274363]] (duration: 00m 59s) |
[production] |
| 16:28 |
<urandom> |
starting post-bootstrap (1009-b) cleanup on restbase100{5,6,9-a}.eqiad.wmnet : T95253 |
[production] |
| 16:25 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.15/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: SWAT: Translator widget: Fix js error if translator does not have recent contributions [[gerrit:274340]] (duration: 01m 05s) |
[production] |
| 16:07 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Do not send Referer from private wikis [[gerrit:274414]] (duration: 01m 18s) |
[production] |
| 15:53 |
<apergos> |
extending maintenance window for dataset1001 by one hour to 5 pm UTC |
[production] |