|
2016-05-19
ยง
|
| 20:44 |
<urandom> |
restbase-test2003.codfw.wmnet upgraded and online : T126629 |
[production] |
| 20:42 |
<urandom> |
Upgrading Cassandra to 2.2.6 on restbase-test2003.codfw.wmnet : T126629 |
[production] |
| 20:41 |
<urandom> |
Stopping Cassandra on restbase-test2003.codfw.wmnet : T126629 |
[production] |
| 20:38 |
<urandom> |
restbase-test2002.codfw.wmnet upgraded and online : T126629 |
[production] |
| 20:36 |
<krenair@tin> |
Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: slight update to the debug logging to try to find the callers (duration: 00m 31s) |
[production] |
| 20:36 |
<urandom> |
Upgrading Cassandra to 2.2.6 on restbase-test2002.codfw.wmnet : T126629 |
[production] |
| 20:35 |
<urandom> |
Stopping Cassandra on restbase-test2002.codfw.wmnet : T126629 |
[production] |
| 20:35 |
<chasemp> |
iridium try to block vandal by ip temp so puppet disable and edit of /etc/apache2/phabbanlist.conf |
[production] |
| 20:31 |
<urandom> |
restbase-test2001.codfw.wmnet upgraded and online : T126629 |
[production] |
| 20:29 |
<urandom> |
Upgrading Cassandra to 2.2.6 on restbase-test2001.codfw.wmnet : T126629 |
[production] |
| 20:28 |
<urandom> |
Stopping Cassandra on restbase-test2001.codfw.wmnet : T126629 |
[production] |
| 20:19 |
<urandom> |
praseodymium.eqiad.wmnet upgraded and online : T126629 |
[production] |
| 20:16 |
<urandom> |
Upgrading Cassandra to 2.2.6 on praseodymium.eqiad.wmnet : T126629 |
[production] |
| 20:16 |
<aaron@tin> |
Synchronized wmf-config/CommonSettings.php: Lowered to 5 (duration: 00m 29s) |
[production] |
| 20:15 |
<urandom> |
Stopping Cassandra on praseodymium.eqiad.wmnet : T126629 |
[production] |
| 20:00 |
<urandom> |
Upgrading Cassandra on cerium.eqiad.wmnet, and forcing puppet run : T126629 |
[production] |
| 19:59 |
<urandom> |
Stopping Cassanra on cerium.eqiad.wmnet for Cassandra 2.2.6 upgrade : T126629 |
[production] |
| 19:32 |
<urandom> |
Disabling puppet in RESTBase staging : T126629 |
[production] |
| 19:26 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.2 |
[production] |
| 18:38 |
<cscott> |
updated Parsoid to version 67816adf (T100681, T130638) |
[production] |
| 18:27 |
<urandom> |
Enabling puppet on xenon.eqiad.wmnet and forcing run : T126629 |
[production] |
| 18:26 |
<urandom> |
Upgrading Cassandra on xenon.eqiad.wmnet to 2.2.6 : T126629 |
[production] |
| 18:26 |
<cscott> |
synced; restarted Parsoid on wtp1001.eqiad as a canary |
[production] |
| 18:25 |
<urandom> |
Stopping Cassandra on xenon.eqiad.wmnet : T126629 |
[production] |
| 18:23 |
<cscott> |
re-attempting parsoid deploy of 67816adf |
[production] |
| 18:22 |
<cscott> |
to cleanup Parsoid repos ori ran: salt 'wtp*' cmd.run "sed -i -e '/106801025/d' /srv/deployment/parsoid/deploy/src/lib/api/routes.js" |
[production] |
| 18:13 |
<cscott> |
parsoid deploy reverted to parsoid/deploy-sync-20160504-200410 tag (b0d015fa); 21 repos still dirty |
[production] |
| 18:09 |
<cscott> |
starting to revert Parsoid deploy due to unresolved dirty repos |
[production] |
| 18:05 |
<cscott> |
git-deploy of Parsoid failed with "21/44 minions completed checkout" due to dirty repos, root had applied patch during the restbase/changeprop/parsoid outage. |
[production] |
| 17:39 |
<cscott> |
starting Parsoid deploy (of 67816adf) |
[production] |
| 17:25 |
<elukey> |
execute sysctl -w net.netfilter.nf_conntrack_max=512000 on kafka1013 as temporary measure (investigating why conntrack count is higher after leader election) - T135557 |
[production] |
| 17:13 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1033 after maintenance with low weight; increase db1029 weight (duration: 00m 29s) |
[production] |
| 16:31 |
<volans> |
Set runtime value for max_allowed_packet, innodb_buffer_pool_dump_at_shutdown, innodb_buffer_pool_load_at_startup to their configured values for s1-s7, es1-es3, x1 T133333 |
[production] |
| 16:31 |
<urandom> |
Disabling puppet on xenon.eqiad.wmnet in preparation for Cassandra upgrade : T126629 |
[production] |
| 16:29 |
<elukey> |
upgrading cassandra from 2.1.12 to 2.1.13 on aqs1001.eqiad.mwnet |
[production] |
| 15:48 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.1/extensions/VisualEditor/ApiVisualEditor.php: SWAT: [[gerrit:289586|Debug log strange-looking ETags being sent to RB]] (duration: 00m 29s) |
[production] |
| 15:41 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: SWAT: [[gerrit:289587|Debug log strange-looking ETags being sent to RB]] (duration: 00m 44s) |
[production] |
| 15:16 |
<hashar> |
Restarted zuul-merger daemons on both gallium and scandium : file descriptors leaked |
[production] |
| 14:20 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1029 (x1) with low weight - T112079 (duration: 00m 40s) |
[production] |
| 14:18 |
<akosiaris> |
enable puppet on maps-test200{2,3,4}. |
[production] |
| 14:02 |
<akosiaris> |
enabled and ran puppet on maps-test2001 |
[production] |
| 13:58 |
<akosiaris> |
disable puppet on maps-test200{1,2,3,4} for enabling cassandra metrics collection selectively |
[production] |
| 13:31 |
<chasemp> |
reboot labstore1003 kernel upgrade |
[production] |
| 12:54 |
<godog> |
bounce carbon-c-relay on graphite1001, run with debug version |
[production] |
| 12:45 |
<elukey> |
restarted oozie on analytics1003 for security upgrades |
[production] |
| 12:28 |
<elukey> |
restarted hue on analytics1027 for security upgrades |
[production] |
| 11:26 |
<moritzm> |
restarting salt-master on neodymium |
[production] |
| 11:09 |
<ori> |
dropped negative values from mc_get_hits_rate ganglia metrics for eqiad memcached hosts by running https://phabricator.wikimedia.org/P3138 |
[production] |
| 10:49 |
<volans> |
db1029 stop, backup and reimage T112079 |
[production] |
| 10:48 |
<jynus> |
db1033 stop, backup and reimage T134555 |
[production] |