|
2016-05-19
§
|
| 17:25 |
<elukey> |
execute sysctl -w net.netfilter.nf_conntrack_max=512000 on kafka1013 as temporary measure (investigating why conntrack count is higher after leader election) - T135557 |
[production] |
| 17:13 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1033 after maintenance with low weight; increase db1029 weight (duration: 00m 29s) |
[production] |
| 16:31 |
<volans> |
Set runtime value for max_allowed_packet, innodb_buffer_pool_dump_at_shutdown, innodb_buffer_pool_load_at_startup to their configured values for s1-s7, es1-es3, x1 T133333 |
[production] |
| 16:31 |
<urandom> |
Disabling puppet on xenon.eqiad.wmnet in preparation for Cassandra upgrade : T126629 |
[production] |
| 16:29 |
<elukey> |
upgrading cassandra from 2.1.12 to 2.1.13 on aqs1001.eqiad.mwnet |
[production] |
| 15:48 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.1/extensions/VisualEditor/ApiVisualEditor.php: SWAT: [[gerrit:289586|Debug log strange-looking ETags being sent to RB]] (duration: 00m 29s) |
[production] |
| 15:41 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: SWAT: [[gerrit:289587|Debug log strange-looking ETags being sent to RB]] (duration: 00m 44s) |
[production] |
| 15:16 |
<hashar> |
Restarted zuul-merger daemons on both gallium and scandium : file descriptors leaked |
[production] |
| 14:20 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1029 (x1) with low weight - T112079 (duration: 00m 40s) |
[production] |
| 14:18 |
<akosiaris> |
enable puppet on maps-test200{2,3,4}. |
[production] |
| 14:02 |
<akosiaris> |
enabled and ran puppet on maps-test2001 |
[production] |
| 13:58 |
<akosiaris> |
disable puppet on maps-test200{1,2,3,4} for enabling cassandra metrics collection selectively |
[production] |
| 13:31 |
<chasemp> |
reboot labstore1003 kernel upgrade |
[production] |
| 12:54 |
<godog> |
bounce carbon-c-relay on graphite1001, run with debug version |
[production] |
| 12:45 |
<elukey> |
restarted oozie on analytics1003 for security upgrades |
[production] |
| 12:28 |
<elukey> |
restarted hue on analytics1027 for security upgrades |
[production] |
| 11:26 |
<moritzm> |
restarting salt-master on neodymium |
[production] |
| 11:09 |
<ori> |
dropped negative values from mc_get_hits_rate ganglia metrics for eqiad memcached hosts by running https://phabricator.wikimedia.org/P3138 |
[production] |
| 10:49 |
<volans> |
db1029 stop, backup and reimage T112079 |
[production] |
| 10:48 |
<jynus> |
db1033 stop, backup and reimage T134555 |
[production] |
| 10:41 |
<volans> |
Disable puppet on db1029 for reimaging T112079 |
[production] |
| 10:03 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1033 (s7 old master) & db1029 (x1-slave) for maintenance (duration: 02m 05s) |
[production] |
| 09:51 |
<moritzm> |
restarting apache2 on pallaium (will impose a few temporary puppet failures) |
[production] |
| 09:46 |
<moritzm> |
restarting apache2 on strontium (will impose a few temporary puppet failures) |
[production] |
| 09:39 |
<hashar> |
Restarting Jenkins |
[production] |
| 09:23 |
<kart_> |
updated cxserver to 4aaec58 |
[production] |
| 09:22 |
<moritzm> |
restarting apache on neon (hosting icinga) for security update |
[production] |
| 09:08 |
<moritzm> |
restarting apache on silver (hosting wikitech) for security update |
[production] |
| 08:35 |
<hashar> |
gallium: purging old Linux kernel packages (~2.2Gbytes) |
[production] |
| 08:27 |
<moritzm> |
restarting apache on ytterbium (hosting gerrit.wikimedia.org) for security update |
[production] |
| 08:06 |
<moritzm> |
rolling restart of hhvm on mediawiki in eqiad to pick up expat security update |
[production] |
| 07:17 |
<jynus> |
performing schema change on s4 T130692 |
[production] |
| 07:04 |
<moritzm> |
installed chromium security updates on osmium |
[production] |
| 06:11 |
<gehel> |
completed rolling restart of Elasticsearch codfw for Java update (T135499) |
[production] |
| 03:10 |
<ejegg|away> |
updated fundraising tools from 220afdeaa36bc3feaaff1f781e7761d7878c4ee8 to b2425aef2154d6b689900f4848cca02880321230 |
[production] |
| 02:47 |
<ejegg|away> |
updated misc fundraising tools from e2978024e6f6b6881d087ac5d07e4c40f7374709 to 220afdeaa36bc3feaaff1f781e7761d7878c4ee8 |
[production] |
| 02:06 |
<ejegg> |
enabled banner history queue consumer |
[production] |
| 02:02 |
<ejegg> |
updated civicrm from 7952ba43a012cb6a2e8d16af19bb13ed520bd56f to b7b46740d701942507dca0a98a75f3f87b6b31b1 |
[production] |
| 01:21 |
<twentyafterfour> |
Phabricator upgrade completed and service restored. |
[production] |
| 01:15 |
<twentyafterfour> |
Phabricator deployment T134443 starting momentarily. Downtime should be minimal but there will be a short interruption while the service restarts. |
[production] |
|
2016-05-18
§
|
| 23:27 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.2/extensions/Echo/: SWAT: fix URLs in notification emails (T135625) (duration: 00m 35s) |
[production] |
| 23:14 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on outreachwiki (T135582) (duration: 00m 31s) |
[production] |
| 22:23 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: second attempt: group1 to 1.28.0-wmf.2 |
[production] |
| 22:19 |
<twentyafterfour@tin> |
Synchronized php-1.28.0-wmf.2/includes/deferred/LinksUpdate.php: deploy https://gerrit.wikimedia.org/r/#/c/289569/ (duration: 00m 29s) |
[production] |
| 22:00 |
<akosiaris> |
restart cassandra on maps-test200{2,3,4} to get logstash changing working |
[production] |
| 21:39 |
<akosiaris> |
enable puppet on aqs1001 aqs1002 aqs1003 aqs1004 aqs1005 aqs1006 maps-test2001 maps-test2002 maps-test2003 maps-test2004 maps2001 maps2002 maps2003 maps2004 |
[production] |
| 21:32 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: roll back group1 to 1.28.0-wmf.1 |
[production] |
| 21:25 |
<ejegg> |
updated DjangoBannerStats from 8172c4176ea2b78df0dccbfa052064f2739c64bd to 220f80ece976aecbfbfbe1cb476d79910eb48ff4 |
[production] |
| 20:41 |
<urandom> |
Renabling puppet on restbase production cluster : T126629 |
[production] |
| 20:27 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.2 |
[production] |