|
2016-05-19
ยง
|
| 16:41 |
<thcipriani> |
beta-code-update eqiad hung for past few hours |
[releng] |
| 16:31 |
<volans> |
Set runtime value for max_allowed_packet, innodb_buffer_pool_dump_at_shutdown, innodb_buffer_pool_load_at_startup to their configured values for s1-s7, es1-es3, x1 T133333 |
[production] |
| 16:31 |
<urandom> |
Disabling puppet on xenon.eqiad.wmnet in preparation for Cassandra upgrade : T126629 |
[production] |
| 16:29 |
<elukey> |
upgrading cassandra from 2.1.12 to 2.1.13 on aqs1001.eqiad.mwnet |
[production] |
| 15:48 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.1/extensions/VisualEditor/ApiVisualEditor.php: SWAT: [[gerrit:289586|Debug log strange-looking ETags being sent to RB]] (duration: 00m 29s) |
[production] |
| 15:43 |
<YuviPanda> |
rebooting all tools worker instances |
[tools] |
| 15:41 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: SWAT: [[gerrit:289587|Debug log strange-looking ETags being sent to RB]] (duration: 00m 44s) |
[production] |
| 15:16 |
<hashar> |
Restarted zuul-merger daemons on both gallium and scandium : file descriptors leaked |
[releng] |
| 15:16 |
<hashar> |
Restarted zuul-merger daemons on both gallium and scandium : file descriptors leaked |
[production] |
| 14:20 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1029 (x1) with low weight - T112079 (duration: 00m 40s) |
[production] |
| 14:18 |
<akosiaris> |
enable puppet on maps-test200{2,3,4}. |
[production] |
| 14:02 |
<akosiaris> |
enabled and ran puppet on maps-test2001 |
[production] |
| 13:58 |
<akosiaris> |
disable puppet on maps-test200{1,2,3,4} for enabling cassandra metrics collection selectively |
[production] |
| 13:31 |
<chasemp> |
reboot labstore1003 kernel upgrade |
[production] |
| 13:12 |
<chasemp> |
reboot tools-exec-1220 stuck in state of unresponsivenss |
[tools] |
| 12:54 |
<godog> |
bounce carbon-c-relay on graphite1001, run with debug version |
[production] |
| 12:51 |
<Amir1> |
deploying e42885d to wikilabels |
[wikilabels] |
| 12:45 |
<elukey> |
restarted oozie on analytics1003 for security upgrades |
[production] |
| 12:44 |
<elukey> |
restarted oozie on analytics1003 for security upgrades |
[analytics] |
| 12:32 |
<elukey> |
suspended all the oozie bundles as prep step for oozie's restart (security upgrades) |
[analytics] |
| 12:28 |
<elukey> |
restarted hue on analytics1027 for security upgrades |
[analytics] |
| 12:28 |
<elukey> |
restarted hue on analytics1027 for security upgrades |
[production] |
| 11:59 |
<hashar> |
CI: salt -v '*' cmd.run 'pip install --upgrade pip==8.1.2' |
[releng] |
| 11:54 |
<hashar> |
Upgrading pip on CI slaves from 7.0.1 to 8.1.2 https://gerrit.wikimedia.org/r/#/c/289639/ |
[releng] |
| 11:50 |
<Amir1> |
manually rebooting wikilabels-staging-01.wikilabels.eqiad.wmflabs |
[wikilabels] |
| 11:26 |
<moritzm> |
restarting salt-master on neodymium |
[production] |
| 11:09 |
<ori> |
dropped negative values from mc_get_hits_rate ganglia metrics for eqiad memcached hosts by running https://phabricator.wikimedia.org/P3138 |
[production] |
| 10:49 |
<volans> |
db1029 stop, backup and reimage T112079 |
[production] |
| 10:48 |
<jynus> |
db1033 stop, backup and reimage T134555 |
[production] |
| 10:41 |
<volans> |
Disable puppet on db1029 for reimaging T112079 |
[production] |
| 10:15 |
<hashar> |
puppet broken on deployment-tin : ?[1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter trusted_group on node deployment-tin.deployment-prep.eqiad.wmflabs?[0m |
[releng] |
| 10:03 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1033 (s7 old master) & db1029 (x1-slave) for maintenance (duration: 02m 05s) |
[production] |
| 09:51 |
<moritzm> |
restarting apache2 on pallaium (will impose a few temporary puppet failures) |
[production] |
| 09:46 |
<moritzm> |
restarting apache2 on strontium (will impose a few temporary puppet failures) |
[production] |
| 09:39 |
<hashar> |
Restarting Jenkins |
[production] |
| 09:23 |
<kart_> |
updated cxserver to 4aaec58 |
[production] |
| 09:22 |
<moritzm> |
restarting apache on neon (hosting icinga) for security update |
[production] |
| 09:08 |
<moritzm> |
restarting apache on silver (hosting wikitech) for security update |
[production] |
| 08:35 |
<hashar> |
gallium: purging old Linux kernel packages (~2.2Gbytes) |
[production] |
| 08:27 |
<moritzm> |
restarting apache on ytterbium (hosting gerrit.wikimedia.org) for security update |
[production] |
| 08:06 |
<moritzm> |
rolling restart of hhvm on mediawiki in eqiad to pick up expat security update |
[production] |
| 07:17 |
<jynus> |
performing schema change on s4 T130692 |
[production] |
| 07:04 |
<moritzm> |
installed chromium security updates on osmium |
[production] |
| 06:11 |
<gehel> |
completed rolling restart of Elasticsearch codfw for Java update (T135499) |
[production] |
| 03:10 |
<ejegg|away> |
updated fundraising tools from 220afdeaa36bc3feaaff1f781e7761d7878c4ee8 to b2425aef2154d6b689900f4848cca02880321230 |
[production] |
| 02:47 |
<ejegg|away> |
updated misc fundraising tools from e2978024e6f6b6881d087ac5d07e4c40f7374709 to 220afdeaa36bc3feaaff1f781e7761d7878c4ee8 |
[production] |
| 02:06 |
<ejegg> |
enabled banner history queue consumer |
[production] |
| 02:02 |
<ejegg> |
updated civicrm from 7952ba43a012cb6a2e8d16af19bb13ed520bd56f to b7b46740d701942507dca0a98a75f3f87b6b31b1 |
[production] |
| 01:21 |
<twentyafterfour> |
Phabricator upgrade completed and service restored. |
[production] |
| 01:15 |
<twentyafterfour> |
Phabricator deployment T134443 starting momentarily. Downtime should be minimal but there will be a short interruption while the service restarts. |
[production] |