|
2015-06-05
§
|
| 22:42 |
<godog> |
powercycle graphite2001, no console no ssh |
[production] |
| 22:06 |
<andrewbogott> |
restarted apache on virt1000 |
[production] |
| 20:49 |
<ori> |
Upgrading hhvm-fss on application servers to 1.1.7; expect brief 5xx spike. |
[production] |
| 20:14 |
<demon> |
Synchronized php-1.26wmf8: live hack (duration: 02m 32s) |
[production] |
| 20:10 |
<mutante> |
apt-get upgrade on terbium |
[production] |
| 19:52 |
<godog> |
bounce redis on rdb1001/rdb1003 to pick up new slave limits |
[production] |
| 19:51 |
<mutante> |
chown root:root / on terbium |
[production] |
| 19:50 |
<godog> |
bounce redis on rdb1002/rdb1004 to pick up new slave limits |
[production] |
| 19:29 |
<godog> |
bounce redis again on rdb1003 after increasing the slave limits more |
[production] |
| 19:17 |
<godog> |
bounce redis on rdb1003 after bumping slave limits |
[production] |
| 19:07 |
<godog> |
redis master logs shows periodic 'cmd=sync scheduled to be closed ASAP for overcoming of output buffer limits.' indicating the slave fails to sync |
[production] |
| 18:40 |
<godog> |
spike in redis network starting at ~15.00 UTC, correlates with ocg failures |
[production] |
| 18:01 |
<moritzm> |
restarted gerrit on ytterbium for java update |
[production] |
| 14:43 |
<jynus> |
short lag period on db1049, traffic automatically redirected to other slave and back to normal |
[production] |
| 14:07 |
<moritzm> |
added ubuntu-meta-1.325+wmf1 for trusty-wikimedia to apt.wikimedia.org (T100004) |
[production] |
| 14:07 |
<moritzm> |
added ubuntu-meta-1.267.1+wmf1 for precise-wikimedia to apt.wikimedia.org (T100004) |
[production] |
| 12:44 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Depool es1007 (duration: 01m 08s) |
[production] |
| 12:08 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Repool es1009 (duration: 01m 08s) |
[production] |
| 11:30 |
<_joe_> |
uploaded new HHVM package, installing on mw1025 for testing |
[production] |
| 09:17 |
<moritzm> |
added redis_2.6.13-1+wmf1 to precise-wikimedia on apt.wikimedia.org |
[production] |
| 06:24 |
<moritzm> |
added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org |
[production] |
| 05:24 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jun 5 05:22:50 UTC 2015 (duration 22m 49s) |
[production] |
| 04:10 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 01m 08s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.26wmf8) at 2015-06-05 02:25:20+00:00 |
[production] |
| 02:21 |
<l10nupdate> |
Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 09s) |
[production] |
| 01:27 |
<tgr> |
deploying schema changes for Gather on enwiki, enwikivoyage, hewiki (T98490, T101460) |
[production] |
| 00:08 |
<catrope> |
Synchronized php-1.26wmf8/vendor/oojs/oojs-ui/php/Tag.php: Fix OOUI fatals (T99210) (duration: 00m 13s) |
[production] |
|
2015-06-04
§
|
| 23:40 |
<catrope> |
Synchronized php-1.26wmf8/extensions/MobileFrontend: SWAT (duration: 00m 13s) |
[production] |
| 23:28 |
<catrope> |
Synchronized wmf-config/InitialiseSettings.php: Disable VE A/B test for new accounts on enwiki (duration: 00m 13s) |
[production] |
| 22:39 |
<ejegg> |
updated payments from d22e44e3fab2b937707c2776384cb93a49b4cfd3 to 43c7952d2a31deaea97e8319f5612d644dce43c8 |
[production] |
| 22:21 |
<ottomata> |
doing controlled restart of kafka brokers services to apply auto create topic config |
[production] |
| 21:48 |
<jgage> |
analyics1013 crashed, rebooted |
[production] |
| 21:42 |
<ori> |
Synchronized php-1.26wmf8/includes/libs/ReplacementArray.php: 1b20d62c26: Revert "awful hack: disable fss on zhwiki only, except on mw1017" (duration: 00m 13s) |
[production] |
| 21:34 |
<ori> |
performing rolling restart of HHVMs for hhvm-fss upgrade |
[production] |
| 21:27 |
<bd808> |
restarted logstash and elasticsearch on logstash100[1-3] to pick up latest jre updates |
[production] |
| 18:48 |
<mutante> |
restarted apache on silver/wikitech |
[production] |
| 18:20 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Depool es1009 and master-slave switchover (duration: 00m 13s) |
[production] |
| 18:01 |
<awight> |
Enabling PayPal audit parser job |
[production] |
| 17:57 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Repool es1008 (duration: 00m 15s) |
[production] |
| 17:44 |
<jynus> |
Synchronized wmf-config/db-codfw.php: Repool es2008 and its slaves (duration: 00m 13s) |
[production] |
| 17:21 |
<ori> |
Disabling Puppet and nutcracker on mw1017 to control for parser cache |
[production] |
| 17:18 |
<jynus> |
Synchronized wmf-config/db-codfw.php: Depool es2008 and its slaves (duration: 00m 13s) |
[production] |
| 17:17 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Depool es1008 (duration: 00m 12s) |
[production] |
| 16:33 |
<kartik> |
Finished scap: Update ContentTranslation (duration: 09m 17s) |
[production] |
| 16:24 |
<kartik> |
Started scap: Update ContentTranslation |
[production] |
| 15:54 |
<moritzm> |
added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org |
[production] |
| 15:48 |
<anomie> |
Synchronized php-1.26wmf8/includes/jobqueue/: SWAT: jobqueue: Record stats on how long it takes before a job is run [[gerrit:215748]] (duration: 00m 14s) |
[production] |
| 15:38 |
<anomie> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ApiFeatureUsage everywhere [[gerrit:215901]] (duration: 00m 19s) |
[production] |