|
2015-07-09
§
|
| 14:44 |
<gwicke> |
re-enabled compaction throttling (60mb/s) on cassandra nodes |
[production] |
| 14:44 |
<bblack> |
reprepro: jessie-wikimedia/backports openssl pkg, 1.0.2c-1 => 1.0.2d-1~wmf1 |
[production] |
| 14:29 |
<_joe_> |
reimaging mw1152 for wiping any leftover local hacks. Depooling, scheduling downtime |
[production] |
| 14:28 |
<moritzm> |
installed python-django security updates on labmon, netmon and californium |
[production] |
| 14:24 |
<godog> |
really upgrade python-django on graphite2001 |
[production] |
| 13:48 |
<mobrovac> |
restbase cassandra rolling restart to apply https://gerrit.wikimedia.org/r/223774 |
[production] |
| 13:02 |
<godog> |
upgrade python-django on graphite1001 and graphite2001 following http://www.ubuntu.com/usn/usn-2671-1/ |
[production] |
| 11:34 |
<godog> |
restart cassandra on restbase1001 |
[production] |
| 11:22 |
<krinkle> |
Synchronized php-1.26wmf13/resources/src/mediawiki/mediawiki.util.js: T105265 (duration: 00m 11s) |
[production] |
| 11:21 |
<krinkle> |
Synchronized php-1.26wmf13/includes/GlobalFunctions.php: T105265 (duration: 00m 12s) |
[production] |
| 11:09 |
<mobrovac> |
restbase deploying https://gerrit.wikimedia.org/r/#/c/223297/ which bumps the back-end module version ( https://github.com/wikimedia/restbase-mod-table-cassandra/pull/117 ) |
[production] |
| 10:53 |
<mobrovac> |
restbase started thinner 15 days for wikimedia group |
[production] |
| 10:38 |
<mark> |
Shutdown AMS-IX route server BGP sessions on cr1-esams |
[production] |
| 07:48 |
<oblivian> |
Synchronized php-1.26wmf13/thumb.php: Re-add fix for thumb.php 404s on HHVM (duration: 00m 13s) |
[production] |
| 06:27 |
<twentyafterfour> |
restarted apache2 on iridium to fix phab exception |
[production] |
| 06:15 |
<springle> |
db1037 is repartitioning tables; it will lag intermittently for a day |
[production] |
| 06:05 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jul 9 06:05:30 UTC 2015 (duration 5m 29s) |
[production] |
| 05:23 |
<gwicke> |
dynamically limited cassandra compaction throughput to 80mb/s; please review https://gerrit.wikimedia.org/r/#/c/223722/ to make this permanent |
[production] |
| 03:01 |
<LocalisationUpdate> |
completed (1.26wmf13) at 2015-07-09 03:01:13+00:00 |
[production] |
| 02:58 |
<l10nupdate> |
Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 05m 29s) |
[production] |
| 02:42 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-09 02:42:56+00:00 |
[production] |
| 02:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jul 9 02:40:16 UTC 2015 (duration 40m 15s) |
[production] |
| 02:36 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 10m 32s) |
[production] |
| 02:28 |
<twentyafterfour> |
restarted phd |
[production] |
| 02:28 |
<twentyafterfour> |
moved phd log to free disk space on iridium |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.26wmf13) at 2015-07-09 02:24:00+00:00 |
[production] |
| 02:23 |
<l10nupdate> |
Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 00m 34s) |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-09 02:17:02+00:00 |
[production] |
| 02:16 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 00m 47s) |
[production] |
| 02:00 |
<springle> |
pkg upgrade and restart db1037 |
[production] |
| 01:49 |
<gwicke> |
switched remaining cassandra nodes to JDK8 |
[production] |
| 01:37 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1037 (duration: 00m 11s) |
[production] |
| 01:07 |
<mutante> |
uranium - deleted apache logs older than 90 days |
[production] |
| 00:45 |
<RoanKattouw> |
Running populateContentModel.php --wiki=cawiki --table=revision --ns=5 |
[production] |
| 00:20 |
<RoanKattouw> |
Ran populateContentModel.php --table=revision for odd-numbered namespaces on officewiki for T105245 |
[production] |
|
2015-07-08
§
|
| 23:07 |
<catrope> |
Synchronized php-1.26wmf13/extensions/Flow: SWAT (duration: 00m 14s) |
[production] |
| 23:06 |
<bd808> |
Restarted logstash on logstash1001; no hhvm input seen for last hour |
[production] |
| 22:56 |
<gwicke> |
finished rolling restart of cassandra cluster to apply https://gerrit.wikimedia.org/r/#/c/223495/ |
[production] |
| 22:45 |
<mutante> |
zirconium - stop puppet for role switch |
[production] |
| 22:33 |
<legoktm> |
Synchronized php-1.26wmf13/includes/changes/EnhancedChangesList.php: Unbreak missing flags in enhanced RC (duration: 00m 12s) |
[production] |
| 22:08 |
<hoo> |
Synchronized php-1.26wmf13/extensions/Wikidata/: Update Wikibase: Fix JavaScript ULS usage (duration: 00m 20s) |
[production] |
| 21:51 |
<manybubbles> |
Synchronized php-1.26wmf12/extensions/CirrusSearch/: Stop some fatals in cirrus (duration: 00m 13s) |
[production] |
| 21:41 |
<bd808> |
Synchronized php-1.26wmf13/includes/api/ApiMain.php: Revert Count API module instantiations and Hook runs (2/2) (duration: 00m 12s) |
[production] |
| 21:40 |
<bd808> |
Synchronized php-1.26wmf13/includes/Hooks.php: Revert Count API module instantiations and Hook runs (1/2) (duration: 00m 12s) |
[production] |
| 21:39 |
<bd808> |
Synchronized php-1.26wmf13/extensions/CirrusSearch/includes/CirrusSearch.php: Suppress interwiki results when they would break (duration: 00m 12s) |
[production] |
| 21:08 |
<bblack> |
graphite: wiped /var/log/upstart/statsite* logs, restarted statsite processes |
[production] |
| 20:56 |
<csteipp> |
deployed patches for T103022 & T103023 |
[production] |
| 20:53 |
<csteipp> |
deployed patch for T94116 for wmf12/wmf13 |
[production] |
| 20:30 |
<gwicke> |
added explicit exit 1 in /etc/init.d/cassandra on restbase1008 to prevent cassandra from starting up there; is puppet restarting it? |
[production] |
| 20:29 |
<subbu> |
deployed parsoid sha c4cfc527 |
[production] |