|
2015-04-24
§
|
| 15:08 |
<andrewbogott> |
dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1005, rebooting |
[production] |
| 14:24 |
<andrewbogott> |
dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1006, rebooting |
[production] |
| 10:31 |
<akosiaris> |
nova migrated a couple of etcd's project VMs |
[production] |
| 09:09 |
<_joe_> |
parsoid restart done |
[production] |
| 08:59 |
<_joe_> |
restarting parsoid cluster-wide |
[production] |
| 08:47 |
<ori> |
deployed parsoid/deploy 8b5de6aba / I4d55f6d50: Bump src to d2135c6b69 for deploy |
[production] |
| 08:09 |
<tstarling> |
Synchronized php-1.26wmf2/includes/filerepo/file/LocalFile.php: reverting live hack (duration: 00m 16s) |
[production] |
| 06:40 |
<ori> |
nuked http://commons.wikimedia.org/wiki/User:Niteshift/MVneu/2015_April_21-30 |
[production] |
| 05:44 |
<ori> |
Synchronized php-1.26wmf1/includes/filerepo/file/LocalFile.php: Undo local hack on version that is inactive (1.26wmf1). No-op. (duration: 00m 17s) |
[production] |
| 05:35 |
<ori> |
restart hhvm on mw1222; locked up in pthread_cond_wait, backtrace: https://phabricator.wikimedia.org/P552 |
[production] |
| 05:28 |
<ori> |
nuked https://commons.wikimedia.org/wiki/User:Niteshift/MVneu/2015_April_21-20 |
[production] |
| 05:18 |
<ori> |
Synchronized wmf-config/InitialiseSettings.php: $wgExportAllowHistory default false, $wgExportMaxHistory default 1000 -> 10 (duration: 00m 16s) |
[production] |
| 05:05 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Apr 24 05:04:42 UTC 2015 (duration 4m 41s) |
[production] |
| 04:47 |
<ori> |
Synchronized php-1.26wmf2/includes/filerepo/file/LocalFile.php: Short-circuit LocalFile::loadExtraFromDB in attempt to mitigate outage (duration: 00m 12s) |
[production] |
| 04:42 |
<springle> |
killing LocalFile::loadExtraFromDB wholesale on s4 |
[production] |
| 04:32 |
<ori> |
Synchronized php-1.26wmf1/includes/filerepo/file/LocalFile.php: Short-circuit LocalFile::loadExtraFromDB in attempt to mitigate outage (duration: 00m 14s) |
[production] |
| 04:25 |
<ori> |
Did a cluster-wide 'service hhvm restart'. |
[production] |
| 02:48 |
<LocalisationUpdate> |
completed (1.26wmf3) at 2015-04-24 02:47:12+00:00 |
[production] |
| 02:44 |
<l10nupdate> |
Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 06m 00s) |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.26wmf2) at 2015-04-24 02:28:58+00:00 |
[production] |
| 02:25 |
<l10nupdate> |
Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 06m 35s) |
[production] |
| 00:47 |
<catrope> |
Synchronized wmf-config/CommonSettings.php: Revert RESTbase URL change (duration: 00m 13s) |
[production] |
| 00:21 |
<catrope> |
Synchronized php-1.26wmf3/extensions/VisualEditor: Fix RESTbase revid bug (duration: 00m 18s) |
[production] |
| 00:21 |
<catrope> |
Synchronized php-1.26wmf2/extensions/VisualEditor: Fix RESTbase revid bug (duration: 00m 17s) |
[production] |
| 00:07 |
<catrope> |
Synchronized wmf-config/CommonSettings.php: Use same-domain entry point for RESTbase (duration: 00m 13s) |
[production] |
| 00:07 |
<catrope> |
Synchronized wmf-config/InitialiseSettings.php: Temp disable direct RESTbase on enwiki (duration: 00m 17s) |
[production] |
|
2015-04-23
§
|
| 23:54 |
<rmoen> |
Synchronized php-1.26wmf3/extensions/Flow: Bump flow for cherry-pick (duration: 00m 23s) |
[production] |
| 23:46 |
<mattflaschen> |
Synchronized wmf-config/CommonSettings.php: Bump Flow cache version to 4.7 (1e28cf78e64eb860d6eade775abae43d11c1dd75) (duration: 00m 16s) |
[production] |
| 23:41 |
<andrewbogott> |
updating labvirt1002 to 3.13.0-49-generic, dist-upgrade, rebooting |
[production] |
| 23:41 |
<andrewbogott> |
reverted labvirt1001 to 3.13.0-49-generic because 3.16 wouldn’t mount the fs |
[production] |
| 23:31 |
<rmoen> |
Synchronized php-1.26wmf2/extensions/MobileFrontend/: Update MobileFrontend to cherry picks (duration: 00m 20s) |
[production] |
| 23:30 |
<rmoen> |
Synchronized php-1.26wmf3/extensions/MobileFrontend/: Update MobileFrontend to cherry picks (duration: 00m 38s) |
[production] |
| 22:48 |
<andrewbogott> |
upgrading labvirt1001 to linux-image-3.16.0-34-generic, dist-upgrading, and rebooting |
[production] |
| 22:10 |
<bd808> |
Synchronized wmf-config/logging.php: logstash: Fix log level detection (c09014d) (duration: 00m 17s) |
[production] |
| 21:56 |
<ori> |
Additional (planned) outcome of Ie22658727 and Ice65e7e70: xff log flowing to fluorine, causing bytes-in to climb from ~1.2M/s to ~2.1M/s |
[production] |
| 21:54 |
<ori> |
Syncing Ie22658727 and Ice65e7e70 (which introduce new InitialiseSettings vars) in one go caused a small burst of 500s (peaking at 500/sec and lasting a few seconds) on four app servers. |
[production] |
| 21:42 |
<ori> |
Synchronized wmf-config: Ie22658727 and Ice65e7e70: use Monolog to configure logging (duration: 00m 15s) |
[production] |
| 21:04 |
<awight> |
update payments from 88b9f621bfee1de14a8cdef556a90e5567721754 to 83d09e09178c634ad35dbb684d1c3aebbb709969 |
[production] |
| 19:31 |
<mutante> |
restarting icinga-wm for config change |
[production] |
| 18:05 |
<andrewbogott> |
rebooting labvirt1006 |
[production] |
| 17:51 |
<kartik> |
Synchronized php-1.26wmf2/extensions/ContentTranslation: (no message) (duration: 00m 15s) |
[production] |
| 17:29 |
<kartik> |
Finished scap: Update ContentTranslation (duration: 26m 11s) |
[production] |
| 17:28 |
<ori> |
scap stuck on snapshot1004; not accepting mwdeploy key |
[production] |
| 17:03 |
<kartik> |
Started scap: Update ContentTranslation |
[production] |
| 16:53 |
<aaron> |
Synchronized php-1.26wmf2/includes/jobqueue/JobRunner.php: d23777e6832f660984ce4445ab04f98b7ff0d25f (duration: 00m 12s) |
[production] |
| 16:33 |
<andrewbogott> |
rebooting labvirt1005 |
[production] |
| 15:03 |
<manybubbles> |
Synchronized wmf-config/CommonSettings.php: swat: Re-enable Special:SupportedLanguages (duration: 00m 11s) |
[production] |
| 12:29 |
<godog> |
investigating icinga UNKNOWN for hhvm queue/threads |
[production] |
| 09:15 |
<godog> |
restart carbon on graphite1001, replace with carbon-c-relay |
[production] |
| 08:31 |
<godog> |
restart carbon on labmon1001, replace with carbon-c-relay |
[production] |