|
2015-02-09
§
|
| 23:55 |
<yurik> |
Started scap: syncing ZeroBanner i18n |
[production] |
| 23:54 |
<yurik> |
Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 05s) |
[production] |
| 23:54 |
<yurik> |
Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 07s) |
[production] |
| 23:01 |
<yurik> |
Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s) |
[production] |
| 23:00 |
<yurik> |
Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s) |
[production] |
| 17:46 |
<bd808> |
restarted elasticsearch on logstash1003; OOM |
[production] |
| 16:53 |
<marktraceur> |
Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s) |
[production] |
| 16:48 |
<marktraceur> |
Synchronized php-1.25wmf16/extensions/UploadWizard/: [SWAT] [wmf16] Trying to force UploadWizard to update (duration: 00m 05s) |
[production] |
| 16:48 |
<marktraceur> |
Synchronized php-1.25wmf15/extensions/UploadWizard/: [SWAT] [wmf15] Trying to force UploadWizard to update (duration: 00m 06s) |
[production] |
| 16:47 |
<jgage> |
restarted eventlogging on hafnium (with deploy from master on tin this time) |
[production] |
| 16:38 |
<marktraceur> |
Synchronized php-1.25wmf16/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf16] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 06s) |
[production] |
| 16:37 |
<marktraceur> |
Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s) |
[production] |
| 16:22 |
<jgage> |
restarted eventlogging on hafnium for nuria via ~root/upgrade-eventlogging --no-update |
[production] |
| 16:18 |
<marktraceur> |
Synchronized wmf-config/throttle.php: [SWAT] [config] Add throttle rules for two workshops (duration: 00m 07s) |
[production] |
| 16:14 |
<marktraceur> |
Synchronized wmf-config/: [SWAT] [config] Un-subscribe frequently failing recipients (duration: 00m 05s) |
[production] |
| 16:12 |
<marktraceur> |
Synchronized php-1.25wmf16/extensions/OAuth/: [SWAT] [wmf16] OAuth: Support ListDefinedTags and ChangeTagsListActive hooks (duration: 00m 11s) |
[production] |
| 15:00 |
<cmjohnson1> |
cp1070 down for h/w troubleshooting. Already depooled by bblack |
[production] |
| 11:58 |
<godog> |
bounce mwprof-profiler-to-carbon on tungsten |
[production] |
| 10:47 |
<hoo> |
Manually removed wikidatawiki.wb_changes_dispatch entries for test wikis (test2wiki, testwiki, testwikidata). |
[production] |
| 09:11 |
<gwicke> |
cassandra load testing on xenon, praseodymium and cerium; disk space is tight, might run out on one of those boxes but they are purely test boxes right now, so np |
[production] |
| 05:32 |
<gwicke> |
stopped puppet on cerium, praseodymium & xenon |
[production] |
| 05:31 |
<gwicke> |
manually updated cassandra on cerium, praseodymium & xenon to 2.1.2 (see https://phabricator.wikimedia.org/T88956) |
[production] |
| 03:50 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Feb 9 03:48:57 UTC 2015 (duration 48m 56s) |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.25wmf16) at 2015-02-09 02:12:45+00:00 |
[production] |
| 02:12 |
<l10nupdate> |
Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s) |
[production] |
| 02:12 |
<LocalisationUpdate> |
completed (1.25wmf15) at 2015-02-09 02:11:13+00:00 |
[production] |
| 02:11 |
<l10nupdate> |
Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
|
2015-02-07
§
|
| 15:36 |
<apergos> |
started nginx on daaset1001, it was not running for some reason |
[production] |
| 09:40 |
<bblack> |
depooled cp1070 in pybal |
[production] |
| 09:33 |
<bblack> |
rebooting cp1070 (dead network, dead console) |
[production] |
| 05:10 |
<subbu> |
deployed parsoid hotfiix 8ca7ef40 (cherry-pick of 447a0565) |
[production] |
| 04:48 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Feb 7 04:47:30 UTC 2015 (duration 47m 29s) |
[production] |
| 03:13 |
<gwicke> |
restarting parsoid cluster |
[production] |
| 02:34 |
<LocalisationUpdate> |
completed (1.25wmf16) at 2015-02-07 02:33:09+00:00 |
[production] |
| 02:33 |
<l10nupdate> |
Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.25wmf15) at 2015-02-07 02:19:09+00:00 |
[production] |
| 02:19 |
<l10nupdate> |
Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s) |
[production] |
| 02:11 |
<qchris> |
Ran kafka leader re-election as analytics1021 dropped out of it's partition leader role. |
[production] |
| 01:48 |
<bblack> |
leaving cp1064 (jessie upload eqiad) pooled front+back. it's experimental but looks stable. if upload-related 503 spikes and I'm not around, feel free to depool it. |
[production] |
| 00:18 |
<qchris> |
Manually bumping heap for the Hadoop namenodes and revived them after both of them running out of heap and not coming back. |
[production] |