|
2015-11-10
§
|
| 21:15 |
<mutante> |
scb1001 - mobile endpoint health CRIT |
[production] |
| 21:15 |
<mutante> |
mira unmerged changes in mw-staging |
[production] |
| 21:14 |
<mutante> |
restbase2001: cassandra CQl refused |
[production] |
| 21:14 |
<mutante> |
aqs1001-1003: CRITICAL: Test Get aggregate page views |
[production] |
| 21:12 |
<mutante> |
tungsten - signed new puppet certs, re-installed |
[production] |
| 20:48 |
<moritzm> |
kafka re-enabled on kafka1013 |
[production] |
| 20:41 |
<moritzm> |
stopping kafka broker on kafka1013 to activate ferm |
[production] |
| 20:26 |
<ottomata> |
starting kafka broker on kafka1012 after applying base ferm |
[production] |
| 20:14 |
<ottomata> |
stopping kafka broker on kafka1012 to apply base ferm |
[production] |
| 20:09 |
<moritzm> |
disabled puppet on kafk1012/1013 in preparation of ferm activation |
[production] |
| 20:08 |
<_joe_> |
restarting jobchron across the cluster |
[production] |
| 18:41 |
<ottomata> |
powercycling analytics1031 |
[production] |
| 16:51 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/252231/ (duration: 00m 35s) |
[production] |
| 16:45 |
<krenair@tin> |
Synchronized wmf-config/Wikibase-labs.php: (no message) (duration: 00m 34s) |
[production] |
| 16:44 |
<krenair@tin> |
Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/252204/ (duration: 00m 34s) |
[production] |
| 16:41 |
<krenair@tin> |
Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/250000/ (duration: 00m 34s) |
[production] |
| 16:34 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250471/ (duration: 00m 34s) |
[production] |
| 16:28 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.5/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: https://gerrit.wikimedia.org/r/#/c/252172/ (duration: 00m 35s) |
[production] |
| 16:22 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.5/extensions/Wikidata/Wikidata.php: https://gerrit.wikimedia.org/r/#/c/252227/ (duration: 00m 45s) |
[production] |
| 15:51 |
<mutante> |
restarted krypton |
[production] |
| 15:32 |
<godog> |
upgrade rsync on ms-be1* per T93587 |
[production] |
| 14:02 |
<godog> |
swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 3000 |
[production] |
| 10:00 |
<jynus> |
performing mysql maintenance of db1035 (depooled from production). Replication delay and reboots are expected. |
[production] |
| 09:46 |
<godog> |
swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 2000 |
[production] |
| 09:39 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1035 for maintenance (duration: 00m 36s) |
[production] |
| 03:12 |
<ottomata> |
restarted statsv on hafnium just in case it stopped after kafka1013 broker crash and restart (ping ori :) ) |
[production] |
| 03:10 |
<ottomata> |
kafka preferred-replica-election to bring kafka1013 back as leader for its partitions |
[production] |
| 03:00 |
<mutante> |
started kafka on kafka1013 |
[production] |
| 02:55 |
<mutante> |
tried to start kafka on kafka1018 |
[production] |
| 02:31 |
<mutante> |
switched people.wm.org to new backend. please use "rutherfordium.eqiad.wmnet" instead of terbium now. all files have been rsynced already |
[production] |
| 02:22 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s) |
[production] |
| 00:13 |
<hoo> |
Attached Richardelainechambers@commonswiki to the global account of the same name |
[production] |
| 00:06 |
<bd808@tin> |
Synchronized wmf-config/logging.php: Disable microsecond timestamps on all loggers (bf61bfc) (duration: 00m 34s) |
[production] |
|
2015-11-09
§
|
| 23:55 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Disable cirrus writes to labsearch, the machine cant take the load and some jobs are timing out (duration: 00m 34s) |
[production] |
| 23:13 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable CirrusSearch writes to labsearch for all but enwiki and dewiki (duration: 00m 35s) |
[production] |
| 22:49 |
<ottomata> |
rebooting analytics1032: https://phabricator.wikimedia.org/T118175 |
[production] |
| 21:52 |
<bearND> |
MobileApps deployed sha1 6c63984 |
[production] |
| 19:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1060 after performance patch (duration: 00m 35s) |
[production] |
| 18:42 |
<jynus> |
restarting mysql in db1060 to test new performance configuration |
[production] |
| 17:55 |
<bblack> |
upgrading pybal (1.10 -> 1.12) on lvs200[456].codfw.wmnet |
[production] |
| 16:50 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250454/ (duration: 00m 34s) |
[production] |
| 16:44 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init: https://gerrit.wikimedia.org/r/#/c/251972/ (duration: 00m 34s) |
[production] |
| 16:40 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251674/ (duration: 00m 35s) |
[production] |
| 16:35 |
<krenair@tin> |
Synchronized docroot/wikipedia.org/apple-app-site-association: https://gerrit.wikimedia.org/r/#/c/250897/ (duration: 00m 34s) |
[production] |
| 16:30 |
<matt_flaschen> |
Ran mwscript extensions/Flow/maintenance/FlowUpdateBetaFeaturePreference.php --wiki=wikidatawiki |
[production] |
| 16:24 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251677/ (duration: 00m 35s) |
[production] |
| 16:14 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.5/extensions/Flow/modules/mw.flow.Initializer.js: https://gerrit.wikimedia.org/r/#/c/251560/ (duration: 00m 44s) |
[production] |
| 16:10 |
<_joe_> |
removing old builds from copper |
[production] |
| 15:45 |
<godog> |
reimage graphite1002 |
[production] |
| 15:06 |
<ottomata> |
running kafka preferred-replica-election |
[production] |