|
2015-03-04
§
|
| 20:29 |
<hoo> |
Manually completed the global rename Gabriel2517 -> WikiGuy2517 (was stuck on WD.o) |
[production] |
| 19:03 |
<twentyafterfour> |
Creating new deployment branch 1.25wmf20 |
[production] |
| 17:34 |
<Krinkle> |
Depooling all new integation-slave12xx and integration-slave14xx instances again |
[releng] |
| 17:34 |
<Krinkle> |
"ImportError: Entry point ('console_scripts', 'tox') not found" on integration-slave12xx instances for operations-puppet-tox-data_admin_lint |
[releng] |
| 17:33 |
<Krinkle> |
"/usr/local/bin/tox: Permission denied" on integration-slave14xx instances |
[releng] |
| 17:11 |
<Krinkle> |
Pooled integration-slave1201, integration-slave1202, integration-slave1203, integration-slave1204 |
[releng] |
| 17:06 |
<Krinkle> |
Pooled integration-slave1402, integration-slave1403, integration-slave1404, integration-slave1405 |
[releng] |
| 16:56 |
<Krinkle> |
Pooled integration-slave1401 |
[releng] |
| 16:26 |
<Krinkle> |
integration-slave14xx are now provisioned and being added to the pool. Old trusty slaves will be depooled later and eventually deleted. |
[releng] |
| 16:09 |
<anomie> |
Synchronized wmf-config: SWAT: Beta-only change: CX: Add wgContentTranslationCampaigns [[gerrit:194265]] (duration: 00m 07s) |
[production] |
| 14:46 |
<anomie> |
Synchronized php-1.25wmf19/extensions/Graph: early SWAT: Update Graph extension to fix IE bug [[gerrit:194326]] (duration: 00m 06s) |
[production] |
| 13:11 |
<bblack> |
depooled amssq31 in esams for reinstall |
[production] |
| 07:24 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Mar 4 07:23:48 UTC 2015 (duration 23m 47s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.25wmf19) at 2015-03-04 02:28:15+00:00 |
[production] |
| 02:28 |
<l10nupdate> |
Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.25wmf18) at 2015-03-04 02:15:10+00:00 |
[production] |
| 02:15 |
<l10nupdate> |
Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 00:44 |
<demon> |
Finished scap: evening swat: centralauth, VE, user.php fix (duration: 30m 19s) |
[production] |
| 00:14 |
<demon> |
Started scap: evening swat: centralauth, VE, user.php fix |
[production] |
| 00:09 |
<gwicke> |
running concurrent test dumps of enwiki and dewiki through xenon |
[production] |
|
2015-03-03
§
|
| 23:39 |
<mutante> |
ran puppet on ruthenium (keeps showing up in icinga but then no issue when you run it) |
[production] |
| 22:56 |
<ori> |
adding four additional txstatsd backends to graphite1001 to cope with load |
[production] |
| 22:00 |
<hashar> |
reboot integration-puppetmaster in case it solves a NFS mount issue |
[releng] |
| 21:49 |
<demon> |
Synchronized docroot/mediawiki/keys/: (no message) (duration: 00m 05s) |
[production] |
| 21:45 |
<gwicke> |
re-initializing cassandra on test hosts xenon, praseodymium and cerium for new test run; expect some downtime |
[production] |
| 20:33 |
<legoktm> |
manually created centralauth.users_to_rename table |
[releng] |
| 20:05 |
<bblack> |
cp10(60|64|65|70),amssq42 repooled in pybal |
[production] |
| 19:30 |
<demon> |
Synchronized multiversion/MWMultiVersion.php: moar debugging (duration: 00m 07s) |
[production] |
| 19:30 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf19 |
[production] |
| 19:05 |
<twentyafterfour> |
Starting the tuesday train deploy ( Group1 wikis to 1.25wmf19 ) |
[production] |
| 18:48 |
<Krinkle> |
New builds run fine, but there's 30 stuck builds occupying executors. Their output is finished and Zuul/Gerrit got the event already, but they won't die. |
[releng] |
| 18:39 |
<bblack> |
depooled cp10(60|65|70),amssq42 temporarily in pybal for reinstalls |
[production] |
| 18:30 |
<Krinkle> |
The stuck builds have a line of "Finished: .." in them, but are still showing a loader spinner on the bottom of their build log 10 minutes later. |
[releng] |
| 18:28 |
<Krinkle> |
Lots of Jenkins builds are stuck half-way executing. No clear cause. Everything seems up. |
[releng] |
| 17:18 |
<Krinkle> |
Reloading Zuul to deploy Icad0a26dc8 and Icac172b16 |
[releng] |
| 17:03 |
<marktraceur> |
Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Enable BounceHandler on non-wikipedias (duration: 00m 05s) |
[production] |
| 16:53 |
<marktraceur> |
Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Remove redundant namespace aliases for Nepali Wikipedia (newiki) (duration: 00m 05s) |
[production] |
| 16:41 |
<marktraceur> |
Synchronized php-1.25wmf19/extensions/Graph/: [SWAT] [wmf19] Update Graph to 25.19 (duration: 00m 07s) |
[production] |
| 16:30 |
<marktraceur> |
Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Add namespace aliases for Nepali Wikipedia (newiki) (duration: 00m 08s) |
[production] |
| 16:22 |
<marktraceur> |
Synchronized wmf-config/InitialiseSettings-labs.php: [SWAT] [config] Kartik forgot to sync this beta config patch for CX (duration: 00m 10s) |
[production] |
| 16:22 |
<marktraceur> |
Synchronized wmf-config/CommonSettings-labs.php: [SWAT] [config] Enable Flickr uploads on betacommons (duration: 00m 06s) |
[production] |
| 15:39 |
<hashar> |
cancelled logrotate update of all jobs since that seems to kill the Jenkins/Zuul gearman connection. Probably because all jobs are registered on each config change. |
[releng] |
| 15:31 |
<hashar> |
updating all jobs in Jenkins based on PS2 of https://gerrit.wikimedia.org/r/194109 |
[releng] |
| 10:56 |
<hashar> |
Created instance i-000008fb with image "ubuntu-14.04-trusty" and hostname i-000008fb.eqiad.wmflabs. |
[releng] |
| 10:52 |
<hashar> |
deleting integration-puppetmaster to recreate it with a new image {bug|T87484} . Will have to reapply I5335ea7cbfba33e84b3ddc6e3dd83a7232b8acfd and I30e5bfeac398e0f88e538c75554439fe82fcc1cf |
[releng] |
| 03:47 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts 05a5593..1e64ed9 |
[releng] |
| 03:31 |
<springle> |
codfw replag coming up, schema changes |
[production] |
| 02:29 |
<andyrussg> |
Synchronized php-1.25wmf18/extensions/CentralNotice/: CenralNotice update (duration: 00m 07s) |
[production] |
| 02:18 |
<andyrussg> |
Synchronized php-1.25wmf19/extensions/CentralNotice/: CenralNotice update (duration: 00m 09s) |
[production] |
| 02:17 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Mar 3 02:16:47 UTC 2015 (duration 16m 46s) |
[production] |