|
2013-05-02
§
|
| 17:37 |
<Krinkle> |
Jenkins keeps clogging up. Starting an emergency restart. |
[production] |
| 16:59 |
<Krinkle> |
Jenkins is nearing 100% CPU on gallium, what is Jenkins doing? |
[production] |
| 16:59 |
<Krinkle> |
Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating... |
[production] |
| 16:52 |
<nikerabbit> |
synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master' |
[production] |
| 15:12 |
<hashar> |
Jenkins made Jenkins to instantly restart (was {{bug|47120}}) ) by deleting the downstream-buildview plugin. |
[production] |
| 15:11 |
<hashar> |
restarted Zuul |
[production] |
| 14:11 |
<hashar> |
Upgraded Jenkins from 1.480.3 to LTS 1.509.1. Restarted it. |
[production] |
| 14:09 |
<hashar> |
shutting down Zuul |
[production] |
| 14:08 |
<hashar> |
upgrading Jenkins (unscheduled maintenance). |
[production] |
| 13:01 |
<maxsem> |
synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/61941/' |
[production] |
| 12:58 |
<mark> |
Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams |
[production] |
| 09:54 |
<maxsem> |
synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/' |
[production] |
| 09:53 |
<maxsem> |
synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/' |
[production] |
| 09:50 |
<maxsem> |
synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/' |
[production] |
| 09:39 |
<maxsem> |
synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php 'https://gerrit.wikimedia.org/r/#/c/61962/' |
[production] |
| 06:03 |
<apergos> |
powercycle snapshot1001, upgrade to precise |
[production] |
| 05:45 |
<apergos> |
powercycle snapshot1004, upgrade to precise |
[production] |
| 05:27 |
<apergos> |
snapshot1003 powercucle, upgrading to precise |
[production] |
| 05:27 |
<Tim> |
on professor: manually started carbon-cache.py |
[production] |
| 05:14 |
<Tim> |
professor is down, no response on serial console, rebooting |
[production] |
| 05:11 |
<tstarling> |
synchronized wmf-config/jobqueue-pmtpa.php |
[production] |
| 05:07 |
<tstarling> |
synchronized wmf-config/jobqueue-eqiad.php |
[production] |
| 04:46 |
<Tim> |
on rdb1001-1002, set requirepass and masterauth in soft state |
[production] |
| 04:44 |
<Tim> |
on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state |
[production] |
| 04:39 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu May 2 04:39:48 UTC 2013 |
[production] |
| 04:03 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 03:59 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 03:56 |
<tstarling> |
synchronized wmf-config/jobqueue-eqiad.php |
[production] |
| 03:46 |
<tstarling> |
synchronized php-1.22wmf2/extensions/GettingStarted |
[production] |
| 03:39 |
<tstarling> |
synchronized php-1.22wmf3/extensions/GettingStarted |
[production] |
| 02:45 |
<kaldari> |
Finished syncing Wikimedia installation... : |
[production] |
| 02:32 |
<LocalisationUpdate> |
completed (1.22wmf2) at Thu May 2 02:32:10 UTC 2013 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.22wmf3) at Thu May 2 02:21:30 UTC 2013 |
[production] |
| 02:20 |
<Tim> |
on wikibugs-l: disabled bounce processing and re-enabled mail delivery to wikibugs-irc (was disabled due to excessive bounces) |
[production] |
| 02:20 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
| 01:20 |
<mwalker> |
synchronized php-1.22wmf3/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice' |
[production] |
| 01:17 |
<mwalker> |
synchronized php-1.22wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice' |
[production] |
| 00:15 |
<mutante> |
@stafford:~# puppetstoredconfigclean.rb db10.pmtpa.wmnet |
[production] |
|
2013-05-01
§
|
| 23:28 |
<tstarling> |
synchronized php-1.22wmf3/extensions/GettingStarted |
[production] |
| 23:23 |
<tstarling> |
synchronized php-1.22wmf2/extensions/GettingStarted |
[production] |
| 23:18 |
<tstarling> |
synchronized php-1.22wmf2/extensions/GettingStarted |
[production] |
| 22:42 |
<mutante> |
powercycling frozen mw1041 |
[production] |
| 22:13 |
<mwalker> |
Finished syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring |
[production] |
| 22:06 |
<dzahn> |
synchronized ./multiversion/MWMultiVersion.php |
[production] |
| 22:02 |
<dzahn> |
synchronized docroot |
[production] |
| 21:43 |
<mwalker> |
Started syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring |
[production] |
| 21:32 |
<dzahn> |
gracefulled all apaches |
[production] |
| 21:25 |
<ottomata> |
syncing new redirects.conf to apachces to redirect wikipediazero.org |
[production] |
| 21:13 |
<Jeff_Green> |
stopping mysql on db78 for cloning |
[production] |
| 20:49 |
<ottomata> |
adding new wikipediazero.org zone alias, running authdns-update on dobson |
[production] |