|
2013-05-02
ยง
|
| 20:15 |
<reedy> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 19:51 |
<Jeff_Green> |
authdns-update to move db1025 to frack.eqiad.wmnet |
[production] |
| 19:46 |
<reedy> |
synchronized wmf-config/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 19:39 |
<ottomata> |
rebooting oxygen |
[production] |
| 19:32 |
<ottomata> |
re-enabling puppet on cp1031, it was administratively disabled. running puppet there. |
[production] |
| 19:23 |
<RobH> |
professor was already powered down (why?) starting it back up now |
[production] |
| 19:22 |
<ottomata> |
all webrequest udp2log loggers (squid and varnish) now send to gadolinium for socat unicast -> multicast relay |
[production] |
| 19:22 |
<RobH> |
rebooting professor |
[production] |
| 19:16 |
<Jeff_Green> |
moving db1025 into frack-fundraising1-c-eqiad |
[production] |
| 19:13 |
<kaldari> |
synchronized php-1.22wmf2/extensions/Echo 'sync Echo ext for en.wiki' |
[production] |
| 19:13 |
<ottomata> |
deployed squid frontend.conf.php changes to remove locke and send logs directly to gadolinium for multicast relay |
[production] |
| 19:11 |
<kaldari> |
synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext' |
[production] |
| 19:08 |
<kaldari> |
synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext' |
[production] |
| 18:51 |
<ottomata> |
removed varnishcsa-locke instance from varnish hosts: (dsh -c -g varnishncsa-all 'test -f /etc/init.d/varnishncsa-locke && service varnishncsa-locke stop && update-rc.d -f varnishncsa-locke remove && rm -v /etc/init.d/varnishncsa-locke') |
[production] |
| 18:46 |
<reedy> |
synchronized wmf-config/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 18:40 |
<ottomata> |
varnishncsa now sends traffic to gadolinium instead of oxygen for multicast relay |
[production] |
| 18:29 |
<Krinkle> |
Installed Monitoring plugin from Jenkins control panel <https://wiki.jenkins-ci.org/display/JENKINS/Monitoring> |
[production] |
| 17:59 |
<Krinkle> |
Jenkins restart complete. No visible improvement. Jenkins is still idling most of the time while Zuul is still halted by an unknown factor on spawning jobs. |
[production] |
| 17:56 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 17:53 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 17:47 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 17:37 |
<Krinkle> |
Jenkins keeps clogging up. Starting an emergency restart. |
[production] |
| 16:59 |
<Krinkle> |
Jenkins is nearing 100% CPU on gallium, what is Jenkins doing? |
[production] |
| 16:59 |
<Krinkle> |
Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating... |
[production] |
| 16:52 |
<nikerabbit> |
synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master' |
[production] |
| 15:12 |
<hashar> |
Jenkins made Jenkins to instantly restart (was {{bug|47120}}) ) by deleting the downstream-buildview plugin. |
[production] |
| 15:11 |
<hashar> |
restarted Zuul |
[production] |
| 14:11 |
<hashar> |
Upgraded Jenkins from 1.480.3 to LTS 1.509.1. Restarted it. |
[production] |
| 14:09 |
<hashar> |
shutting down Zuul |
[production] |
| 14:08 |
<hashar> |
upgrading Jenkins (unscheduled maintenance). |
[production] |
| 13:01 |
<maxsem> |
synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/61941/' |
[production] |
| 12:58 |
<mark> |
Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams |
[production] |
| 09:54 |
<maxsem> |
synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/' |
[production] |
| 09:53 |
<maxsem> |
synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/' |
[production] |
| 09:50 |
<maxsem> |
synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/' |
[production] |
| 09:39 |
<maxsem> |
synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php 'https://gerrit.wikimedia.org/r/#/c/61962/' |
[production] |
| 06:03 |
<apergos> |
powercycle snapshot1001, upgrade to precise |
[production] |
| 05:45 |
<apergos> |
powercycle snapshot1004, upgrade to precise |
[production] |
| 05:27 |
<apergos> |
snapshot1003 powercucle, upgrading to precise |
[production] |
| 05:27 |
<Tim> |
on professor: manually started carbon-cache.py |
[production] |
| 05:14 |
<Tim> |
professor is down, no response on serial console, rebooting |
[production] |
| 05:11 |
<tstarling> |
synchronized wmf-config/jobqueue-pmtpa.php |
[production] |
| 05:07 |
<tstarling> |
synchronized wmf-config/jobqueue-eqiad.php |
[production] |
| 04:46 |
<Tim> |
on rdb1001-1002, set requirepass and masterauth in soft state |
[production] |
| 04:44 |
<Tim> |
on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state |
[production] |
| 04:39 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu May 2 04:39:48 UTC 2013 |
[production] |
| 04:03 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 03:59 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 03:56 |
<tstarling> |
synchronized wmf-config/jobqueue-eqiad.php |
[production] |
| 03:46 |
<tstarling> |
synchronized php-1.22wmf2/extensions/GettingStarted |
[production] |