|
2013-05-02
§
|
| 23:20 |
<mutante> |
restarting lucene on search1021, search1022, search1017, search1018 (with some waiting in between) |
[production] |
| 23:17 |
<hashar> |
yeah after hours and hours of fighting, Jenkins is finally working again. |
[production] |
| 23:10 |
<aaron> |
synchronized wmf-config/jobqueue-eqiad.php |
[production] |
| 22:58 |
<hashar> |
restarting jenkins… it got a few threads blocked and the main process is at 100% usage for now reason |
[production] |
| 22:58 |
<mutante> |
repooling search1016 |
[production] |
| 22:40 |
<mutante> |
gallium, run puppet, graceful Apache to deploy split log files |
[production] |
| 22:34 |
<asher> |
synchronized wmf-config/db-eqiad.php 'pulling db1020 for upgrade' |
[production] |
| 22:31 |
<mutante> |
depooling search1016, restarting lucene, etc.. (Search#Adding_new_wikis) |
[production] |
| 22:29 |
<mutante> |
repooling search1015 |
[production] |
| 22:22 |
<hashar> |
Zuul restarted. The bug about slowness is {{bug|48025}} |
[production] |
| 22:15 |
<hashar> |
restarting Zuul |
[production] |
| 22:13 |
<hashar> |
Deploying a workaround on Zuul to make it stop querying the Jenkins API when it just want to check whether a job exist. {{gerrit|62095}} |
[production] |
| 22:04 |
<mutante> |
depooling search1015 in pybal |
[production] |
| 21:43 |
<mflaschen> |
synchronized php-1.22wmf3/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf3 for E3 deployment' |
[production] |
| 21:41 |
<mflaschen> |
synchronized php-1.22wmf2/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf2 for E3 deployment' |
[production] |
| 21:38 |
<mutante> |
importing wikimania2014wiki into search indexers |
[production] |
| 21:34 |
<mutante> |
restarting search indexers on searchidx2, searchidx1001 to make sure the indexer knows about new wiki |
[production] |
| 21:27 |
<mflaschen> |
synchronized php-1.22wmf3/skins/common/shared.css 'Sync font-size change for edit section links' |
[production] |
| 21:15 |
<spage> |
synchronized php-1.22wmf2/extensions/ConfirmEdit 'update 1.22wmf2 to wmf3 version of ConfirmEdit' |
[production] |
| 21:13 |
<LeslieCarr> |
adding GEANT via fiberring to avoid-paths |
[production] |
| 20:20 |
<reedy> |
synchronized php-1.22wmf3/includes/ |
[production] |
| 20:15 |
<reedy> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 19:51 |
<Jeff_Green> |
authdns-update to move db1025 to frack.eqiad.wmnet |
[production] |
| 19:46 |
<reedy> |
synchronized wmf-config/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 19:39 |
<ottomata> |
rebooting oxygen |
[production] |
| 19:32 |
<ottomata> |
re-enabling puppet on cp1031, it was administratively disabled. running puppet there. |
[production] |
| 19:23 |
<RobH> |
professor was already powered down (why?) starting it back up now |
[production] |
| 19:22 |
<ottomata> |
all webrequest udp2log loggers (squid and varnish) now send to gadolinium for socat unicast -> multicast relay |
[production] |
| 19:22 |
<RobH> |
rebooting professor |
[production] |
| 19:16 |
<Jeff_Green> |
moving db1025 into frack-fundraising1-c-eqiad |
[production] |
| 19:13 |
<kaldari> |
synchronized php-1.22wmf2/extensions/Echo 'sync Echo ext for en.wiki' |
[production] |
| 19:13 |
<ottomata> |
deployed squid frontend.conf.php changes to remove locke and send logs directly to gadolinium for multicast relay |
[production] |
| 19:11 |
<kaldari> |
synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext' |
[production] |
| 19:08 |
<kaldari> |
synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext' |
[production] |
| 18:51 |
<ottomata> |
removed varnishcsa-locke instance from varnish hosts: (dsh -c -g varnishncsa-all 'test -f /etc/init.d/varnishncsa-locke && service varnishncsa-locke stop && update-rc.d -f varnishncsa-locke remove && rm -v /etc/init.d/varnishncsa-locke') |
[production] |
| 18:46 |
<reedy> |
synchronized wmf-config/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 18:40 |
<ottomata> |
varnishncsa now sends traffic to gadolinium instead of oxygen for multicast relay |
[production] |
| 18:29 |
<Krinkle> |
Installed Monitoring plugin from Jenkins control panel <https://wiki.jenkins-ci.org/display/JENKINS/Monitoring> |
[production] |
| 17:59 |
<Krinkle> |
Jenkins restart complete. No visible improvement. Jenkins is still idling most of the time while Zuul is still halted by an unknown factor on spawning jobs. |
[production] |
| 17:56 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 17:53 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 17:47 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 17:37 |
<Krinkle> |
Jenkins keeps clogging up. Starting an emergency restart. |
[production] |
| 16:59 |
<Krinkle> |
Jenkins is nearing 100% CPU on gallium, what is Jenkins doing? |
[production] |
| 16:59 |
<Krinkle> |
Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating... |
[production] |