|
2013-11-25
§
|
| 22:23 |
<Jamesofur> |
upload replacement arbcom election config |
[production] |
| 21:59 |
<ori-l> |
finished updating / restarting eqiad ssls |
[production] |
| 21:37 |
<ori-l> |
Depooling ssl1004 to restart nginx |
[production] |
| 21:22 |
<springle> |
restarting sanitarium mysqld processes, db1053, db1054, db1057 |
[production] |
| 21:06 |
<ori-l> |
Restarting nginx on ssl1008 |
[production] |
| 20:49 |
<ori-l> |
Depooling ssl1008 to test {{Gerrit|96961}} |
[production] |
| 20:43 |
<ori-l> |
Also disabling puppet on cp400* |
[production] |
| 20:42 |
<ori-l> |
Disabling Puppet on ssl* in preparation for merging https://gerrit.wikimedia.org/r/#/c/96961/ |
[production] |
| 20:10 |
<LeslieCarr> |
text-lb.esams, bits-lb.esams and more (amslvs1/3) are now load shared amongst amslvs1 and amslvs3 instead of just amslvs1 |
[production] |
| 18:48 |
<mholmquist> |
synchronized php-1.23wmf5/extensions/UploadWizard/resources/ext.uploadWizard.uploadCampaign.list.css |
[production] |
| 18:46 |
<mholmquist> |
synchronized php-1.23wmf4/extensions/UploadWizard/resources/ext.uploadWizard.uploadCampaign.list.css |
[production] |
| 16:20 |
<hashar> |
jenkins : installed curl on lanthanum.eqiad.wmnet puppet change is {{gerrit|97526}} |
[production] |
| 13:40 |
<hashar> |
jenkins : restarted slave daemon on gallium.wikimedia.org |
[production] |
| 13:38 |
<hashar> |
jenkins : restarted slave daemon on lanthanum.eqiad.wmnet |
[production] |
| 13:38 |
<hashar> |
Jenkins: upgrading openjdk 6 on gallium and lanthanum |
[production] |
| 07:43 |
<springle> |
imported de-debianized mariadb 5.5.34 debs into rerepro on brewster |
[production] |
| 04:45 |
<springle> |
synchronized wmf-config/db-eqiad.php |
[production] |
| 04:10 |
<springle> |
mariadb 5.5.34 live-fire test on db1045 |
[production] |
| 04:09 |
<springle> |
synchronized wmf-config/db-eqiad.php 'warm up db1045' |
[production] |
| 03:58 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1045 for upgrade' |
[production] |
| 03:05 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Nov 25 03:05:19 UTC 2013 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.23wmf5) at Mon Nov 25 02:26:16 UTC 2013 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf4) at Mon Nov 25 02:14:09 UTC 2013 |
[production] |
| 01:41 |
<springle> |
synchronized wmf-config/db-eqiad.php |
[production] |
| 01:20 |
<springle> |
mariadb 5.5.34 live-fire test on db1037 |
[production] |
| 01:15 |
<springle> |
synchronized wmf-config/db-eqiad.php 'warm up db1037' |
[production] |
| 00:26 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1050. depool db1037' |
[production] |
|
2013-11-24
§
|
| 10:04 |
<ori> |
synchronized wmf-config/throttle.php 'Icf844849e: (bug 57345) Raise account creation limit for cawiki GLAM event' |
[production] |
| 10:03 |
<ori> |
updated /a/common to {{Gerrit|Icf844849e}}: (bug 57345) Raise account creation limit for cawiki GLAM event |
[production] |
| 09:58 |
<ori-l> |
(...thereby fixing the global job queue length graph) |
[production] |
| 09:55 |
<ori-l> |
hume: ln -s /usr/local/apache/common-local /a/common (previously clobbered by Iaf19f0e8c) |
[production] |
| 09:21 |
<Nemo_bis> |
global job queue length graph is still dead https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large |
[production] |
| 06:44 |
<paravoid> |
wikitech broken with commit 716887 (Cleanup puppetmaster's apache Listen ports); workarounded manually with /etc/apache2/conf.d/ports-wikitech.conf |
[production] |
| 06:27 |
<paravoid> |
force-rebooting cp3012, kernel deadlocked |
[production] |
| 06:02 |
<ori-l> |
(cp3012) CPU wait, not wio. |
[production] |
| 05:46 |
<ori-l> |
cp3012: kern.log, every 2s: XFS: possible memory allocation deadlock in kmem_alloc |
[production] |
| 05:43 |
<ori-l> |
cp3012: any attempt to access /proc/17722 immediately hangs ssh |
[production] |
| 05:31 |
<ori-l> |
cp3012 wio through the roof; process table filled with '/bin/ps axwwo stat uid pid ppid vsz rss pcpu etime comm args', uid icinga |
[production] |
| 02:50 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Nov 24 02:50:44 UTC 2013 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.23wmf5) at Sun Nov 24 02:20:50 UTC 2013 |
[production] |
| 02:11 |
<LocalisationUpdate> |
completed (1.23wmf4) at Sun Nov 24 02:11:21 UTC 2013 |
[production] |
|
2013-11-23
§
|
| 14:04 |
<springle> |
testing 5.5.34 mariadb.org debs on db60 |
[production] |
| 10:13 |
<apergos> |
...on zirconium |
[production] |
| 10:13 |
<apergos> |
stopped etherpadlite, removed all but the last 5000 lines of its log file, (root running out of space), restarted |
[production] |
| 03:12 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Nov 23 03:12:24 UTC 2013 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.23wmf5) at Sat Nov 23 02:26:18 UTC 2013 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf4) at Sat Nov 23 02:14:03 UTC 2013 |
[production] |
| 00:57 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable OAuth on fishbowl wikis' |
[production] |
| 00:48 |
<reedy> |
synchronized php-1.23wmf5/extensions/Wikibase/client/ 'Fix InfoAction fatal' |
[production] |
| 00:48 |
<reedy> |
synchronized php-1.23wmf5/extensions/MobileFrontend/ 'Fix history fatal' |
[production] |