|
2013-11-24
§
|
| 09:21 |
<Nemo_bis> |
global job queue length graph is still dead https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large |
[production] |
| 06:44 |
<paravoid> |
wikitech broken with commit 716887 (Cleanup puppetmaster's apache Listen ports); workarounded manually with /etc/apache2/conf.d/ports-wikitech.conf |
[production] |
| 06:27 |
<paravoid> |
force-rebooting cp3012, kernel deadlocked |
[production] |
| 06:02 |
<ori-l> |
(cp3012) CPU wait, not wio. |
[production] |
| 05:46 |
<ori-l> |
cp3012: kern.log, every 2s: XFS: possible memory allocation deadlock in kmem_alloc |
[production] |
| 05:43 |
<ori-l> |
cp3012: any attempt to access /proc/17722 immediately hangs ssh |
[production] |
| 05:31 |
<ori-l> |
cp3012 wio through the roof; process table filled with '/bin/ps axwwo stat uid pid ppid vsz rss pcpu etime comm args', uid icinga |
[production] |
| 02:50 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Nov 24 02:50:44 UTC 2013 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.23wmf5) at Sun Nov 24 02:20:50 UTC 2013 |
[production] |
| 02:11 |
<LocalisationUpdate> |
completed (1.23wmf4) at Sun Nov 24 02:11:21 UTC 2013 |
[production] |
|
2013-11-23
§
|
| 14:04 |
<springle> |
testing 5.5.34 mariadb.org debs on db60 |
[production] |
| 10:13 |
<apergos> |
...on zirconium |
[production] |
| 10:13 |
<apergos> |
stopped etherpadlite, removed all but the last 5000 lines of its log file, (root running out of space), restarted |
[production] |
| 03:12 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Nov 23 03:12:24 UTC 2013 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.23wmf5) at Sat Nov 23 02:26:18 UTC 2013 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf4) at Sat Nov 23 02:14:03 UTC 2013 |
[production] |
| 00:57 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable OAuth on fishbowl wikis' |
[production] |
| 00:48 |
<reedy> |
synchronized php-1.23wmf5/extensions/Wikibase/client/ 'Fix InfoAction fatal' |
[production] |
| 00:48 |
<reedy> |
synchronized php-1.23wmf5/extensions/MobileFrontend/ 'Fix history fatal' |
[production] |
| 00:47 |
<reedy> |
synchronized php-1.23wmf4/extensions/MobileFrontend/ 'Fix history fatal' |
[production] |
| 00:34 |
<aschulz|laptop> |
Manually fixed img_metadata for File:Zentralbibliothek_Zürich_-_Heinrich_Bullingers_Westerhemd_-_000012135.tif by forcing /bin/identify usage |
[production] |
| 00:22 |
<ori> |
synchronized php-1.23wmf4/extensions/PagedTiffHandler 'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359' |
[production] |
| 00:18 |
<ori> |
synchronized php-1.23wmf5/extensions/PagedTiffHandler 'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359' |
[production] |
|
2013-11-22
§
|
| 21:35 |
<ori> |
synchronized wmf-config/CommonSettings.php 'I615e2b210: correct schema ID of Schema:Echo' |
[production] |
| 21:34 |
<ori> |
updated /a/common to {{Gerrit|I615e2b210}}: Schema:Echo has outdated revision id |
[production] |
| 21:09 |
<mutante> |
mediawiki 1.22.0rc3 is out http://dumps.wikimedia.org/mediawiki/1.22/ |
[production] |
| 20:04 |
<Jeff_Green> |
dist-upgrade aluminium and reboot |
[production] |
| 19:57 |
<mutante> |
starting apache on alumunium |
[production] |
| 19:54 |
<mutante> |
killed all processes by user sahar on aluminium |
[production] |
| 19:27 |
<reedy> |
finished scap: Rebuild localisation cache to update ce |
[production] |
| 19:10 |
<reedy> |
started scap: Rebuild localisation cache to update ce |
[production] |
| 17:21 |
<reedy> |
synchronized php-1.23wmf5/extensions/EducationProgram/includes/Events/EditEventCreator.php |
[production] |
| 17:20 |
<reedy> |
synchronized php-1.23wmf4/extensions/EducationProgram/includes/Events/EditEventCreator.php |
[production] |
| 17:10 |
<reedy> |
synchronized php-1.23wmf5/languages/messages/MessagesCe.php 'Update ce language special page aliases' |
[production] |
| 17:09 |
<reedy> |
synchronized php-1.23wmf4/languages/messages/MessagesCe.php 'Update ce language special page aliases' |
[production] |
| 12:59 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Ic36cd656233dfc959c92a33bade794cc5c1e1bd2' |
[production] |
| 11:21 |
<akosiaris> |
neon booted and seems ok. running updatedb manually |
[production] |
| 10:53 |
<akosiaris> |
rebooting neon for fsck |
[production] |
| 09:20 |
<apergos> |
snmptt stuck waiting for a submit_check_result that had exited (or died), restarted it to clear the snmptt spool queue |
[production] |
| 08:14 |
<apergos> |
rebooting neon again, found four filenames with deleted inode but side effect is back in r/o for / |
[production] |
| 07:06 |
<apergos> |
rebooting neon, no underlying errors found except for the initial ext3 'deleted inode reference' which caused / to be remounted r/o |
[production] |
| 06:53 |
<apergos> |
neon back to read-only root filesystem, investigating |
[production] |
| 03:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Nov 22 03:23:33 UTC 2013 |
[production] |
| 02:35 |
<LocalisationUpdate> |
completed (1.23wmf5) at Fri Nov 22 02:35:51 UTC 2013 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.23wmf4) at Fri Nov 22 02:16:36 UTC 2013 |
[production] |
| 01:41 |
<csteipp> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 01:40 |
<csteipp> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 01:37 |
<csteipp> |
finished scap: Really send the new messages this time.. |
[production] |
| 01:16 |
<csteipp> |
started scap: Really send the new messages this time.. |
[production] |
| 00:56 |
<ori> |
synchronized php-1.23wmf4/resources/startup.js 'touch' |
[production] |