|
2014-10-02
ยง
|
| 20:11 |
<bblack> |
stopping -> starting uwsgi/apache -type stuff on tungsten |
[production] |
| 19:55 |
<aude> |
populated sites table for fawikivoyage |
[production] |
| 19:27 |
<bd808> |
hosts that failed Trebuchet update of scap: virt0.wikimedia.org, fenari.wikimedia.org, mw1110.eqiad.wmnet, mw1053.eqiad.wmnet. mw1053.eqiad.wmnet only failed checkout |
[production] |
| 19:23 |
<bd808> |
Trebuchet reports for scap sync "231/234 minions completed fetch; 230/234 minions completed checkout" Some stale entries need to be removed from Trebuchet redis cache |
[production] |
| 19:21 |
<bd808> |
Updated scap to eff0d01 (Fix format specifier for error message) |
[production] |
| 19:12 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: fawikivoyage (duration: 00m 16s) |
[production] |
| 19:12 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: fawikivoyage |
[production] |
| 19:11 |
<reedy> |
Synchronized database lists: fawikivoyage (duration: 00m 16s) |
[production] |
| 18:47 |
<mutante> |
graceful'ed apache on mw1030,mw1164 |
[production] |
| 18:44 |
<reedy> |
Purged l10n cache for 1.24wmf21 |
[production] |
| 18:43 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf2 |
[production] |
| 18:41 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf2 |
[production] |
| 18:40 |
<reedy> |
Finished scap: testwiki to 1.25wmf2 (duration: 24m 19s) |
[production] |
| 18:32 |
<cmjohnson1> |
replacing failed disk es1005 |
[production] |
| 18:15 |
<reedy> |
Started scap: testwiki to 1.25wmf2 |
[production] |
| 17:57 |
<manybubbles> |
going to try to restart lsearchd on the misc pool machines to see if that makes it responsive |
[production] |
| 17:42 |
<ejegg> |
updated payments-wiki from 83464deed3b66da655ca5d1086852237c4793b71 to 9417bbd95057a87824be157dbbb5965a1f09d202 |
[production] |
| 17:26 |
<aaron> |
Synchronized php-1.25wmf1/maintenance/findMissingFiles.php: aa2eb3c0de08256822a2b0c985ebb3a6145d28cd (duration: 00m 05s) |
[production] |
| 16:30 |
<ori> |
graceful'd all apaches for I98bcdbfc7: mediawiki: add vhost_combined log format to apache2.conf |
[production] |
| 15:32 |
<demon> |
Synchronized php-1.25wmf1/extensions/Wikidata: (no message) (duration: 00m 20s) |
[production] |
| 15:32 |
<demon> |
Synchronized php-1.25wmf1/includes/jobqueue/jobs/ThumbnailRenderJob.php: (no message) (duration: 00m 05s) |
[production] |
| 15:21 |
<godog> |
shutting down nfs1 |
[production] |
| 15:15 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) |
[production] |
| 15:11 |
<demon> |
Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 15:11 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 15:09 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 15:09 |
<mark> |
Replaced exim4-deamon-light by exim4-daemon-heavy on tools-mail |
[production] |
| 15:06 |
<demon> |
Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 15:06 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s) |
[production] |
| 13:30 |
<hashar> |
Zuul back around :] |
[production] |
| 13:23 |
<hashar> |
Zuul deadlocked somehow again :( |
[production] |
| 12:32 |
<godog> |
rolling-restart swift on ms-be1*, saw increased load possibly as a cause of 5xx spike |
[production] |
| 12:07 |
<godog> |
restarting rsyslog in eqiad |
[production] |
| 12:03 |
<akosiaris> |
restarted apparmor throughout the fleet |
[production] |
| 11:58 |
<hashar> |
Migrated all mediawiki-core-regression* jobs to Zuul cloner {{bug|71549}} |
[production] |
| 10:57 |
<godog> |
restarted rsyslog in codfw |
[production] |
| 10:57 |
<godog> |
restarted rsyslog in ulsfo |
[production] |
| 10:54 |
<godog> |
restarted rsyslog in esams |
[production] |
| 09:28 |
<godog> |
start rolling depooling/restart/pooling of swift frontends in eqiad to pick up syslog change |
[production] |
| 09:11 |
<_joe_> |
removing /mnt/tmpfs/fd29e937fea41d186175bcb880ef96980825dd1c.rdf2latex from ocg1001, it contains a 32 gb pdf |
[production] |
| 09:08 |
<_joe_> |
restarting node-ocg on ocg1001; a _lot_ of deleted files with the FD still opened |
[production] |
| 08:10 |
<hashar> |
Jenkins has been upgraded to latest LTS version 1.565.3 |
[production] |
| 07:33 |
<hashar> |
Jenkins restarting |
[production] |
| 07:27 |
<hashar> |
Stopping Jenkins |
[production] |
| 06:58 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool es1010 (duration: 00m 07s) |
[production] |
| 06:53 |
<springle> |
upgrade es1010 mariadb 10, restart |
[production] |
| 06:36 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool es1010 (duration: 00m 07s) |
[production] |
| 06:07 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool es1008, renable writes to external storage cluster 25 (duration: 00m 06s) |
[production] |
| 06:00 |
<ori> |
mw1053 still flooding error logs with "Unrecognized job type 'EchoNotificationDeleteJob'." Disabling Puppet and jobrunner for now, planning to investigate during SF daytime hours. |
[production] |
| 05:59 |
<springle> |
upgrade es1008 mariadb 10, restart |
[production] |