|
2014-10-02
§
|
| 13:23 |
<hashar> |
Zuul deadlocked somehow again :( |
[production] |
| 12:32 |
<godog> |
rolling-restart swift on ms-be1*, saw increased load possibly as a cause of 5xx spike |
[production] |
| 12:07 |
<godog> |
restarting rsyslog in eqiad |
[production] |
| 12:03 |
<akosiaris> |
restarted apparmor throughout the fleet |
[production] |
| 11:58 |
<hashar> |
Migrated all mediawiki-core-regression* jobs to Zuul cloner {{bug|71549}} |
[production] |
| 10:57 |
<godog> |
restarted rsyslog in codfw |
[production] |
| 10:57 |
<godog> |
restarted rsyslog in ulsfo |
[production] |
| 10:54 |
<godog> |
restarted rsyslog in esams |
[production] |
| 09:28 |
<godog> |
start rolling depooling/restart/pooling of swift frontends in eqiad to pick up syslog change |
[production] |
| 09:11 |
<_joe_> |
removing /mnt/tmpfs/fd29e937fea41d186175bcb880ef96980825dd1c.rdf2latex from ocg1001, it contains a 32 gb pdf |
[production] |
| 09:08 |
<_joe_> |
restarting node-ocg on ocg1001; a _lot_ of deleted files with the FD still opened |
[production] |
| 08:10 |
<hashar> |
Jenkins has been upgraded to latest LTS version 1.565.3 |
[production] |
| 07:33 |
<hashar> |
Jenkins restarting |
[production] |
| 07:27 |
<hashar> |
Stopping Jenkins |
[production] |
| 06:58 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool es1010 (duration: 00m 07s) |
[production] |
| 06:53 |
<springle> |
upgrade es1010 mariadb 10, restart |
[production] |
| 06:36 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool es1010 (duration: 00m 07s) |
[production] |
| 06:07 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool es1008, renable writes to external storage cluster 25 (duration: 00m 06s) |
[production] |
| 06:00 |
<ori> |
mw1053 still flooding error logs with "Unrecognized job type 'EchoNotificationDeleteJob'." Disabling Puppet and jobrunner for now, planning to investigate during SF daytime hours. |
[production] |
| 05:59 |
<springle> |
upgrade es1008 mariadb 10, restart |
[production] |
| 05:21 |
<MaxSem> |
Manual sync-common on mw1053 |
[production] |
| 05:19 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool es1008 (duration: 00m 08s) |
[production] |
| 05:04 |
<springle> |
Synchronized wmf-config/db-eqiad.php: divert writes away from external storage cluster 25 (duration: 00m 10s) |
[production] |
| 04:40 |
<springle> |
Synchronized wmf-config/db-eqiad.php: upgraded db1073, repool, warm up (duration: 00m 07s) |
[production] |
| 04:23 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 00m 13s) |
[production] |
| 04:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Oct 2 04:08:17 UTC 2014 (duration 8m 16s) |
[production] |
| 02:47 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-02 02:47:20+00:00 |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.24wmf22) at 2014-10-02 02:24:08+00:00 |
[production] |
|
2014-10-01
§
|
| 23:26 |
<demon> |
Synchronized php-1.25wmf1/extensions/GuidedTour: (no message) (duration: 00m 04s) |
[production] |
| 23:26 |
<demon> |
Synchronized php-1.24wmf22/extensions/GuidedTour: (no message) (duration: 00m 04s) |
[production] |
| 23:19 |
<demon> |
Synchronized php-1.25wmf1/extensions/VisualEditor: (no message) (duration: 00m 04s) |
[production] |
| 23:09 |
<Krinkle> |
Jenkins restart finished. Patched git-client-plugin seems to work as expected (bug 71533). |
[production] |
| 21:28 |
<aaron> |
Synchronized php-1.25wmf1/maintenance/findMissingFiles.php: 832ed2ce9938dc51fdb4190423ce03e93e65c639 (duration: 00m 05s) |
[production] |
| 21:04 |
<mutante> |
fenari - shutdown -h now (omg) :) |
[production] |
| 20:32 |
<hoo> |
Ran sync-common on mw1053 to stop "Unrecognized job type 'EchoNotificationDeleteJob'." exceptions |
[production] |
| 20:27 |
<cscott> |
updated OCG to version 48c495e3656f528abe636ce0cd7562270505534f |
[production] |
| 19:40 |
<yurik> |
Synchronized php-1.25wmf1/extensions/ZeroBanner/: (no message) (duration: 01m 05s) |
[production] |
| 19:38 |
<yurik> |
Synchronized php-1.24wmf22/extensions/ZeroBanner/: (no message) (duration: 01m 03s) |
[production] |
| 19:17 |
<mutante> |
fenari - removed from dsh - rejoice deployers, should be faster now |
[production] |
| 19:03 |
<yurik> |
Synchronized php-1.24wmf22/extensions/ZeroBanner/: (no message) (duration: 01m 08s) |
[production] |
| 18:59 |
<yurik> |
Synchronized php-1.24wmf22/extensions/ZeroBanner/: (no message) (duration: 01m 09s) |
[production] |
| 18:56 |
<yurik> |
Synchronized php-1.24wmf22/extensions/ZeroBanner/: (no message) (duration: 01m 04s) |
[production] |
| 18:34 |
<Krinkle> |
(..jenkins) The command runs fine when done in that workspace from shell. Looks like a bug with Jenkins Java abstraction layer. |
[production] |
| 18:28 |
<ori> |
disabling puppet on mw1019 to test impact of ProxyBadHeader apache directive |
[production] |
| 18:25 |
<Krinkle> |
Jenkins jobs fir repos with git submodules broken ("git-submodule: git reset: not found") |
[production] |
| 17:41 |
<andrewbogott> |
graceful'd apache on logstash1001 logstash1002 logstash1003 |
[production] |
| 16:56 |
<cmjohnson1> |
swapping disk db1020 |
[production] |
| 16:32 |
<godog> |
reverted change to syslog.eqiad.wmnet, back to nfs-home.pmtpa.wmnet |
[production] |
| 15:40 |
<JetLaggedPanda> |
purged graphite logs for deployment-mediawiki04 by hand on labmon1001 to prevent it from causing issues on icinga, since the instance has been deleted previously |
[production] |
| 15:32 |
<ottomata> |
starting upgrade of stat1002 from precise to trusty |
[production] |