|
2014-08-19
§
|
| 09:58 |
<godog> |
depool mw1019 from appservers, testing trusty+hhvm reinstall RT #8153 |
[production] |
| 07:39 |
<bblack> |
strontium ok, icinga-wm back |
[production] |
| 07:17 |
<hashar> |
Jenkins: manually cleared out a tmpfs partition on lanthanum.eqiad.wmnet which was causing all MediaWiki / extensions jobs to fail completely. {{bug|69731}}. We need disk space monitoring which is {{bug|69733}}. |
[production] |
| 07:10 |
<bblack> |
... and strontium passenger is failing to start up correctly again. icinga-wm disabled to avoid spam |
[production] |
| 07:07 |
<bblack> |
restarted apache2 service on strontium/palladium, expect another small spike of puppet fail->ok |
[production] |
| 03:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Aug 19 03:20:21 UTC 2014 (duration 20m 20s) |
[production] |
| 02:37 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-19 02:36:21+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-19 02:15:14+00:00 |
[production] |
|
2014-08-18
§
|
| 23:03 |
<andrewbogott> |
isolated virt1006, re-enabling puppet on virt1000 and virt1006 |
[production] |
| 22:36 |
<andrewbogott> |
disabling puppet on virt1000 and virt1006 while I try to convince the scheduler to overlook virt1006 |
[production] |
| 22:01 |
<bblack> |
done futzing w/ puppetmasters+neon, all agents enabled and bot back online |
[production] |
| 21:28 |
<hashar> |
Zuul processing again. Definitely need to write doc about how to unstuck it |
[production] |
| 21:02 |
<hashar> |
Zuul / Jenkins stalled again :-/ |
[production] |
| 21:02 |
<hashar> |
Zuul / Jenkins stalled again :-/ |
[production] |
| 19:35 |
<bblack> |
testing new passenger perf params on strontium/palladium. agents on those two and icinga-wm still disabled |
[production] |
| 19:04 |
<bblack> |
restarted service apache2 on strontium - passenger for puppet master was dead again |
[production] |
| 17:00 |
<andrewbogott> |
added a (yuvi-built) python-txstatsd package to trusty on Carbon. |
[production] |
| 16:37 |
<bd808> |
deployment-prep Restarted Apache and HHVM on deployment-mediawiki02 to pick up removal of /etc/php5/conf.d/mail.ini |
[production] |
| 16:27 |
<yurik> |
Synchronized php-1.24wmf17/extensions: Syncing JsonConfig,ZeroPortal,ZeroBanner (duration: 01m 13s) |
[production] |
| 16:22 |
<yurik> |
Synchronized php-1.24wmf16/extensions: Syncing JsonConfig,ZeroPortal,ZeroBanner (duration: 01m 22s) |
[production] |
| 16:18 |
<legoktm> |
migrateAccount.php finished, 2014-08-18 15:42:12 processed 1528652 usernames (22.9/sec), 10 (0.0%) fully migrated, 7938 (0.5%) partially migrated |
[production] |
| 16:05 |
<hashar> |
Jenkins tox based jobs are now runnable in parallel {{gerrit|154834}} |
[production] |
| 15:36 |
<manybubbles> |
swat complete |
[production] |
| 15:29 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT - enable cirrus optimization - weighted all fields - on group0 wikis (duration: 00m 07s) |
[production] |
| 15:29 |
<manybubbles> |
Synchronized wmf-config/CirrusSearch-common.php: SWAT - drop unused Cirrus parameter (duration: 00m 05s) |
[production] |
| 15:25 |
<manybubbles> |
Synchronized php-1.24wmf16/extensions/CentralAuth: SWAT - two centralauth fixes (duration: 00m 05s) |
[production] |
| 15:22 |
<bblack> |
resuming slowly wiping varnish caches for mmap update (49 hosts to go), expect small 5xx spikes every ~1.5 hrs for the next few days |
[production] |
| 15:22 |
<manybubbles> |
Synchronized wmf-config/: SWAT - noop - sync files adding bouncehandler to betalabs (duration: 00m 04s) |
[production] |
| 15:19 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT - create portal/portal talk namespaces on kowikisource (duration: 00m 04s) |
[production] |
| 15:18 |
<manybubbles> |
Synchronized php-1.24wmf17/extensions/CentralAuth/: SWAT - two centralauth fixes (duration: 00m 04s) |
[production] |
| 15:13 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT - create eliminator role on viwiki (duration: 00m 05s) |
[production] |
| 15:11 |
<manybubbles> |
Synchronized php-1.24wmf17/extensions/Wikidata/: (no message) (duration: 00m 07s) |
[production] |
| 15:08 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT - Add global-renamer group to metawiki (duration: 00m 04s) |
[production] |
| 14:50 |
<hashar> |
Jenkins: reverting PHP CodeSniffer upgrade. We are back to 1.4.7. {{gerrit|154825}} had some issue. |
[production] |
| 14:42 |
<hashar> |
Jenkins: upgrading PHP Codesniffer from 1.4.7 to 1.4.8 (thanks to addshore {{gerrit|154053}}) |
[production] |
| 14:39 |
<bd808> |
No apache2.log in fluorine:/a/mw-log; Last file in /a/mw-log/archive is apache2.log-20140816.gz |
[production] |
| 14:31 |
<bd808> |
Restarted logstash on logstash1001; event volume was lower than expected |
[production] |
| 13:49 |
<hashar> |
restarting zuul. Got stuck again. |
[production] |
| 13:29 |
<hashar_> |
Restarted Zuul, some items where stuck in queue. Retrigger your jobs (revote +2 / new patchset / 'recheck' comment) |
[production] |
| 13:23 |
<reedy> |
Synchronized php-1.24wmf17/extensions/ExtensionDistributor: Unbreak ExtensionDistributor (duration: 00m 13s) |
[production] |
| 13:18 |
<hashar> |
Zuul stuck, looking. |
[production] |
| 13:06 |
<Reedy> |
Large amount of incoming traffic to bast1001 is me uploading files |
[production] |
| 12:11 |
<godog> |
rebalanced swift object ring in eqiad |
[production] |
| 09:34 |
<godog> |
reenabled puppet on neon and started ircecho |
[production] |
| 09:24 |
<godog> |
stop ircecho again on neon, disable puppet on neon |
[production] |
| 09:11 |
<godog> |
restarted apache2 on strontium |
[production] |
| 08:58 |
<godog> |
stopped ircecho on neon while diagnosing puppet failure |
[production] |
| 03:13 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Aug 18 03:12:27 UTC 2014 (duration 12m 26s) |
[production] |
| 03:06 |
<hoo> |
Ran sync-common on mw1053 to stop "Unrecognized job type 'ChangeNotification'." exceptions |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-18 02:30:17+00:00 |
[production] |