3351-3400 of 10000 results (29ms)
2014-08-18 §
22:36 <andrewbogott> disabling puppet on virt1000 and virt1006 while I try to convince the scheduler to overlook virt1006 [production]
22:01 <bblack> done futzing w/ puppetmasters+neon, all agents enabled and bot back online [production]
21:28 <hashar> Zuul processing again. Definitely need to write doc about how to unstuck it [production]
21:02 <hashar> Zuul / Jenkins stalled again :-/ [production]
21:02 <hashar> Zuul / Jenkins stalled again :-/ [production]
19:35 <bblack> testing new passenger perf params on strontium/palladium. agents on those two and icinga-wm still disabled [production]
19:04 <bblack> restarted service apache2 on strontium - passenger for puppet master was dead again [production]
17:00 <andrewbogott> added a (yuvi-built) python-txstatsd package to trusty on Carbon. [production]
16:37 <bd808> deployment-prep Restarted Apache and HHVM on deployment-mediawiki02 to pick up removal of /etc/php5/conf.d/mail.ini [production]
16:27 <yurik> Synchronized php-1.24wmf17/extensions: Syncing JsonConfig,ZeroPortal,ZeroBanner (duration: 01m 13s) [production]
16:22 <yurik> Synchronized php-1.24wmf16/extensions: Syncing JsonConfig,ZeroPortal,ZeroBanner (duration: 01m 22s) [production]
16:18 <legoktm> migrateAccount.php finished, 2014-08-18 15:42:12 processed 1528652 usernames (22.9/sec), 10 (0.0%) fully migrated, 7938 (0.5%) partially migrated [production]
16:05 <hashar> Jenkins tox based jobs are now runnable in parallel {{gerrit|154834}} [production]
15:36 <manybubbles> swat complete [production]
15:29 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT - enable cirrus optimization - weighted all fields - on group0 wikis (duration: 00m 07s) [production]
15:29 <manybubbles> Synchronized wmf-config/CirrusSearch-common.php: SWAT - drop unused Cirrus parameter (duration: 00m 05s) [production]
15:25 <manybubbles> Synchronized php-1.24wmf16/extensions/CentralAuth: SWAT - two centralauth fixes (duration: 00m 05s) [production]
15:22 <bblack> resuming slowly wiping varnish caches for mmap update (49 hosts to go), expect small 5xx spikes every ~1.5 hrs for the next few days [production]
15:22 <manybubbles> Synchronized wmf-config/: SWAT - noop - sync files adding bouncehandler to betalabs (duration: 00m 04s) [production]
15:19 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT - create portal/portal talk namespaces on kowikisource (duration: 00m 04s) [production]
15:18 <manybubbles> Synchronized php-1.24wmf17/extensions/CentralAuth/: SWAT - two centralauth fixes (duration: 00m 04s) [production]
15:13 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT - create eliminator role on viwiki (duration: 00m 05s) [production]
15:11 <manybubbles> Synchronized php-1.24wmf17/extensions/Wikidata/: (no message) (duration: 00m 07s) [production]
15:08 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT - Add global-renamer group to metawiki (duration: 00m 04s) [production]
14:50 <hashar> Jenkins: reverting PHP CodeSniffer upgrade. We are back to 1.4.7. {{gerrit|154825}} had some issue. [production]
14:42 <hashar> Jenkins: upgrading PHP Codesniffer from 1.4.7 to 1.4.8 (thanks to addshore {{gerrit|154053}}) [production]
14:39 <bd808> No apache2.log in fluorine:/a/mw-log; Last file in /a/mw-log/archive is apache2.log-20140816.gz [production]
14:31 <bd808> Restarted logstash on logstash1001; event volume was lower than expected [production]
13:49 <hashar> restarting zuul. Got stuck again. [production]
13:29 <hashar_> Restarted Zuul, some items where stuck in queue. Retrigger your jobs (revote +2 / new patchset / 'recheck' comment) [production]
13:23 <reedy> Synchronized php-1.24wmf17/extensions/ExtensionDistributor: Unbreak ExtensionDistributor (duration: 00m 13s) [production]
13:18 <hashar> Zuul stuck, looking. [production]
13:06 <Reedy> Large amount of incoming traffic to bast1001 is me uploading files [production]
12:11 <godog> rebalanced swift object ring in eqiad [production]
09:34 <godog> reenabled puppet on neon and started ircecho [production]
09:24 <godog> stop ircecho again on neon, disable puppet on neon [production]
09:11 <godog> restarted apache2 on strontium [production]
08:58 <godog> stopped ircecho on neon while diagnosing puppet failure [production]
03:13 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Aug 18 03:12:27 UTC 2014 (duration 12m 26s) [production]
03:06 <hoo> Ran sync-common on mw1053 to stop "Unrecognized job type 'ChangeNotification'." exceptions [production]
02:31 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-18 02:30:17+00:00 [production]
02:20 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-18 02:18:52+00:00 [production]
2014-08-17 §
21:08 <legoktm> running migrateAccount.php without --safe or --auto on terbium for bug 69291 [production]
18:45 <hashar> Zuul upgraded [production]
18:41 <hashar> Upgrading Zuul to latest version (that is not a friday afterall) [production]
09:22 <springle> ongoing schema change wikidatawiki & testwiki wb_entity_per_page.epp_redirect_target. osc_host.sh processes on terbium ok to kill in emergency [production]
04:34 <ottomata> restarted udp2log on oxygen [production]
03:05 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Aug 17 03:04:22 UTC 2014 (duration 4m 21s) [production]
02:49 <springle> killed stuff on labsdb1003 using all disk for temp tables. investigating [production]
02:24 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-17 02:23:08+00:00 [production]