8401-8450 of 10000 results (28ms)
2014-08-06 §
18:49 <hashar> Stopping Jenkins. Reverting upgrade of artifact deployer plugin [production]
18:10 <mutante> puppet-catalog-compiler says to "wait while Jenkins is getting ready to work" [production]
17:20 <hashar> Jenkins process jobs again, the UI will take a bunch of hours to load though due to some issue when initializing [production]
17:14 <hashar> killed Jenkins [production]
17:12 <_joe_> stopped the jobrunner on mw1053, was running in fcgi mode unpuppetized and with a broken vhost. Fixed it, it started spawning exceptions. DO NOT enable puppet again [production]
17:02 <^d> jenkins restarted, was stuck [production]
15:52 <hashar> Restarted Zuul and Zuul-merger on gallium to tweak logging settings {{gerrit|152118}} [production]
11:30 <hoo> Synchronized wmf-config/CommonSettings.php: Grant 'centralauth-rename' to 'steward' (duration: 00m 24s) [production]
11:26 <demon> Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 19s) [production]
10:10 <hashar> Jenkins web interface is back up [production]
09:54 <demon> Synchronized wmf-config/abusefilter.php: abuse filter settings for fawiki (duration: 00m 21s) [production]
07:33 <hashar> restarting Jenkins. It apparently like to parse the whole history on reload, so aborting that. [production]
07:13 <hashar> Upgrading Jenkins plugin and restarting. [production]
07:04 <hashar> upgrading Jenkins to latest LTS [production]
03:11 <LocalisationUpdate> ResourceLoader cache refresh completed at Wed Aug 6 03:10:06 UTC 2014 (duration 10m 5s) [production]
02:30 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-06 02:29:00+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf15) at 2014-08-06 02:16:38+00:00 [production]
2014-08-05 §
15:07 <root> gracefulled all apaches [production]
15:03 <root> gracefulled all apaches [production]
12:30 <hasharEat> Upgrading python-gear on gallium and restarting zuul and zuul-merger [production]
12:26 <akosiaris> uploaded python-gear_0.5.5-1 on apt.wikimedia.org [production]
03:09 <LocalisationUpdate> ResourceLoader cache refresh completed at Tue Aug 5 03:08:30 UTC 2014 (duration 8m 29s) [production]
02:29 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-05 02:27:58+00:00 [production]
02:18 <LocalisationUpdate> completed (1.24wmf15) at 2014-08-05 02:16:52+00:00 [production]
00:41 <springle> ongoing schema changes: ar_content_model, ar_content_format. on terbium, osc_host.sh processes ok to kill in emergency [production]
2014-08-04 §
22:58 <bblack> rebooting ms-be1012 [production]
21:47 <ottomata> reenabling puppet on analytics1027 [production]
21:47 <jgage> all kafka brokers upraded to 0.8.1.1 and data replicated: done [production]
20:37 <ottomata> stopping puppet on analytics1027 to temporarily disable camus cron job [production]
19:07 <ottomata> starting upgrade of kafka cluster [production]
19:02 <maxsem> Synchronized php-1.24wmf16/includes/User.php: https://gerrit.wikimedia.org/r/#/c/151691/ (duration: 00m 06s) [production]
18:57 <jgage> beginning kafka upgrade: disabling puppet on brokers [production]
13:17 <apergos> stopped labs rsync job from dataset1001, mount of labstore1003 was borked, removed 90GB of stuff on /mnt/data (= /) filesystem, restarted nfsd on dataset1001, dumps back to going [production]
03:12 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Aug 4 03:11:03 UTC 2014 (duration 11m 2s) [production]
02:29 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-04 02:27:58+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf15) at 2014-08-04 02:16:46+00:00 [production]
2014-08-03 §
03:30 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Aug 3 03:28:56 UTC 2014 (duration 28m 55s) [production]
02:28 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-03 02:27:44+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf15) at 2014-08-03 02:16:39+00:00 [production]
2014-08-02 §
15:28 <godog> reboot ms-be1008, stuck on xfs errors and most processes in D state [production]
14:10 <Krinkle> Restarting Zuul [production]
14:09 <hashar> Jenkins / Zuul stuck {{bug|69045}} [production]
14:00 <Krinkle> Restarting Jenkins in attempt to unstuck the clogged Zuul pipeline for gallium [production]
04:21 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Aug 2 04:20:45 UTC 2014 (duration 20m 44s) [production]
02:33 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-02 02:32:36+00:00 [production]
02:21 <LocalisationUpdate> completed (1.24wmf15) at 2014-08-02 02:20:02+00:00 [production]
01:49 <springle> Synchronized wmf-config/db-eqiad.php: depool db1018, replag (duration: 00m 06s) [production]
00:43 <Krinkle> Restarting Jenkins on gallium because the pipeline is clogged [production]
2014-08-01 §
20:25 <andrewbogott> shorted the logrotate interval on vanadium; disk space critical should resolve soon [production]
18:11 <csteipp> Synchronized php-1.24wmf16/extensions/CentralAuth: Fix for bug 69007 - logins failing for old style hashes (duration: 00m 06s) [production]