|
2014-08-06
§
|
| 18:10 |
<mutante> |
puppet-catalog-compiler says to "wait while Jenkins is getting ready to work" |
[production] |
| 17:20 |
<hashar> |
Jenkins process jobs again, the UI will take a bunch of hours to load though due to some issue when initializing |
[production] |
| 17:14 |
<hashar> |
killed Jenkins |
[production] |
| 17:12 |
<_joe_> |
stopped the jobrunner on mw1053, was running in fcgi mode unpuppetized and with a broken vhost. Fixed it, it started spawning exceptions. DO NOT enable puppet again |
[production] |
| 17:02 |
<^d> |
jenkins restarted, was stuck |
[production] |
| 15:52 |
<hashar> |
Restarted Zuul and Zuul-merger on gallium to tweak logging settings {{gerrit|152118}} |
[production] |
| 11:30 |
<hoo> |
Synchronized wmf-config/CommonSettings.php: Grant 'centralauth-rename' to 'steward' (duration: 00m 24s) |
[production] |
| 11:26 |
<demon> |
Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 19s) |
[production] |
| 10:10 |
<hashar> |
Jenkins web interface is back up |
[production] |
| 09:54 |
<demon> |
Synchronized wmf-config/abusefilter.php: abuse filter settings for fawiki (duration: 00m 21s) |
[production] |
| 07:33 |
<hashar> |
restarting Jenkins. It apparently like to parse the whole history on reload, so aborting that. |
[production] |
| 07:13 |
<hashar> |
Upgrading Jenkins plugin and restarting. |
[production] |
| 07:04 |
<hashar> |
upgrading Jenkins to latest LTS |
[production] |
| 03:11 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Aug 6 03:10:06 UTC 2014 (duration 10m 5s) |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-06 02:29:00+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-08-06 02:16:38+00:00 |
[production] |
|
2014-08-04
§
|
| 22:58 |
<bblack> |
rebooting ms-be1012 |
[production] |
| 21:47 |
<ottomata> |
reenabling puppet on analytics1027 |
[production] |
| 21:47 |
<jgage> |
all kafka brokers upraded to 0.8.1.1 and data replicated: done |
[production] |
| 20:37 |
<ottomata> |
stopping puppet on analytics1027 to temporarily disable camus cron job |
[production] |
| 19:07 |
<ottomata> |
starting upgrade of kafka cluster |
[production] |
| 19:02 |
<maxsem> |
Synchronized php-1.24wmf16/includes/User.php: https://gerrit.wikimedia.org/r/#/c/151691/ (duration: 00m 06s) |
[production] |
| 18:57 |
<jgage> |
beginning kafka upgrade: disabling puppet on brokers |
[production] |
| 13:17 |
<apergos> |
stopped labs rsync job from dataset1001, mount of labstore1003 was borked, removed 90GB of stuff on /mnt/data (= /) filesystem, restarted nfsd on dataset1001, dumps back to going |
[production] |
| 03:12 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Aug 4 03:11:03 UTC 2014 (duration 11m 2s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-04 02:27:58+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-08-04 02:16:46+00:00 |
[production] |
|
2014-08-02
§
|
| 15:28 |
<godog> |
reboot ms-be1008, stuck on xfs errors and most processes in D state |
[production] |
| 14:10 |
<Krinkle> |
Restarting Zuul |
[production] |
| 14:09 |
<hashar> |
Jenkins / Zuul stuck {{bug|69045}} |
[production] |
| 14:00 |
<Krinkle> |
Restarting Jenkins in attempt to unstuck the clogged Zuul pipeline for gallium |
[production] |
| 04:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Aug 2 04:20:45 UTC 2014 (duration 20m 44s) |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-02 02:32:36+00:00 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-08-02 02:20:02+00:00 |
[production] |
| 01:49 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1018, replag (duration: 00m 06s) |
[production] |
| 00:43 |
<Krinkle> |
Restarting Jenkins on gallium because the pipeline is clogged |
[production] |