|
2014-08-04
§
|
| 22:58 |
<bblack> |
rebooting ms-be1012 |
[production] |
| 21:47 |
<ottomata> |
reenabling puppet on analytics1027 |
[production] |
| 21:47 |
<jgage> |
all kafka brokers upraded to 0.8.1.1 and data replicated: done |
[production] |
| 20:37 |
<ottomata> |
stopping puppet on analytics1027 to temporarily disable camus cron job |
[production] |
| 19:07 |
<ottomata> |
starting upgrade of kafka cluster |
[production] |
| 19:02 |
<maxsem> |
Synchronized php-1.24wmf16/includes/User.php: https://gerrit.wikimedia.org/r/#/c/151691/ (duration: 00m 06s) |
[production] |
| 18:57 |
<jgage> |
beginning kafka upgrade: disabling puppet on brokers |
[production] |
| 13:17 |
<apergos> |
stopped labs rsync job from dataset1001, mount of labstore1003 was borked, removed 90GB of stuff on /mnt/data (= /) filesystem, restarted nfsd on dataset1001, dumps back to going |
[production] |
| 03:12 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Aug 4 03:11:03 UTC 2014 (duration 11m 2s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-04 02:27:58+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-08-04 02:16:46+00:00 |
[production] |
|
2014-08-02
§
|
| 15:28 |
<godog> |
reboot ms-be1008, stuck on xfs errors and most processes in D state |
[production] |
| 14:10 |
<Krinkle> |
Restarting Zuul |
[production] |
| 14:09 |
<hashar> |
Jenkins / Zuul stuck {{bug|69045}} |
[production] |
| 14:00 |
<Krinkle> |
Restarting Jenkins in attempt to unstuck the clogged Zuul pipeline for gallium |
[production] |
| 04:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Aug 2 04:20:45 UTC 2014 (duration 20m 44s) |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-02 02:32:36+00:00 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-08-02 02:20:02+00:00 |
[production] |
| 01:49 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1018, replag (duration: 00m 06s) |
[production] |
| 00:43 |
<Krinkle> |
Restarting Jenkins on gallium because the pipeline is clogged |
[production] |
|
2014-08-01
§
|
| 20:25 |
<andrewbogott> |
shorted the logrotate interval on vanadium; disk space critical should resolve soon |
[production] |
| 18:11 |
<csteipp> |
Synchronized php-1.24wmf16/extensions/CentralAuth: Fix for bug 69007 - logins failing for old style hashes (duration: 00m 06s) |
[production] |
| 17:32 |
<AaronSchulz> |
Restarted maintenance/populateBacklinkNamespace.php on enwiki |
[production] |
| 17:31 |
<aaron> |
Synchronized php-1.24wmf15/maintenance/populateBacklinkNamespace.php: e1cea29342f964cd9a720310185b09ca41eb1a4a (duration: 00m 04s) |
[production] |
| 17:16 |
<akosiaris> |
upgraded etherpad-lite on zirconium to 1.4.0-2. Uploaded etherpad-lite_1.4.0-2 on apt.wikimedia.org |
[production] |
| 17:12 |
<aaron> |
Synchronized php-1.24wmf15/includes: d218d86dff90a5f0110353c492bd2e8ddaf35497 (duration: 00m 08s) |
[production] |
| 17:09 |
<aaron> |
Synchronized php-1.24wmf16/includes: f1a8ff7f802b57cc9f452d47c4c762a185ed93c2 (duration: 00m 06s) |
[production] |
| 15:48 |
<reedy> |
Synchronized php-1.24wmf16/includes/specials/SpecialRecentchangeslinked.php: (no message) (duration: 00m 14s) |
[production] |
| 12:07 |
<apergos> |
powercycled dataset1001, inaccessible via mgmt console, only visible message was 'mnt.nfs failed' |
[production] |
| 09:10 |
<_joe_> |
apache mediawiki::web train finished its run. re-enabling puppet on all appservers |
[production] |
| 07:47 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Aug 1 07:46:04 UTC 2014 (duration 46m 3s) |
[production] |
| 07:24 |
<_joe_> |
stopping puppet on appservers to deploy a potentially dangerous case |
[production] |
| 05:16 |
<springle> |
Synchronized wmf-config/db-eqiad.php: Move enwiki api traffic away from lagging slaves (duration: 00m 07s) |
[production] |
| 03:12 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-01 03:11:14+00:00 |
[production] |