|
2016-02-29
§
|
| 16:02 |
<_joe_> |
stopping hhvm on mw1050 (depooled) for testing bug T128380 |
[production] |
| 15:50 |
<urandom> |
Rolling restart of restbase production complete : T103124 |
[production] |
| 15:43 |
<urandom> |
Perform rolling restart of restbase in production cluster : T103124 |
[production] |
| 15:42 |
<urandom> |
Rolling restart of restbase staging complete : T103124 |
[production] |
| 15:33 |
<urandom> |
Perform rolling restart of restbase in staging cluster : T103124 |
[production] |
| 15:32 |
<urandom> |
forcing puppet run in restbase clustter ((noop) config deploy) : T103124 |
[production] |
| 15:30 |
<urandom> |
forcing puppet run in restbase staging ((noop) config deploy) : T103124 |
[production] |
| 14:17 |
<bblack> |
upgrading nginx on cp* to 1.9.4-1+wmf2 for T126616 |
[production] |
| 14:16 |
<gehel> |
elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 14:11 |
<gehel> |
re-enabling icinga notifications for elasticsearch cluster on codfw |
[production] |
| 14:00 |
<gehel> |
icinga: adding authorization for Gehel |
[production] |
| 13:54 |
<bblack> |
puppet back to normal on caches |
[production] |
| 13:35 |
<bblack> |
disabling puppet on cp* for traffic-pool work |
[production] |
| 12:50 |
<godog> |
gmetad on uranium restarted following https://gerrit.wikimedia.org/r/273881 it will converge asap |
[production] |
| 12:16 |
<gehel> |
elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 11:57 |
<jynus> |
reimaging dbproxy1005 with jessie before its put into production (potential alerts due to new role) |
[production] |
| 10:46 |
<ema> |
hhvm restarted on mw1119 |
[production] |
| 10:25 |
<jynus> |
powercycling mw1140, almost 100% unresponsive, OOM probable cause |
[production] |
| 09:58 |
<godog> |
bootstrap restbase1009-b T95253 |
[production] |
| 02:32 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Feb 29 02:32:43 UTC 2016 (duration 7m 43s) |
[production] |
| 02:25 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.14) (duration: 11m 12s) |
[production] |
|
2016-02-27
§
|
| 23:19 |
<legoktm> |
23:19:43 Synchronized php-1.27.0-wmf.14/extensions/wikihiero/: Revert "Compress PNGs with zopflipng" - https://gerrit.wikimedia.org/r/273661 (duration: 00m 46s) |
[production] |
| 22:39 |
<legoktm> |
legoktm just deployed https://gerrit.wikimedia.org/r/#/c/273655/ (MF cache pollution debug log) for T124356 |
[production] |
| 21:19 |
<urandom> |
issuing test repair on cerium (restbase staging), wp_parsoid html keyspace, (-w 5) : T108611 |
[production] |
| 20:08 |
<krinkle@tin> |
Synchronized php-1.27.0-wmf.14/includes/mime.types: Iec9459c921: Unbreak ULS fonts via wmfstatic (duration: 00m 41s) |
[production] |
| 19:32 |
<ori@tin> |
Synchronized php-1.27.0-wmf.14/includes/user/User.php: Revert I43cde3a48: Prevent duplicate memcached lookups for user record (T128246) (duration: 00m 56s) |
[production] |
| 04:15 |
<ori> |
On silver: commented out exception from I3eae28a58 to work around T128246 |
[production] |
| 03:53 |
<bd808> |
test |
[production] |
| 03:47 |
<bd808> |
Tried to fix CAS error for Labslogbot |
[production] |
| 03:36 |
<bd808> |
Backfilled missing missing https://wmflabs.org/sal/production data (2016-02-21T14:39 to 2016-02-24T16:53) |
[production] |
| 03:14 |
<urandom> |
bootstrap of restbase1008-a.eqiad.wmnet complete; begining `nodetool cleanup' of 1003, 1004-a, and 1008-b : T95253 |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Feb 27 02:31:10 UTC 2016 (duration 7m 44s) |
[production] |
| 02:23 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.14) (duration: 10m 24s) |
[production] |
| 01:50 |
<twentyafterfour> |
updated phabricator/tools to https://phabricator.wikimedia.org/rPHTO1433a20886a4168d41b81b5ecf5e29687678f2e2 (applying fix for T127524 ) |
[production] |
| 00:52 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.14/includes/filebackend/FileBackendMultiWrite.php: 7ca624b8b93 (duration: 00m 47s) |
[production] |
|
2016-02-26
§
|
| 22:20 |
<mutante> |
powercycle mw1130 |
[production] |
| 20:42 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: update wmf address in echo footer thingie (duration: 00m 59s) |
[production] |
| 20:29 |
<hashar> |
Restarting Jenkins |
[production] |
| 20:04 |
<urandom> |
issuing test repair on cerium (restbase staging), keyspace : T108611 |
[production] |
| 18:41 |
<ori@tin> |
Synchronized php-1.27.0-wmf.14/includes/user/User.php: I43cde3a48: Prevent duplicate memcached lookups for user record (duration: 01m 02s) |
[production] |
| 16:46 |
<chasemp> |
labstore1001 'mdadm --manage /dev/md126 --add /dev/sdaf' |
[production] |
| 15:51 |
<jynus> |
shutting down mariadb on db2030 to clone from db1009 |
[production] |
| 15:24 |
<urandom> |
forcing puppet run on restbase1009.eqiad.wmnet |
[production] |
| 15:24 |
<urandom> |
re-enabling puppet on restbase1009.eqiad.wmnet |
[production] |
| 15:20 |
<jynus> |
performing backup of m5-master mysql data |
[production] |