|
2014-02-11
§
|
| 18:06 |
<mflaschen> |
synchronized php-1.23wmf13/extensions/GettingStarted/ 'Sync GettingStarted on wmf13 again to fix our foreachwiki script' |
[production] |
| 18:05 |
<mflaschen> |
synchronized php-1.23wmf12/extensions/GettingStarted/ 'Sync GettingStarted on wmf12 again to fix our foreachwiki script' |
[production] |
| 17:48 |
<mflaschen> |
synchronized php-1.23wmf13/extensions/GettingStarted/ 'Sync GettingStarted on wmf13 for i18n rollout' |
[production] |
| 17:46 |
<mflaschen> |
synchronized php-1.23wmf12/extensions/GettingStarted/ 'Sync GettingStarted on wmf12 for i18n rollout' |
[production] |
| 16:18 |
<mutante> |
revoking puppet cert for locke |
[production] |
| 16:01 |
<mutante> |
locke is being decom'ed momentarily (purchase date 2006-12-04, heh) |
[production] |
| 15:56 |
<bd808> |
mw1094 segfaulting since 2014-02-11T22:42. Current rate ~45/hr. |
[production] |
| 15:56 |
<bd808> |
mw1185 continues to segfault at a rate of ~1000/hr |
[production] |
| 15:48 |
<mutante> |
DNS update - removing harmon |
[production] |
| 08:39 |
<nikerabbit> |
synchronized wmf-config/CommonSettings.php 'uls prep' |
[production] |
| 08:38 |
<nikerabbit> |
synchronized wmf-config/InitialiseSettings.php 'uls prep' |
[production] |
| 08:10 |
<nikerabbit> |
updated /a/common to {{Gerrit|Ib325d4aa9}}: Update ULS config |
[production] |
| 07:47 |
<gwicke> |
deployed parsoid hotfix to avoid recursive log spew filling up the logs |
[production] |
| 03:39 |
<RoanKattouw> |
Freed up disk space on wtp* by blanking /var/log/parsoid/parsoid.log |
[production] |
| 03:30 |
<RoanKattouw> |
Managed to restart Parsoid cleanly in the end. Turns out dsh -g parsoid service restart parsoid doesn't work but dsh -cM -g parsoid /etc/init.d/parsoid restart does work |
[production] |
| 03:23 |
<RoanKattouw> |
Doing rolling restart of the Parsoid cluster |
[production] |
| 03:21 |
<RoanKattouw> |
Copying wtp1005's Parsoid log for future reference |
[production] |
| 02:43 |
<Tim> |
restarting apache on servers with workers in a futex wait: mw1056,mw1082,mw1189,mw1196,mw1199,mw1201,mw1202,mw1203,mw1204,mw1208 |
[production] |
| 02:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-11 02:40:11+00:00 |
[production] |
| 02:19 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-11 02:19:13+00:00 |
[production] |
| 02:10 |
<LocalisationUpdate> |
completed (1.23wmf12) at 2014-02-11 02:10:38+00:00 |
[production] |
| 02:02 |
<awight> |
update crm from 7e5786aaf71363c04e5c766e6b403fa4767fe51a to 06f7e4d6d6c2653f1d4aef1e8b8e6293a82b39ef |
[production] |
| 01:25 |
<mwalker> |
updated fundraising civicrm from d96164f76877d75ab97c02e6b47449f7a45b31b3 to 7e5786aaf71363c04e5c766e6b403fa4767fe51a for unsbuscribe and quick search changes |
[production] |
| 00:56 |
<^demon> |
gerrit upgraded from 2.8.1 stable to 2.8.1-1-g83098d0 (custom build) to work around mysql issue pending upstream release. |
[production] |
| 00:34 |
<maxsem> |
finished scap: MobileApp deployment (duration: 28m 19s) |
[production] |
| 00:06 |
<maxsem> |
started scap: MobileApp deployment |
[production] |
| 00:04 |
<maxsem> |
scap aborted: MobileApp deployment (duration: 06m 33s) |
[production] |
|
2014-02-10
§
|
| 23:58 |
<maxsem> |
started scap: MobileApp deployment |
[production] |
| 23:03 |
<mutante> |
all parsoid machines reployed per gwicke's |
[production] |
| 23:00 |
<bd808> |
mw1185 segfaulting starting at 22:39Z. ~240 occurrences in last 20 minutes |
[production] |
| 22:59 |
<aaron> |
synchronized php-1.23wmf13/includes/db/LoadBalancer.php '8f6471e04ce0f33c64c090cbe5561deed82f60ee' |
[production] |
| 22:59 |
<springle> |
restarting db1050 for investigation |
[production] |
| 22:45 |
<mutante> |
restarting parsoid on wtp1008 |
[production] |
| 22:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 'sync proper non-hot depool db1050' |
[production] |
| 22:39 |
<springle> |
synchronized wmf-config/db-eqiad.php 'move s1 vslow dump' |
[production] |
| 22:36 |
<maxsem> |
synchronized wmf-config 'https://gerrit.wikimedia.org/r/112597' |
[production] |
| 22:34 |
<mark> |
Power cycled ms-be1001 |
[production] |
| 22:32 |
<springle> |
pt-kill jobs on s1 slaves killing anything sleeping longer than 10s |
[production] |
| 22:28 |
<springle> |
killed thousands of broken connections on s1 slaves in Sleep state |
[production] |
| 22:25 |
<maxsem> |
scap aborted: Extension:MobileApp deployment (duration: 14m 41s) |
[production] |
| 22:23 |
<matanya> |
big dberror spike. "Error connecting" to various ips from various ips |
[production] |
| 22:17 |
<springle> |
synchronized wmf-config/db-eqiad.php 'db1050 crashed, depool' |
[production] |
| 22:12 |
<mutante> |
fixing broken parsoid deploy on wtp*, one by one |
[production] |
| 22:10 |
<maxsem> |
started scap: Extension:MobileApp deployment |
[production] |
| 22:02 |
<mutante> |
wtp1016 - delete deployment/parsoid, salt-call fetch/checkout.., restart parsoid |
[production] |
| 21:57 |
<gwicke> |
unsuccessful Parsoid deploy as trebuchet failed to update the submodule with the parsoid source, need trebuchet bug fix |
[production] |
| 21:21 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 21:11 |
<catrope> |
synchronized visualeditor-default.dblist 'fix missing entries' |
[production] |
| 20:14 |
<mutante> |
harmon - revoke puppet cert,disable puppet,disable icinga notifications, shutting down |
[production] |
| 20:09 |
<mutante> |
harmon - removing from puppet stored configs, complete decom, unused Tampa spare |
[production] |