|
2014-02-11
§
|
| 15:56 |
<bd808> |
mw1185 continues to segfault at a rate of ~1000/hr |
[production] |
| 15:48 |
<mutante> |
DNS update - removing harmon |
[production] |
| 08:39 |
<nikerabbit> |
synchronized wmf-config/CommonSettings.php 'uls prep' |
[production] |
| 08:38 |
<nikerabbit> |
synchronized wmf-config/InitialiseSettings.php 'uls prep' |
[production] |
| 08:10 |
<nikerabbit> |
updated /a/common to {{Gerrit|Ib325d4aa9}}: Update ULS config |
[production] |
| 07:47 |
<gwicke> |
deployed parsoid hotfix to avoid recursive log spew filling up the logs |
[production] |
| 03:39 |
<RoanKattouw> |
Freed up disk space on wtp* by blanking /var/log/parsoid/parsoid.log |
[production] |
| 03:30 |
<RoanKattouw> |
Managed to restart Parsoid cleanly in the end. Turns out dsh -g parsoid service restart parsoid doesn't work but dsh -cM -g parsoid /etc/init.d/parsoid restart does work |
[production] |
| 03:23 |
<RoanKattouw> |
Doing rolling restart of the Parsoid cluster |
[production] |
| 03:21 |
<RoanKattouw> |
Copying wtp1005's Parsoid log for future reference |
[production] |
| 02:43 |
<Tim> |
restarting apache on servers with workers in a futex wait: mw1056,mw1082,mw1189,mw1196,mw1199,mw1201,mw1202,mw1203,mw1204,mw1208 |
[production] |
| 02:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-11 02:40:11+00:00 |
[production] |
| 02:19 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-11 02:19:13+00:00 |
[production] |
| 02:10 |
<LocalisationUpdate> |
completed (1.23wmf12) at 2014-02-11 02:10:38+00:00 |
[production] |
| 02:02 |
<awight> |
update crm from 7e5786aaf71363c04e5c766e6b403fa4767fe51a to 06f7e4d6d6c2653f1d4aef1e8b8e6293a82b39ef |
[production] |
| 01:25 |
<mwalker> |
updated fundraising civicrm from d96164f76877d75ab97c02e6b47449f7a45b31b3 to 7e5786aaf71363c04e5c766e6b403fa4767fe51a for unsbuscribe and quick search changes |
[production] |
| 00:56 |
<^demon> |
gerrit upgraded from 2.8.1 stable to 2.8.1-1-g83098d0 (custom build) to work around mysql issue pending upstream release. |
[production] |
| 00:34 |
<maxsem> |
finished scap: MobileApp deployment (duration: 28m 19s) |
[production] |
| 00:06 |
<maxsem> |
started scap: MobileApp deployment |
[production] |
| 00:04 |
<maxsem> |
scap aborted: MobileApp deployment (duration: 06m 33s) |
[production] |
|
2014-02-10
§
|
| 23:58 |
<maxsem> |
started scap: MobileApp deployment |
[production] |
| 23:03 |
<mutante> |
all parsoid machines reployed per gwicke's |
[production] |
| 23:00 |
<bd808> |
mw1185 segfaulting starting at 22:39Z. ~240 occurrences in last 20 minutes |
[production] |
| 22:59 |
<aaron> |
synchronized php-1.23wmf13/includes/db/LoadBalancer.php '8f6471e04ce0f33c64c090cbe5561deed82f60ee' |
[production] |
| 22:59 |
<springle> |
restarting db1050 for investigation |
[production] |
| 22:45 |
<mutante> |
restarting parsoid on wtp1008 |
[production] |
| 22:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 'sync proper non-hot depool db1050' |
[production] |
| 22:39 |
<springle> |
synchronized wmf-config/db-eqiad.php 'move s1 vslow dump' |
[production] |
| 22:36 |
<maxsem> |
synchronized wmf-config 'https://gerrit.wikimedia.org/r/112597' |
[production] |
| 22:34 |
<mark> |
Power cycled ms-be1001 |
[production] |
| 22:32 |
<springle> |
pt-kill jobs on s1 slaves killing anything sleeping longer than 10s |
[production] |
| 22:28 |
<springle> |
killed thousands of broken connections on s1 slaves in Sleep state |
[production] |
| 22:25 |
<maxsem> |
scap aborted: Extension:MobileApp deployment (duration: 14m 41s) |
[production] |
| 22:23 |
<matanya> |
big dberror spike. "Error connecting" to various ips from various ips |
[production] |
| 22:17 |
<springle> |
synchronized wmf-config/db-eqiad.php 'db1050 crashed, depool' |
[production] |
| 22:12 |
<mutante> |
fixing broken parsoid deploy on wtp*, one by one |
[production] |
| 22:10 |
<maxsem> |
started scap: Extension:MobileApp deployment |
[production] |
| 22:02 |
<mutante> |
wtp1016 - delete deployment/parsoid, salt-call fetch/checkout.., restart parsoid |
[production] |
| 21:57 |
<gwicke> |
unsuccessful Parsoid deploy as trebuchet failed to update the submodule with the parsoid source, need trebuchet bug fix |
[production] |
| 21:21 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 21:11 |
<catrope> |
synchronized visualeditor-default.dblist 'fix missing entries' |
[production] |
| 20:14 |
<mutante> |
harmon - revoke puppet cert,disable puppet,disable icinga notifications, shutting down |
[production] |
| 20:09 |
<mutante> |
harmon - removing from puppet stored configs, complete decom, unused Tampa spare |
[production] |
| 17:45 |
<mwalker> |
updated civicrm from 97a5146124168096148b6167e2968052b3dda468 to d96164f76877d75ab97c02e6b47449f7a45b31b3 for thank you translations |
[production] |
| 16:28 |
<manybubbles> |
correction: done with link count update for cirrus |
[production] |
| 16:28 |
<manybubbles> |
done with links count update for cirurs |
[production] |
| 15:58 |
<hashar> |
Jenkins: migrating labs jenkins-deploy user homedir from /home/jenkins-deploy (GlusterFS) to local directories under /mnt/home/jenkins-deploy to avoid GlusterFS and race conditions between instances. {{bug|61144}} |
[production] |
| 15:54 |
<manybubbles> |
reindex went well. performing a links recount so we can push more code changes next week safely. |
[production] |
| 15:28 |
<manybubbles> |
reindexing phase 0 wikis after Cirrus deploy last Thursday |
[production] |
| 12:20 |
<hashar> |
Jenkins: deleted /srv/slave-scrips from old jenkins servers, everything should now use /srv/deployment/integration/slave-scripts |
[production] |