|
2014-02-10
§
|
| 23:00 |
<bd808> |
mw1185 segfaulting starting at 22:39Z. ~240 occurrences in last 20 minutes |
[production] |
| 22:59 |
<aaron> |
synchronized php-1.23wmf13/includes/db/LoadBalancer.php '8f6471e04ce0f33c64c090cbe5561deed82f60ee' |
[production] |
| 22:59 |
<springle> |
restarting db1050 for investigation |
[production] |
| 22:45 |
<mutante> |
restarting parsoid on wtp1008 |
[production] |
| 22:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 'sync proper non-hot depool db1050' |
[production] |
| 22:39 |
<springle> |
synchronized wmf-config/db-eqiad.php 'move s1 vslow dump' |
[production] |
| 22:36 |
<maxsem> |
synchronized wmf-config 'https://gerrit.wikimedia.org/r/112597' |
[production] |
| 22:34 |
<mark> |
Power cycled ms-be1001 |
[production] |
| 22:32 |
<springle> |
pt-kill jobs on s1 slaves killing anything sleeping longer than 10s |
[production] |
| 22:28 |
<springle> |
killed thousands of broken connections on s1 slaves in Sleep state |
[production] |
| 22:25 |
<maxsem> |
scap aborted: Extension:MobileApp deployment (duration: 14m 41s) |
[production] |
| 22:23 |
<matanya> |
big dberror spike. "Error connecting" to various ips from various ips |
[production] |
| 22:17 |
<springle> |
synchronized wmf-config/db-eqiad.php 'db1050 crashed, depool' |
[production] |
| 22:12 |
<mutante> |
fixing broken parsoid deploy on wtp*, one by one |
[production] |
| 22:10 |
<maxsem> |
started scap: Extension:MobileApp deployment |
[production] |
| 22:02 |
<mutante> |
wtp1016 - delete deployment/parsoid, salt-call fetch/checkout.., restart parsoid |
[production] |
| 21:57 |
<gwicke> |
unsuccessful Parsoid deploy as trebuchet failed to update the submodule with the parsoid source, need trebuchet bug fix |
[production] |
| 21:21 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 21:11 |
<catrope> |
synchronized visualeditor-default.dblist 'fix missing entries' |
[production] |
| 20:14 |
<mutante> |
harmon - revoke puppet cert,disable puppet,disable icinga notifications, shutting down |
[production] |
| 20:09 |
<mutante> |
harmon - removing from puppet stored configs, complete decom, unused Tampa spare |
[production] |
| 17:45 |
<mwalker> |
updated civicrm from 97a5146124168096148b6167e2968052b3dda468 to d96164f76877d75ab97c02e6b47449f7a45b31b3 for thank you translations |
[production] |
| 16:28 |
<manybubbles> |
correction: done with link count update for cirrus |
[production] |
| 16:28 |
<manybubbles> |
done with links count update for cirurs |
[production] |
| 15:58 |
<hashar> |
Jenkins: migrating labs jenkins-deploy user homedir from /home/jenkins-deploy (GlusterFS) to local directories under /mnt/home/jenkins-deploy to avoid GlusterFS and race conditions between instances. {{bug|61144}} |
[production] |
| 15:54 |
<manybubbles> |
reindex went well. performing a links recount so we can push more code changes next week safely. |
[production] |
| 15:28 |
<manybubbles> |
reindexing phase 0 wikis after Cirrus deploy last Thursday |
[production] |
| 12:20 |
<hashar> |
Jenkins: deleted /srv/slave-scrips from old jenkins servers, everything should now use /srv/deployment/integration/slave-scripts |
[production] |
| 03:41 |
<springle> |
synchronized wmf-config/db-pmtpa.php 's2 switch master to db1023 (eqiad)' |
[production] |
| 03:40 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 switch master to db1023 (eqiad)' |
[production] |
| 03:16 |
<springle> |
synchronized wmf-config/db-pmtpa.php 's2 switch master to db1023 (pmtpa)' |
[production] |
| 03:16 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 switch master to db1023 (eqiad)' |
[production] |
| 03:03 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'prepare for s2 master rotation db1036 to db1024 (pmtpa)' |
[production] |
| 03:03 |
<springle> |
synchronized wmf-config/db-eqiad.php 'prepare for s2 master rotation db1036 to db1024 (eqiad)' |
[production] |
| 02:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 repool db1060 warm up' |
[production] |
| 02:33 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 repool db1002, depool db1060 schema changes' |
[production] |
| 02:31 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-10 02:31:08+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-10 02:15:59+00:00 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf12) at 2014-02-10 02:08:14+00:00 |
[production] |
| 02:07 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 depool db1002 schema changes' |
[production] |
|
2014-02-08
§
|
| 21:00 |
<aaron> |
synchronized php-1.23wmf12/extensions/Math 'd96b29ca8b17f35e7068f0d3a16b5e2644e084f9' |
[production] |
| 20:41 |
<aaron> |
synchronized php-1.23wmf13/extensions/Math '4844f52139593f4a324bf99b74d7abb91aac2e54' |
[production] |
| 09:31 |
<apergos> |
powercycling ms-be1001, 'soft lockup cpu stuck' on console, no login prompt |
[production] |
| 02:47 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-08 02:47:46+00:00 |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-08 02:27:14+00:00 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.23wmf12) at 2014-02-08 02:13:35+00:00 |
[production] |
| 00:57 |
<Nemo_bis> |
Job queue rather long (400k on en.wiki), OTRS reports of password resets not being delivered (bug 43936?), almost no jobs run today according to gdash |
[production] |