|
2014-02-10
§
|
| 22:32 |
<springle> |
pt-kill jobs on s1 slaves killing anything sleeping longer than 10s |
[production] |
| 22:28 |
<springle> |
killed thousands of broken connections on s1 slaves in Sleep state |
[production] |
| 22:25 |
<maxsem> |
scap aborted: Extension:MobileApp deployment (duration: 14m 41s) |
[production] |
| 22:23 |
<matanya> |
big dberror spike. "Error connecting" to various ips from various ips |
[production] |
| 22:17 |
<springle> |
synchronized wmf-config/db-eqiad.php 'db1050 crashed, depool' |
[production] |
| 22:12 |
<mutante> |
fixing broken parsoid deploy on wtp*, one by one |
[production] |
| 22:10 |
<maxsem> |
started scap: Extension:MobileApp deployment |
[production] |
| 22:02 |
<mutante> |
wtp1016 - delete deployment/parsoid, salt-call fetch/checkout.., restart parsoid |
[production] |
| 21:57 |
<gwicke> |
unsuccessful Parsoid deploy as trebuchet failed to update the submodule with the parsoid source, need trebuchet bug fix |
[production] |
| 21:21 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 21:11 |
<catrope> |
synchronized visualeditor-default.dblist 'fix missing entries' |
[production] |
| 20:14 |
<mutante> |
harmon - revoke puppet cert,disable puppet,disable icinga notifications, shutting down |
[production] |
| 20:09 |
<mutante> |
harmon - removing from puppet stored configs, complete decom, unused Tampa spare |
[production] |
| 17:45 |
<mwalker> |
updated civicrm from 97a5146124168096148b6167e2968052b3dda468 to d96164f76877d75ab97c02e6b47449f7a45b31b3 for thank you translations |
[production] |
| 16:28 |
<manybubbles> |
correction: done with link count update for cirrus |
[production] |
| 16:28 |
<manybubbles> |
done with links count update for cirurs |
[production] |
| 15:58 |
<hashar> |
Jenkins: migrating labs jenkins-deploy user homedir from /home/jenkins-deploy (GlusterFS) to local directories under /mnt/home/jenkins-deploy to avoid GlusterFS and race conditions between instances. {{bug|61144}} |
[production] |
| 15:54 |
<manybubbles> |
reindex went well. performing a links recount so we can push more code changes next week safely. |
[production] |
| 15:28 |
<manybubbles> |
reindexing phase 0 wikis after Cirrus deploy last Thursday |
[production] |
| 12:20 |
<hashar> |
Jenkins: deleted /srv/slave-scrips from old jenkins servers, everything should now use /srv/deployment/integration/slave-scripts |
[production] |
| 03:41 |
<springle> |
synchronized wmf-config/db-pmtpa.php 's2 switch master to db1023 (eqiad)' |
[production] |
| 03:40 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 switch master to db1023 (eqiad)' |
[production] |
| 03:16 |
<springle> |
synchronized wmf-config/db-pmtpa.php 's2 switch master to db1023 (pmtpa)' |
[production] |
| 03:16 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 switch master to db1023 (eqiad)' |
[production] |
| 03:03 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'prepare for s2 master rotation db1036 to db1024 (pmtpa)' |
[production] |
| 03:03 |
<springle> |
synchronized wmf-config/db-eqiad.php 'prepare for s2 master rotation db1036 to db1024 (eqiad)' |
[production] |
| 02:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 repool db1060 warm up' |
[production] |
| 02:33 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 repool db1002, depool db1060 schema changes' |
[production] |
| 02:31 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-10 02:31:08+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-10 02:15:59+00:00 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf12) at 2014-02-10 02:08:14+00:00 |
[production] |
| 02:07 |
<springle> |
synchronized wmf-config/db-eqiad.php 's2 depool db1002 schema changes' |
[production] |
|
2014-02-08
§
|
| 21:00 |
<aaron> |
synchronized php-1.23wmf12/extensions/Math 'd96b29ca8b17f35e7068f0d3a16b5e2644e084f9' |
[production] |
| 20:41 |
<aaron> |
synchronized php-1.23wmf13/extensions/Math '4844f52139593f4a324bf99b74d7abb91aac2e54' |
[production] |
| 09:31 |
<apergos> |
powercycling ms-be1001, 'soft lockup cpu stuck' on console, no login prompt |
[production] |
| 02:47 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-08 02:47:46+00:00 |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-08 02:27:14+00:00 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.23wmf12) at 2014-02-08 02:13:35+00:00 |
[production] |
| 00:57 |
<Nemo_bis> |
Job queue rather long (400k on en.wiki), OTRS reports of password resets not being delivered (bug 43936?), almost no jobs run today according to gdash |
[production] |
|
2014-02-07
§
|
| 22:22 |
<mwalker> |
updating payments wiki from 5f042ece4fa9d19293415ed41536ef93392c44ab to bad46aad4b28435d10570aac8173a7f8dca8751d for minfraud... again |
[production] |
| 21:46 |
<catrope> |
synchronized php-1.23wmf13/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'Use feature flag for Minerva Beta Feature' |
[production] |
| 21:46 |
<catrope> |
synchronized php-1.23wmf13/extensions/MobileFrontend/MobileFrontend.php 'Feature flag for Minerva Beta Feature' |
[production] |
| 21:34 |
<mwalker> |
updated paymentswiki from fac21c9d394f6d7c9da172cee18c805304c8fe1b to 5f042ece4fa9d19293415ed41536ef93392c44ab for Amazon Recurring and Minfraud health check |
[production] |
| 19:27 |
<mutante> |
running puppet on db9,pc1,pc2 etc, fixes mysqld process monitoring |
[production] |
| 19:04 |
<mutante> |
revoked grosley's puppet cert |
[production] |
| 18:49 |
<akosiaris> |
reenable puppet on mchenry as well as the exim cronjob. |
[production] |
| 18:27 |
<akosiaris> |
temporarily disabling puppet on mchenry and then disable collect_exim_stats_via_gmetric cron. Seems like mchenry has not ganglia at all |
[production] |