|
2013-12-02
§
|
| 18:58 |
<K4-713> |
updated fraud rules on payments |
[production] |
| 18:53 |
<demon> |
synchronized wmf-config/CommonSettings.php 'Search updates back off for Cirrus wikis :(' |
[production] |
| 18:35 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Dec 2 18:35:42 UTC 2013 |
[production] |
| 18:34 |
<ottomata> |
added logster deb, installed on cp3003 for testing, will puppetize shortly |
[production] |
| 18:30 |
<demon> |
synchronized wmf-config/CommonSettings.php 'Search updates back on for Cirrus' |
[production] |
| 17:47 |
<LocalisationUpdate> |
completed (1.23wmf5) at Mon Dec 2 17:47:29 UTC 2013 |
[production] |
| 17:31 |
<bblack> |
cp301[12].esams - puppet temporarily disabled, custom crash handler vmod in place to try to catch an error in the next couple of hours |
[production] |
| 17:31 |
<demon> |
synchronized wmf-config/CommonSettings.php 'Search update off for cirrus wikis' |
[production] |
| 17:31 |
<LocalisationUpdate> |
completed (1.23wmf4) at Mon Dec 2 17:31:11 UTC 2013 |
[production] |
| 17:30 |
<demon> |
updated /a/common to {{Gerrit|I507a72cca}}: Fix submodule reference change that snuck into If5b3a27a |
[production] |
| 17:21 |
<demon> |
synchronized wmf-config/InitialiseSettings.php 'Cirrus on all the wikis (that had it before)' |
[production] |
| 17:09 |
<demon> |
synchronized php-1.23wmf5/extensions/CirrusSearch 'Cirrus to master' |
[production] |
| 17:07 |
<demon> |
synchronized php-1.23wmf4/extensions/CirrusSearch 'Cirrus to master' |
[production] |
| 17:06 |
<demon> |
synchronized wmf-config/CirrusSearch-common.php |
[production] |
| 16:33 |
<paravoid> |
rebooting cp1065, usual XFS deadlock |
[production] |
| 12:35 |
<Jeff_Green> |
stopped mysql on db1008 to clone a database |
[production] |
| 07:05 |
<springle> |
upgrade/reboot db1046 m2 slave |
[production] |
| 06:53 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool pc1003 after upgrade, max_connections lowered during warm up' |
[production] |
| 06:28 |
<paravoid> |
fixed ganglia for misc eqiad (possibly others); see {{Gerrit|Icc5376505}} |
[production] |
| 06:04 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool pc1003 for upgrade' |
[production] |
| 02:51 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool pc1002 after upgrade, max_connections lowered during warm up' |
[production] |
| 02:37 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Dec 2 02:37:48 UTC 2013 |
[production] |
| 02:23 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool pc1002 for upgrade' |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.23wmf5) at Mon Dec 2 02:14:57 UTC 2013 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf4) at Mon Dec 2 02:08:15 UTC 2013 |
[production] |
| 01:16 |
<springle> |
restarting labsdb1002 mysqld processes with 25% smaller buffer pools. kernel OOM killer striking. needs investigation |
[production] |
|
2013-11-29
§
|
| 21:09 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'Id9c7321b8: Add a MassMessage-related user group on Meta' |
[production] |
| 21:08 |
<ori> |
updated /a/common to {{Gerrit|Id9c7321b8}}: Add a MassMessage-related user group on Meta |
[production] |
| 20:18 |
<paravoid> |
rebooting cp1065 with new kernel |
[production] |
| 19:26 |
<paravoid> |
"swapoff -a" on all cache_text to deal with strange kernel issue with kswapd dropping the whole page cache on memory pressure |
[production] |
| 14:49 |
<paravoid> |
restarted gmond on ms-fe1001/2, both were stuck 6h ago and we lost all swift eqiad's metrics for that period |
[production] |
| 11:12 |
<Reedy> |
Created EducationProgram tables on arwiki |
[production] |
| 05:51 |
<Tim> |
on cp1052 and cp1053: tweaked /proc/sys/net/core/rmem_default to see if that fixes the observed massive gmond packet loss |
[production] |
| 02:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Nov 29 02:07:55 UTC 2013 |
[production] |
| 02:02 |
<LocalisationUpdate> |
completed (1.23wmf5) at Fri Nov 29 02:02:25 UTC 2013 |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.23wmf4) at Fri Nov 29 02:01:44 UTC 2013 |
[production] |
| 01:22 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool pc1001 after upgrade, max_connections lowered during warm up' |
[production] |
| 00:25 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool pc1001 for package upgrade' |
[production] |