|
2013-11-04
§
|
| 18:30 |
<mark> |
Configured cr1-esams and cr2-knams for new LVS service IP range 91.198.174.192/27 |
[production] |
| 18:25 |
<aaron> |
synchronized php-1.23wmf2/maintenance/cleanupUploadStash.php 'f3332a3d932c6919b3311d89edee20f886e4e86e' |
[production] |
| 17:58 |
<awight> |
rollback crm to 272966e353244e668a64c54ca4fa6f344079f4bf |
[production] |
| 17:57 |
<awight> |
updated crm from 272966e353244e668a64c54ca4fa6f344079f4bf to 63de2d6b8bc92d6c2130a242ff4995ba71b5ce11 |
[production] |
| 17:49 |
<mark> |
Deactivated ae1.101 on cr1-esams and cr2-knams |
[production] |
| 16:42 |
<cmjohnson1> |
dns update |
[production] |
| 16:31 |
<mark> |
Disabled multicast routing between eqiad and pmtpa, setup udpmcast between chromium and dobson instead |
[production] |
| 14:44 |
<mark> |
Disabled multicast traffic reduction on csw1-sdtpa |
[production] |
| 14:41 |
<akosiaris> |
removed LACP configuration from asw-b-eqiad for palladium and set it to standard access-port and private1-b-eqiad vlan |
[production] |
| 14:41 |
<akosiaris> |
removed LACP configuration from asw-a-eqiad for strontium and set it to standard access-port and private1-a-eqiad vlan |
[production] |
| 14:24 |
<mark> |
Killed udpmcast on chromium |
[production] |
| 14:13 |
<mark> |
Configured OSPF/OSPF3 on cr1-eqiad:xe-4/2/2 <--> cr2-knams:xe-1/1/0 |
[production] |
| 13:01 |
<hashar> |
Jenkins upgrading gearman (0.0.4 -> 0.0.5) plugin on gallium from http://repo.jenkins-ci.org/repo/org/jenkins-ci/plugins/gearman-plugin/0.0.5/ |
[production] |
| 11:22 |
<mark> |
multicast routing to pmtpa restored for now |
[production] |
| 10:52 |
<mark> |
Deactivated BFD on flapping link ae0 between cr1-eqiad and cr2-eqiad |
[production] |
| 10:47 |
<mark> |
Deactivated anycast PIM RP on cr2-eqiad |
[production] |
| 10:21 |
<mark> |
Reenabled 10G wave between cr2-eqiad and cr1-sdtpa, set high OSPF metrics instead |
[production] |
| 09:56 |
<akosiaris> |
added python-apscheduler to apt.wikimedia.org |
[production] |
| 09:48 |
<hashar> |
Jenkins: upgrading PHP_CodeSniffer from 1.4.6 to 1.4.7 |
[production] |
| 08:32 |
<reedy> |
synchronized php-1.23wmf2/ 'Ibcf77ed7f04c14a477d7cfd0e244929c552c3394' |
[production] |
| 08:00 |
<paravoid> |
cr2-eqiad set xe-5/2/1 disable; 10g wave to cr1-sdtpa, flapping since yesterday, causing packet loss and outages |
[production] |
| 06:24 |
<jeremyb> |
left morebots running on tools-login because it doesn't work on all grid hosts. see https://gerrit.wikimedia.org/r/93426 |
[production] |
| 05:58 |
<jeremyb> |
test2 |
[production] |
| 05:50 |
<jeremyb> |
test |
[production] |
| 05:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Nov 4 05:08:07 UTC 2013 |
[production] |
| 04:40 |
<LocalisationUpdate> |
completed (1.23wmf2) at Mon Nov 4 04:40:40 UTC 2013 |
[production] |
| 04:30 |
<LocalisationUpdate> |
completed (1.23wmf1) at Mon Nov 4 04:30:48 UTC 2013 |
[production] |
| 04:19 |
<ori-l> |
l10nupdate failures are due to "No submodule mapping found in .gitmodules for path 'WikibaseDatabase'"; repo appears to have been deleted; fixed with git rm --cached WikibaseDatabase as l10nupdate on tin. |
[production] |
| 04:15 |
<LocalisationUpdate> |
failed: git pull of extensions failed |
[production] |
| 03:12 |
<Tim> |
moved udpmcast to chromium since it actually has an external IP address |
[production] |
| 03:05 |
<Tim> |
moved udpmcast.py from dobson to tungsten to work around failure of multicast routing eqiad -> pmtpa |
[production] |
| 02:01 |
<LocalisationUpdate> |
failed: git pull of extensions failed |
[production] |
|
2013-11-02
§
|
| 19:50 |
<cmjohnson1> |
dns update |
[production] |
| 16:14 |
<reedy> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 12:43 |
<springle> |
start xtrabackup db52->db69 and db39->db71 |
[production] |
| 09:56 |
<apergos> |
restarted twemproxy on ~10 of the api servers in eqiad in the lower memory/core group, was seeing a lot of entries in memcached-serious log and some processes churning on that |
[production] |
| 07:37 |
<apergos> |
restarted a bunch of the apaches on the 12gb memory apis that were maxed out at their 100 client limit |
[production] |
| 07:09 |
<apergos> |
and increased again to 30, after looking at memory on both groups of boxes. |
[production] |
| 06:58 |
<apergos> |
increase weight of mw1189-1208 api servers from 20 to 25, they handle the load better (hopefully) |
[production] |
| 06:03 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'depool db69 and db71 for reassignment during pmtpa decomm' |
[production] |
| 02:00 |
<LocalisationUpdate> |
failed: git pull of extensions failed |
[production] |
| 00:39 |
<awight> |
updated crm from da3448df1404311620adc50b4d4ca28abfd2f3cb to 272966e353244e668a64c54ca4fa6f344079f4bf |
[production] |