|
2016-10-26
ยง
|
| 18:31 |
<mark> |
Disabling BGP session to AS6461 on cr1-eqiad, preparing for port migration |
[production] |
| 18:27 |
<mark> |
Chris is moving cr1-eqiad and cr2-eqiad xe-5/3/0 to xe-3/3/0 (both sides) |
[production] |
| 18:20 |
<maxsem@tin> |
Synchronized php-1.28.0-wmf.23/extensions/GeoData/: https://gerrit.wikimedia.org/r/#/c/318138/ (duration: 00m 47s) |
[production] |
| 18:19 |
<mark> |
Chris is moving cr1-eqiad and cr2-eqiad xe-5/2/0 to xe-3/2/0 (both sides) |
[production] |
| 18:12 |
<mark> |
Disabling cr1-eqiad:xe-5/2/0 |
[production] |
| 18:10 |
<bd808@tin> |
Synchronized wmf-config/CommonSettings.php: wikitech: Re-enable OAuth management interfaces T149150 (duration: 00m 46s) |
[production] |
| 18:04 |
<ema> |
cache_text - finished rolling downtimed reboots for kernel update |
[production] |
| 17:23 |
<cwd> |
updated smashpig from daba8c07108246261ca7ec557e3f1e5da430826e to f5f49d79fac0887df78ddcc68a2f5ced0670c387 |
[production] |
| 16:58 |
<mark> |
Shutting down cr1-eqiad:xe-5/0/[0-2] (part of aggregated links to rows A-C switches) |
[production] |
| 16:58 |
<mark> |
Chris moved cr1-eqiad:xe-5/2/1 to xe-3/0/3 |
[production] |
| 16:54 |
<mark> |
Chris moved cr1-eqiad:xe-5/1/[0-3] to xe-3/1/[0-3] |
[production] |
| 16:39 |
<moritzm> |
restarted ntp on mc2008 (stuck in XFAC state) |
[production] |
| 16:36 |
<jynus> |
stopping db2011 to replace disks T149099 |
[production] |
| 15:59 |
<mark> |
Disabling cr1-eqiad:ae4; VRRP conflict |
[production] |
| 15:58 |
<mark> |
Reenabling cr1-eqiad:ae4 |
[production] |
| 15:57 |
<mark> |
Restored cr1-eqiad:ae4 |
[production] |
| 15:37 |
<ema> |
cache_text - start rolling downtimed reboots for kernel update (~3 hours to completion) |
[production] |
| 15:27 |
<ema> |
cp1054 reboot for kernel update |
[production] |
| 15:16 |
<bblack> |
restarting grrrit-wm |
[production] |
| 14:34 |
<moritzm> |
rearmed keyholder on mira after reboot |
[production] |
| 14:24 |
<bblack> |
cache_upload - start rolling downtimed reboots for kernel update (~4 hours to completion) |
[production] |
| 14:19 |
<moritzm> |
rolling reboot of ocg cluster for kernel update |
[production] |
| 14:10 |
<demon@tin> |
Synchronized w: replacing wiki.phtml with a symlink (duration: 00m 47s) |
[production] |
| 13:39 |
<hashar> |
European SWAT deploy completed |
[production] |
| 13:37 |
<hashar> |
mw2098 is all set now after I ran "scap pull". It is properly in tin:/etc/dsh/group/mediawiki-installation |
[production] |
| 13:30 |
<hashar> |
mw2098: scap pull . It failed yesterday on reboot and is back in pull |
[production] |
| 13:28 |
<hashar> |
mw2098 spurts bunch of Notice: Undefined variable: wmgWatchlistDefault in /srv/mediawiki/wmf-config/CommonSettings.php on line 1871 |
[production] |
| 13:25 |
<hashar@tin> |
Synchronized wmf-config/CommonSettings.php: Remove obsolete config values (duration: 00m 46s) |
[production] |
| 13:24 |
<hashar@tin> |
Synchronized wmf-config/CommonSettings-labs.php: LABS: Enable Tabular data on Commons - T148745 (duration: 00m 45s) |
[production] |
| 13:23 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.23/extensions/Kartographer: T149145: Fix empty groups params T149154: Fix external links (duration: 00m 57s) |
[production] |
| 13:11 |
<hashar@tin> |
Synchronized wmf-config/CommonSettings.php: wikitech: Set wgMWOAuthCentralWiki = false - T149150 (duration: 00m 47s) |
[production] |
| 13:07 |
<moritzm> |
rebooting labtest hosts for kernel update |
[production] |
| 12:36 |
<marostegui> |
Deploy schema change s5 dewiki.revision only codfw - T148967 |
[production] |
| 12:31 |
<moritzm> |
rebooting mira for kernel update |
[production] |
| 12:20 |
<_joe_> |
turned off mw1152, removed salt/puppet data, T149185 |
[production] |
| 12:19 |
<bblack> |
moving git-ssh LVS from low-traffic -> high-traffic2 - T143915 |
[production] |
| 12:05 |
<bblack> |
moving ocg LVS from high-traffic2 -> low-traffic - T143915 |
[production] |
| 10:40 |
<bblack> |
rebooting eqiad lvs primaries (lvs100[1-3]) |
[production] |
| 10:19 |
<bblack> |
rebooting esams lvs primaries (lvs300[12]) |
[production] |
| 10:13 |
<bblack> |
rebooting ulsfo lvs primaries (lvs400[12]) |
[production] |
| 10:02 |
<bblack> |
rebooting codfw lvs primaries (lvs200[1-3]) |
[production] |
| 09:52 |
<jynus> |
starting schema change (imagelinks) on s1 T139090 |
[production] |
| 08:43 |
<elukey> |
increasing the AQS cassandra system_auth keyspace replication from 1 to 6 (and running nodetool-{a,b} repair system_auth on all nodes) |
[production] |
| 08:29 |
<elukey> |
downgraded memcached on mc2009 to the Debian Jessie version (was part of a performance experiment) |
[production] |
| 08:25 |
<dcausse> |
elastic@eqiad reindexing enwiki (take 3) with BM25 from wasat.codfw.wmnet T147508 (logs in ~dcausse/bm25_reindex/cirrus_log) |
[production] |
| 08:06 |
<marostegui> |
Stoppping replication on db2058 - using it to clone another host - T146261 |
[production] |
| 07:47 |
<moritzm> |
bounced ntp on oxygen (stuck in XFAC state) |
[production] |
| 07:45 |
<moritzm> |
rebooting mc* servers in codfw for kernel update |
[production] |
| 07:20 |
<moritzm> |
rebooting oxygen for kernel update |
[production] |
| 07:14 |
<marostegui> |
Deploying ALTER table s4 commonswiki.templatelinks - db2051 - T149079 |
[production] |