|
2016-10-26
§
|
| 16:54 |
<mark> |
Chris moved cr1-eqiad:xe-5/1/[0-3] to xe-3/1/[0-3] |
[production] |
| 16:39 |
<moritzm> |
restarted ntp on mc2008 (stuck in XFAC state) |
[production] |
| 16:36 |
<jynus> |
stopping db2011 to replace disks T149099 |
[production] |
| 15:59 |
<mark> |
Disabling cr1-eqiad:ae4; VRRP conflict |
[production] |
| 15:58 |
<mark> |
Reenabling cr1-eqiad:ae4 |
[production] |
| 15:57 |
<mark> |
Restored cr1-eqiad:ae4 |
[production] |
| 15:37 |
<ema> |
cache_text - start rolling downtimed reboots for kernel update (~3 hours to completion) |
[production] |
| 15:27 |
<ema> |
cp1054 reboot for kernel update |
[production] |
| 15:16 |
<bblack> |
restarting grrrit-wm |
[production] |
| 14:34 |
<moritzm> |
rearmed keyholder on mira after reboot |
[production] |
| 14:24 |
<bblack> |
cache_upload - start rolling downtimed reboots for kernel update (~4 hours to completion) |
[production] |
| 14:19 |
<moritzm> |
rolling reboot of ocg cluster for kernel update |
[production] |
| 14:10 |
<demon@tin> |
Synchronized w: replacing wiki.phtml with a symlink (duration: 00m 47s) |
[production] |
| 13:39 |
<hashar> |
European SWAT deploy completed |
[production] |
| 13:37 |
<hashar> |
mw2098 is all set now after I ran "scap pull". It is properly in tin:/etc/dsh/group/mediawiki-installation |
[production] |
| 13:30 |
<hashar> |
mw2098: scap pull . It failed yesterday on reboot and is back in pull |
[production] |
| 13:28 |
<hashar> |
mw2098 spurts bunch of Notice: Undefined variable: wmgWatchlistDefault in /srv/mediawiki/wmf-config/CommonSettings.php on line 1871 |
[production] |
| 13:25 |
<hashar@tin> |
Synchronized wmf-config/CommonSettings.php: Remove obsolete config values (duration: 00m 46s) |
[production] |
| 13:24 |
<hashar@tin> |
Synchronized wmf-config/CommonSettings-labs.php: LABS: Enable Tabular data on Commons - T148745 (duration: 00m 45s) |
[production] |
| 13:23 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.23/extensions/Kartographer: T149145: Fix empty groups params T149154: Fix external links (duration: 00m 57s) |
[production] |
| 13:11 |
<hashar@tin> |
Synchronized wmf-config/CommonSettings.php: wikitech: Set wgMWOAuthCentralWiki = false - T149150 (duration: 00m 47s) |
[production] |
| 13:07 |
<moritzm> |
rebooting labtest hosts for kernel update |
[production] |
| 12:36 |
<marostegui> |
Deploy schema change s5 dewiki.revision only codfw - T148967 |
[production] |
| 12:31 |
<moritzm> |
rebooting mira for kernel update |
[production] |
| 12:20 |
<_joe_> |
turned off mw1152, removed salt/puppet data, T149185 |
[production] |
| 12:19 |
<bblack> |
moving git-ssh LVS from low-traffic -> high-traffic2 - T143915 |
[production] |
| 12:05 |
<bblack> |
moving ocg LVS from high-traffic2 -> low-traffic - T143915 |
[production] |
| 10:40 |
<bblack> |
rebooting eqiad lvs primaries (lvs100[1-3]) |
[production] |
| 10:19 |
<bblack> |
rebooting esams lvs primaries (lvs300[12]) |
[production] |
| 10:13 |
<bblack> |
rebooting ulsfo lvs primaries (lvs400[12]) |
[production] |
| 10:02 |
<bblack> |
rebooting codfw lvs primaries (lvs200[1-3]) |
[production] |
| 09:52 |
<jynus> |
starting schema change (imagelinks) on s1 T139090 |
[production] |
| 08:43 |
<elukey> |
increasing the AQS cassandra system_auth keyspace replication from 1 to 6 (and running nodetool-{a,b} repair system_auth on all nodes) |
[production] |
| 08:29 |
<elukey> |
downgraded memcached on mc2009 to the Debian Jessie version (was part of a performance experiment) |
[production] |
| 08:25 |
<dcausse> |
elastic@eqiad reindexing enwiki (take 3) with BM25 from wasat.codfw.wmnet T147508 (logs in ~dcausse/bm25_reindex/cirrus_log) |
[production] |
| 08:06 |
<marostegui> |
Stoppping replication on db2058 - using it to clone another host - T146261 |
[production] |
| 07:47 |
<moritzm> |
bounced ntp on oxygen (stuck in XFAC state) |
[production] |
| 07:45 |
<moritzm> |
rebooting mc* servers in codfw for kernel update |
[production] |
| 07:20 |
<moritzm> |
rebooting oxygen for kernel update |
[production] |
| 07:14 |
<marostegui> |
Deploying ALTER table s4 commonswiki.templatelinks - db2051 - T149079 |
[production] |
| 07:10 |
<moritzm> |
rebooting nescio for kernel update |
[production] |
| 06:34 |
<moritzm> |
repooled mw2098 (was previously down for hardware check) |
[production] |
| 03:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Oct 26 03:00:28 UTC 2016 (duration 5m 28s) |
[production] |
| 02:55 |
<l10nupdate@tin> |
scap sync-l10n completed (1.28.0-wmf.23) (duration: 10m 13s) |
[production] |
| 02:28 |
<l10nupdate@tin> |
scap sync-l10n completed (1.28.0-wmf.22) (duration: 09m 38s) |
[production] |
| 01:36 |
<mutante> |
lead - (formerly gerrit) - shutdown -h now (T147905) |
[production] |
| 01:13 |
<mutante> |
palladium - shutdown -h now |
[production] |
| 00:10 |
<mutante> |
restarted icinga-wm, now there is /var/log/icinga/irc.log, it should talk now, but doesnt |
[production] |
| 00:04 |
<mutante> |
let icinga own /var/log/icinga on einsteinium, restart icinga |
[production] |