|
2016-03-16
ยง
|
| 17:59 |
<mutante> |
rcs1002 - traffic graph flat in ganglia, reboot |
[production] |
| 17:57 |
<mutante> |
rcs1002 - the last message was about 1002 |
[production] |
| 17:56 |
<mutante> |
rcs1001 - depool from rcstream service |
[production] |
| 17:55 |
<legoktm@tin> |
scap aborted: Building l10n cache for UrlShortener - T108557 (duration: 02m 51s) |
[production] |
| 17:52 |
<legoktm@tin> |
Started scap: Building l10n cache for UrlShortener - T108557 |
[production] |
| 17:43 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on ja, zh, pl, ar and nlwiki (duration: 00m 29s) |
[production] |
| 17:28 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on ru, fr, pt and itwiki (duration: 00m 28s) |
[production] |
| 17:10 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on eswiki (duration: 00m 25s) |
[production] |
| 17:09 |
<krinkle@tin> |
Synchronized php-1.27.0-wmf.17/extensions/Kartographer/extension.json: (no message) (duration: 00m 25s) |
[production] |
| 16:46 |
<gehel> |
restarting elasticsearch server elastic2024.codfw.wmnet |
[production] |
| 16:36 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on enwiki (duration: 00m 30s) |
[production] |
| 16:32 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 31s) |
[production] |
| 16:11 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on dewiki (duration: 00m 44s) |
[production] |
| 16:11 |
<moritzm> |
upgraded exim packages on magnesium to the version from USN 2933, without it add_environment can't be set exim failed to start |
[production] |
| 16:06 |
<moritzm> |
upgraded remaining exim binary packages on krypton to 4.84-8+deb8u2 (only exim4-daemon-heavy was at the version, resulting in the new add_environment variable being unknown and exim failing to start) |
[production] |
| 16:02 |
<gehel> |
restarting elasticsearch server elastic2023.codfw.wmnet |
[production] |
| 15:25 |
<moritzm> |
rebooting graphite1001 for kernel upgrade |
[production] |
| 15:16 |
<moritzm> |
rebooting graphite2001 for kernel upgrade |
[production] |
| 15:09 |
<volans> |
restarting mysql on dbstore2002 Aria engine crashed |
[production] |
| 15:08 |
<moritzm> |
rebooting lithium for kernel upgrade |
[production] |
| 15:07 |
<jzerebecki@tin> |
Synchronized wmf-config/InitialiseSettings.php: Whitelist feeds on mediawiki.org 622c186b2712c923fbcc48c27f65ebf396176f3e T127176 (duration: 00m 31s) |
[production] |
| 14:52 |
<volans> |
Reimporting table bnwikisource.shorturls into dbstore2002 as InnoDB |
[production] |
| 14:51 |
<gehel> |
restarting elasticsearch server elastic2022.codfw.wmnet |
[production] |
| 14:36 |
<volans> |
REPAIR NO_WRITE_TO_BINLOG TABLE bnwikisource.shorturls replica failed for corrupted table |
[production] |
| 14:18 |
<papaul> |
labstore200[3-4}] - signing puppet certs, salt-key, initial run |
[production] |
| 14:06 |
<papaul> |
wasat signing puppet certs, salt-key, initial run |
[production] |
| 14:06 |
<gehel> |
restarting elasticsearch server elastic2021.codfw.wmnet |
[production] |
| 14:03 |
<chasemp> |
cleanup snapshots on labstore1001 |
[production] |
| 14:02 |
<volans> |
Reimaging db2008 to jessie T130098 |
[production] |
| 13:23 |
<gehel> |
restarting elasticsearch server elastic2020.codfw.wmnet |
[production] |
| 13:15 |
<elukey> |
puppet re-enabled on analytics1027 |
[production] |
| 12:39 |
<gehel> |
restarting elasticsearch server elastic2019.codfw.wmnet |
[production] |
| 12:36 |
<godog> |
bootstrapping restbase1012-a T125842 |
[production] |
| 11:44 |
<elukey> |
rebooting analytics1001/1002 (Yarn/HDFS master nodes) for kernel upgrade |
[production] |
| 11:24 |
<elukey> |
rebooting analytics105* for kernel upgrade |
[production] |
| 11:02 |
<volans> |
Revoking puppet key for db2008.codfw.wmnet (puppet wasn't running already) T130098 |
[production] |
| 10:51 |
<elukey> |
rebooting analytics102[89] and analtics104* for kernel upgrades |
[production] |
| 10:50 |
<moritzm> |
rolling reboot of mw* servers in eqiad (except job runners to not interfere with T129517) |
[production] |
| 10:21 |
<elukey> |
Rebooting analytics103* for kernel upgrade |
[production] |
| 09:33 |
<elukey> |
puppet disabled on analytics1027 as preparation step for the Hadoop nodes reboots |
[production] |
| 09:18 |
<jynus> |
setting up pending cross-datacenter master-master database links |
[production] |
| 09:04 |
<moritzm> |
rebooting stat1002/stat1003 for kernel upgrade |
[production] |
| 08:53 |
<_joe_> |
restarting hhvm on mw1007, stuck in a deadlock, apparently on HPHP::Treadmill::getAgeOldestRequest |
[production] |
| 08:53 |
<moritzm> |
rebooting labtest*2001 for kernel upgrade |
[production] |
| 08:45 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Return s2 slaves to normal weight (duration: 00m 35s) |
[production] |
| 08:04 |
<moritzm> |
restarted mw1129/1002 (HHVM hung) |
[production] |
| 06:47 |
<ebernhardson> |
rebooting elastic2018.codfw.wmnet for kernel upgrade |
[production] |
| 06:10 |
<ebernhardson> |
rebooting elastic2017.codfw.wmnet for kernel upgrade |
[production] |
| 05:39 |
<ebernhardson> |
rebooting elastic2016.codfw.wmnet for kernel upgrade |
[production] |
| 04:42 |
<ebernhardson> |
rebooting elastic2015.codfw.wmnet for kernel upgrade |
[production] |