|
2016-03-16
§
|
| 15:25 |
<moritzm> |
rebooting graphite1001 for kernel upgrade |
[production] |
| 15:16 |
<moritzm> |
rebooting graphite2001 for kernel upgrade |
[production] |
| 15:09 |
<volans> |
restarting mysql on dbstore2002 Aria engine crashed |
[production] |
| 15:08 |
<moritzm> |
rebooting lithium for kernel upgrade |
[production] |
| 15:07 |
<jzerebecki@tin> |
Synchronized wmf-config/InitialiseSettings.php: Whitelist feeds on mediawiki.org 622c186b2712c923fbcc48c27f65ebf396176f3e T127176 (duration: 00m 31s) |
[production] |
| 14:52 |
<volans> |
Reimporting table bnwikisource.shorturls into dbstore2002 as InnoDB |
[production] |
| 14:51 |
<gehel> |
restarting elasticsearch server elastic2022.codfw.wmnet |
[production] |
| 14:36 |
<volans> |
REPAIR NO_WRITE_TO_BINLOG TABLE bnwikisource.shorturls replica failed for corrupted table |
[production] |
| 14:18 |
<papaul> |
labstore200[3-4}] - signing puppet certs, salt-key, initial run |
[production] |
| 14:06 |
<papaul> |
wasat signing puppet certs, salt-key, initial run |
[production] |
| 14:06 |
<gehel> |
restarting elasticsearch server elastic2021.codfw.wmnet |
[production] |
| 14:03 |
<chasemp> |
cleanup snapshots on labstore1001 |
[production] |
| 14:02 |
<volans> |
Reimaging db2008 to jessie T130098 |
[production] |
| 13:23 |
<gehel> |
restarting elasticsearch server elastic2020.codfw.wmnet |
[production] |
| 13:15 |
<elukey> |
puppet re-enabled on analytics1027 |
[production] |
| 12:39 |
<gehel> |
restarting elasticsearch server elastic2019.codfw.wmnet |
[production] |
| 12:36 |
<godog> |
bootstrapping restbase1012-a T125842 |
[production] |
| 11:44 |
<elukey> |
rebooting analytics1001/1002 (Yarn/HDFS master nodes) for kernel upgrade |
[production] |
| 11:24 |
<elukey> |
rebooting analytics105* for kernel upgrade |
[production] |
| 11:02 |
<volans> |
Revoking puppet key for db2008.codfw.wmnet (puppet wasn't running already) T130098 |
[production] |
| 10:51 |
<elukey> |
rebooting analytics102[89] and analtics104* for kernel upgrades |
[production] |
| 10:50 |
<moritzm> |
rolling reboot of mw* servers in eqiad (except job runners to not interfere with T129517) |
[production] |
| 10:21 |
<elukey> |
Rebooting analytics103* for kernel upgrade |
[production] |
| 09:33 |
<elukey> |
puppet disabled on analytics1027 as preparation step for the Hadoop nodes reboots |
[production] |
| 09:18 |
<jynus> |
setting up pending cross-datacenter master-master database links |
[production] |
| 09:04 |
<moritzm> |
rebooting stat1002/stat1003 for kernel upgrade |
[production] |
| 08:53 |
<_joe_> |
restarting hhvm on mw1007, stuck in a deadlock, apparently on HPHP::Treadmill::getAgeOldestRequest |
[production] |
| 08:53 |
<moritzm> |
rebooting labtest*2001 for kernel upgrade |
[production] |
| 08:45 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Return s2 slaves to normal weight (duration: 00m 35s) |
[production] |
| 08:04 |
<moritzm> |
restarted mw1129/1002 (HHVM hung) |
[production] |
| 06:47 |
<ebernhardson> |
rebooting elastic2018.codfw.wmnet for kernel upgrade |
[production] |
| 06:10 |
<ebernhardson> |
rebooting elastic2017.codfw.wmnet for kernel upgrade |
[production] |
| 05:39 |
<ebernhardson> |
rebooting elastic2016.codfw.wmnet for kernel upgrade |
[production] |
| 04:42 |
<ebernhardson> |
rebooting elastic2015.codfw.wmnet for kernel upgrade |
[production] |
| 03:41 |
<ebernhardson> |
rebooting elastic2014.codfw.wmnet for kernel upgrade |
[production] |
| 03:19 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Mar 16 03:19:26 UTC 2016 (duration 9m 41s) |
[production] |
| 03:09 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.17) (duration: 17m 36s) |
[production] |
| 02:45 |
<ebernhardson> |
rebooting elastic2013.codfw.wmnet for kernel upgrade |
[production] |
| 02:34 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.16) (duration: 13m 44s) |
[production] |
| 02:17 |
<ebernhardson> |
rebooting elastic2012.codfw.wmnet for kernel upgrade |
[production] |
| 01:32 |
<ebernhardson> |
rebooting elastic2011.codfw.wmnet for kernel upgrade |
[production] |
| 01:18 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 26s) |
[production] |
| 00:57 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 24s) |
[production] |
| 00:48 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/utils/BacklinkJobUtils.php: I32ec0 (duration: 00m 30s) |
[production] |
| 00:46 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 28s) |
[production] |
| 00:17 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Log REQUEST_URI (duration: 00m 28s) |
[production] |
| 00:15 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/extensions/Wikidata/extensions/Wikibase/client/includes/Hooks/DataUpdateHookHandlers.php: Revert briefly disable Wikibase\Client\Hooks\DataUpdateHookHandlers::onParserCacheSaveComplete (duration: 00m 31s) |
[production] |
| 00:01 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/extensions/Wikidata/extensions/Wikibase/client/includes/Hooks/DataUpdateHookHandlers.php: Briefly disable Wikibase\Client\Hooks\DataUpdateHookHandlers::onParserCacheSaveComplete (duration: 00m 26s) |
[production] |