|
2016-03-17
§
|
| 13:42 |
<mobrovac> |
restbase resrting cassandra on restbase2004 - OOM again |
[production] |
| 13:36 |
<gehel> |
restarting elasticsearch server elastic1012.eqiad.wmnet |
[production] |
| 13:02 |
<gehel> |
restarting elasticsearch server elastic1011.eqiad.wmnet |
[production] |
| 13:00 |
<moritzm> |
rolling reboot of parsoid servers in codfw for kernel upgrade |
[production] |
| 12:38 |
<_joe_> |
switching all services back to eqiad |
[production] |
| 12:25 |
<gehel> |
restarting elasticsearch server elastic1010.eqiad.wmnet |
[production] |
| 12:19 |
<akosiaris> |
restart pybal on lvs2001 |
[production] |
| 12:16 |
<akosiaris> |
restart pybal on lvs2002 |
[production] |
| 12:05 |
<_joe_> |
switching cxserver to codfw |
[production] |
| 11:44 |
<moritzm> |
rebooting acamar for kernel upgrade |
[production] |
| 11:41 |
<gehel> |
restarting elasticsearch server elastic1009.eqiad.wmnet |
[production] |
| 11:33 |
<_joe_> |
switching citoid to use codfw as well |
[production] |
| 11:31 |
<moritzm> |
rebooting install2001 for kernel upgrade |
[production] |
| 11:11 |
<oblivian@tin> |
Synchronized wmf-config/ProductionServices.php: switching mediawiki to use restbase in eqiad again (duration: 00m 32s) |
[production] |
| 11:04 |
<mobrovac> |
restbase restarted cassandra-a on restbase1007 |
[production] |
| 11:02 |
<gehel> |
restarting elasticsearch server elastic1008.eqiad.wmnet |
[production] |
| 11:00 |
<mobrovac> |
restbase restarted cassandra on restbase2004 - OOM |
[production] |
| 10:45 |
<moritzm> |
rebooting francium for kernel upgrade |
[production] |
| 10:28 |
<oblivian@tin> |
Synchronized wmf-config/ProductionServices.php: switching mediawiki to use restbase in codfw (duration: 00m 32s) |
[production] |
| 10:20 |
<moritzm> |
rebooting osmium for kernel upgrade |
[production] |
| 10:14 |
<_joe_> |
external traffic is now flowing through restbase in codfw |
[production] |
| 10:08 |
<_joe_> |
running puppet across eqiad varnishes to switch traffic of restbase eqiad => codfw |
[production] |
| 09:34 |
<volans> |
Copying data from db2009 to db2008 T130098 |
[production] |
| 09:02 |
<jynus> |
reimporting missing rows from production to labs (expect some lag during the day) |
[production] |
| 08:59 |
<gehel> |
restarting elasticsearch server elastic1007.eqiad.wmnet |
[production] |
| 08:44 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 29s) |
[production] |
| 08:44 |
<ori@tin> |
Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 39s) |
[production] |
| 08:26 |
<ori@tin> |
Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s) |
[production] |
| 08:25 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s) |
[production] |
| 07:48 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 24s) |
[production] |
| 07:06 |
<ori> |
Short-circuiting RefreshLinksJob::run() to bail if the root job timestamp is older than March 1st |
[production] |
| 06:46 |
<ori> |
Powercycled elastic1006; unresponsive. |
[production] |
| 06:11 |
<ebernhardson> |
rebooting elastic1006.eqiad.wmnet for kernel update |
[production] |
| 03:22 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Mar 17 03:22:55 UTC 2016 (duration 9m 33s) |
[production] |
| 03:14 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/277452 (labs only change, just keeping file in sync) (duration: 00m 27s) |
[production] |
| 03:13 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.17) (duration: 17m 44s) |
[production] |
| 02:38 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.16) (duration: 17m 36s) |
[production] |
| 00:57 |
<ebernhardson> |
rebooting elastic1005.eqiad.wmnet for kernel update |
[production] |
| 00:05 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable Flow by default in all talk namespaces on gomwiki (duration: 00m 28s) |
[production] |
|
2016-03-16
§
|
| 23:56 |
<ebernhardson> |
rebooting elastic1004.eqiad.wmnet for kernel update |
[production] |
| 23:47 |
<catrope@tin> |
Finished scap: bswiki namespace changes; MobileFrontend SWAT patches (duration: 05m 04s) |
[production] |
| 23:42 |
<catrope@tin> |
Started scap: bswiki namespace changes; MobileFrontend SWAT patches |
[production] |
| 23:15 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Namespace config changes for bswiki (duration: 00m 44s) |
[production] |
| 22:09 |
<ebernhardson> |
rebooting elastic1003.eqiad.wmnet for linux kernel upgrade |
[production] |
| 21:47 |
<csteipp> |
deployed patch for T123071 |
[production] |
| 21:18 |
<gehel> |
restarting elasticsearch server elastic1003.eqiad.wmnet |
[production] |
| 20:06 |
<mutante> |
mira - git pull in mw-staging |
[production] |
| 19:58 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.17 |
[production] |
| 19:57 |
<gehel> |
regenerating puppet SSL certificates for elasticsearch eqiad cluster |
[production] |
| 19:54 |
<mutante> |
rcs1001 - starting redis, disabling puppet (T130147) |
[production] |