|
2016-03-17
§
|
| 10:08 |
<_joe_> |
running puppet across eqiad varnishes to switch traffic of restbase eqiad => codfw |
[production] |
| 09:34 |
<volans> |
Copying data from db2009 to db2008 T130098 |
[production] |
| 09:02 |
<jynus> |
reimporting missing rows from production to labs (expect some lag during the day) |
[production] |
| 08:59 |
<gehel> |
restarting elasticsearch server elastic1007.eqiad.wmnet |
[production] |
| 08:44 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 29s) |
[production] |
| 08:44 |
<ori@tin> |
Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 39s) |
[production] |
| 08:26 |
<ori@tin> |
Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s) |
[production] |
| 08:25 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s) |
[production] |
| 07:48 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 24s) |
[production] |
| 07:06 |
<ori> |
Short-circuiting RefreshLinksJob::run() to bail if the root job timestamp is older than March 1st |
[production] |
| 06:46 |
<ori> |
Powercycled elastic1006; unresponsive. |
[production] |
| 06:11 |
<ebernhardson> |
rebooting elastic1006.eqiad.wmnet for kernel update |
[production] |
| 03:22 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Mar 17 03:22:55 UTC 2016 (duration 9m 33s) |
[production] |
| 03:14 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/277452 (labs only change, just keeping file in sync) (duration: 00m 27s) |
[production] |
| 03:13 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.17) (duration: 17m 44s) |
[production] |
| 02:38 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.16) (duration: 17m 36s) |
[production] |
| 00:57 |
<ebernhardson> |
rebooting elastic1005.eqiad.wmnet for kernel update |
[production] |
| 00:05 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable Flow by default in all talk namespaces on gomwiki (duration: 00m 28s) |
[production] |
|
2016-03-16
§
|
| 23:56 |
<ebernhardson> |
rebooting elastic1004.eqiad.wmnet for kernel update |
[production] |
| 23:47 |
<catrope@tin> |
Finished scap: bswiki namespace changes; MobileFrontend SWAT patches (duration: 05m 04s) |
[production] |
| 23:42 |
<catrope@tin> |
Started scap: bswiki namespace changes; MobileFrontend SWAT patches |
[production] |
| 23:15 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Namespace config changes for bswiki (duration: 00m 44s) |
[production] |
| 22:09 |
<ebernhardson> |
rebooting elastic1003.eqiad.wmnet for linux kernel upgrade |
[production] |
| 21:47 |
<csteipp> |
deployed patch for T123071 |
[production] |
| 21:18 |
<gehel> |
restarting elasticsearch server elastic1003.eqiad.wmnet |
[production] |
| 20:06 |
<mutante> |
mira - git pull in mw-staging |
[production] |
| 19:58 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.17 |
[production] |
| 19:57 |
<gehel> |
regenerating puppet SSL certificates for elasticsearch eqiad cluster |
[production] |
| 19:54 |
<mutante> |
rcs1001 - starting redis, disabling puppet (T130147) |
[production] |
| 19:42 |
<mutante> |
rcs1002 - starting redis-server |
[production] |
| 19:42 |
<mutante> |
rcs1001 - starting redis-server |
[production] |
| 19:32 |
<twentyafterfour> |
Deploying 1.27.0-wmf.17 to group1 wikis |
[production] |
| 19:31 |
<godog> |
restart pybal on lvs1005 T130143 |
[production] |
| 19:18 |
<legoktm> |
created urlshortcodes table on wikishared db for UrlShortener |
[production] |
| 19:16 |
<legoktm@tin> |
Synchronized wmf-config/CommonSettings.php: Configure UrlShortener extension in read-only mode (2/2) (duration: 00m 26s) |
[production] |
| 19:15 |
<legoktm@tin> |
Synchronized wmf-config/InitialiseSettings.php: Configure UrlShortener extension in read-only mode (1/2) (duration: 00m 28s) |
[production] |
| 19:14 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.17/includes/jobqueue/Job.php: 3da38ce (duration: 00m 37s) |
[production] |
| 19:12 |
<legoktm@tin> |
Finished scap: Building l10n cache for UrlShortener - T108557 (try #2) (duration: 63m 05s) |
[production] |
| 19:11 |
<gehel> |
regenerating puppet SSL certificates for elasticsearch codfw cluster |
[production] |
| 19:08 |
<gehel> |
restarting elasticsearch server elastic1002.eqiad.wmnet |
[production] |
| 19:02 |
<volans> |
restarted mysql on dbstore2002 and all replicas except x1, still investigating T130128 |
[production] |
| 19:01 |
<godog> |
restart pybal on lvs1011 T130143 |
[production] |
| 18:20 |
<mutante> |
repooling rcs1001 |
[production] |
| 18:18 |
<gehel> |
restarting elasticsearch server elastic1001.eqiad.wmnet |
[production] |
| 18:14 |
<mutante> |
rebooting rcs1001 |
[production] |
| 18:10 |
<mutante> |
depool rcs1001 |
[production] |
| 18:09 |
<legoktm@tin> |
Started scap: Building l10n cache for UrlShortener - T108557 (try #2) |
[production] |
| 18:09 |
<legoktm> |
deleting /var/lib/l10nupdate/caches/cache-1.27.0-wmf.12 on tin to free up some space |
[production] |
| 18:05 |
<mutante> |
repooling rcs1002 |
[production] |
| 17:59 |
<mutante> |
rcs1002 - traffic graph flat in ganglia, reboot |
[production] |