201-250 of 10000 results (29ms)
2016-03-17 ยง
16:27 <urandom> restarting restbase on xenon.eqiad.wmnet (canary), to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/ [production]
16:27 <akosiaris> do a for i in mathoid citoid graphoid cxserver; do sudo confctl --tags dc=eqiad,cluster=sca,service=$i --action delete sca1002.eqiad.wmnet ; done. Same for sca1001.eqiad.wmnet [production]
16:18 <urandom> bootstrapping restbase1012-b.eqiad.wmnet : T125842 [production]
16:13 <_joe_> load testing done [production]
15:58 <_joe_> performing a load test on mw1239 [production]
15:44 <gehel> restarting elasticsearch server elastic1015.eqiad.wmnet [production]
15:32 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART II [[gerrit:277786]] (duration: 00m 25s) [production]
15:32 <thcipriani@tin> Synchronized wmf-config/ProductionServices.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART I [[gerrit:277786]] (duration: 00m 25s) [production]
15:31 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART II [[gerrit:277786]] (duration: 00m 25s) [production]
15:30 <thcipriani@tin> Synchronized wmf-config/ProductionServices.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART I [[gerrit:277786]] (duration: 00m 26s) [production]
15:22 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Portal namespace to ne.wikipedia [[gerrit:278009]] (duration: 00m 28s) [production]
15:15 <thcipriani@tin> Synchronized portals: SWAT: Bumping portals to master [[gerrit:277964]] (duration: 00m 29s) [production]
15:14 <thcipriani@tin> Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master [[gerrit:277964]] (duration: 00m 28s) [production]
15:09 <thcipriani@tin> Synchronized wmf-config/CirrusSearch-common.php: SWAT: Enable ICU Folding on greek wikipedia PART II [[gerrit:277477]] (duration: 00m 31s) [production]
15:08 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ICU Folding on greek wikipedia PART I [[gerrit:277477]] (duration: 00m 30s) [production]
15:07 <gehel> restarting elasticsearch server elastic1014.eqiad.wmnet [production]
14:27 <gehel> restarting elasticsearch server elastic1013.eqiad.wmnet [production]
14:09 <moritzm> rolling reboot of parsoid servers in eqiad for kernel upgrade [production]
13:42 <mobrovac> restbase resrting cassandra on restbase2004 - OOM again [production]
13:36 <gehel> restarting elasticsearch server elastic1012.eqiad.wmnet [production]
13:02 <gehel> restarting elasticsearch server elastic1011.eqiad.wmnet [production]
13:00 <moritzm> rolling reboot of parsoid servers in codfw for kernel upgrade [production]
12:38 <_joe_> switching all services back to eqiad [production]
12:25 <gehel> restarting elasticsearch server elastic1010.eqiad.wmnet [production]
12:19 <akosiaris> restart pybal on lvs2001 [production]
12:16 <akosiaris> restart pybal on lvs2002 [production]
12:05 <_joe_> switching cxserver to codfw [production]
11:44 <moritzm> rebooting acamar for kernel upgrade [production]
11:41 <gehel> restarting elasticsearch server elastic1009.eqiad.wmnet [production]
11:33 <_joe_> switching citoid to use codfw as well [production]
11:31 <moritzm> rebooting install2001 for kernel upgrade [production]
11:11 <oblivian@tin> Synchronized wmf-config/ProductionServices.php: switching mediawiki to use restbase in eqiad again (duration: 00m 32s) [production]
11:04 <mobrovac> restbase restarted cassandra-a on restbase1007 [production]
11:02 <gehel> restarting elasticsearch server elastic1008.eqiad.wmnet [production]
11:00 <mobrovac> restbase restarted cassandra on restbase2004 - OOM [production]
10:45 <moritzm> rebooting francium for kernel upgrade [production]
10:28 <oblivian@tin> Synchronized wmf-config/ProductionServices.php: switching mediawiki to use restbase in codfw (duration: 00m 32s) [production]
10:20 <moritzm> rebooting osmium for kernel upgrade [production]
10:14 <_joe_> external traffic is now flowing through restbase in codfw [production]
10:08 <_joe_> running puppet across eqiad varnishes to switch traffic of restbase eqiad => codfw [production]
09:34 <volans> Copying data from db2009 to db2008 T130098 [production]
09:02 <jynus> reimporting missing rows from production to labs (expect some lag during the day) [production]
08:59 <gehel> restarting elasticsearch server elastic1007.eqiad.wmnet [production]
08:44 <ori@tin> Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 29s) [production]
08:44 <ori@tin> Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 39s) [production]
08:26 <ori@tin> Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s) [production]
08:25 <ori@tin> Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s) [production]
07:48 <ori@tin> Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 24s) [production]
07:06 <ori> Short-circuiting RefreshLinksJob::run() to bail if the root job timestamp is older than March 1st [production]
06:46 <ori> Powercycled elastic1006; unresponsive. [production]