251-300 of 10000 results (29ms)
2016-03-17 ยง
22:04 <bd808@tin> Synchronized wmf-config/event-schemas: Add ApiRequest kafka logging (T108618) (duration: 00m 38s) [production]
21:47 <bd808> deleted /var/lib/l10nupdate/caches/cache-1.27.0-wmf.1[345] on tin. Freed ~4G of disk [production]
21:46 <gehel> reducing cluster.routing.allocation.disk.watermark.high to 70% on eqiad elasticsearch cluster [production]
21:07 <twentyafterfour@tin> rebuilt wikiversions.php and synchronized wikiversions files: all to 1.27.0-wmf.17 [production]
20:51 <twentyafterfour@tin> Finished scap: sync php-1.27.0-wmf.17 (duration: 26m 57s) [production]
20:40 <urandom> Issuing `nodetool scrub -s -- local_group_wikipedia_T_parsoid_html data` on restbase2004.eqiad.wmnet : T130254 [production]
20:24 <twentyafterfour@tin> Started scap: sync php-1.27.0-wmf.17 [production]
20:14 <urandom> Starting Cassandra on restbase2004.codfw.wmnet (OOM. again.) [production]
20:12 <urandom> Restarting Cassandra on restbase1007-a.eqiad.wmnet (compaction seems stalled) [production]
20:00 <gehel> restarting elasticsearch server elastic1019.eqiad.wmnet [production]
19:41 <gehel> restarting elasticsearch server elastic1018.eqiad.wmnet [production]
18:36 <cscott> updated OCG to version c1a8232594fe846bd2374efd8f7c20d7e97ac449 [production]
18:28 <moritzm> rebooting labnet1001 for kernel update [production]
18:23 <moritzm> rebooting labcontrol1002 for kernel update [production]
18:06 <moritzm> rebooting holmium for kernel update [production]
18:03 <moritzm> rebooting silver for kernel update [production]
18:00 <mobrovac> restbase restarted cassandra on restbase2004, OOM [production]
17:57 <gehel> restarting elasticsearch server elastic1017.eqiad.wmnet [production]
17:10 <urandom> rolling restart of restbase production complete [production]
16:51 <urandom> rolling restart of restbase production to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/ [production]
16:45 <urandom> performing restart of restbase1003.eqiad.wmnet (canary) to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/ [production]
16:41 <urandom> rolling restart of restbase staging complete [production]
16:41 <gehel> restarting elasticsearch server elastic1016.eqiad.wmnet [production]
16:35 <urandom> performing rolling restart of restbase staging to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/ [production]
16:27 <urandom> restarting restbase on xenon.eqiad.wmnet (canary), to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/ [production]
16:27 <akosiaris> do a for i in mathoid citoid graphoid cxserver; do sudo confctl --tags dc=eqiad,cluster=sca,service=$i --action delete sca1002.eqiad.wmnet ; done. Same for sca1001.eqiad.wmnet [production]
16:18 <urandom> bootstrapping restbase1012-b.eqiad.wmnet : T125842 [production]
16:13 <_joe_> load testing done [production]
15:58 <_joe_> performing a load test on mw1239 [production]
15:44 <gehel> restarting elasticsearch server elastic1015.eqiad.wmnet [production]
15:32 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART II [[gerrit:277786]] (duration: 00m 25s) [production]
15:32 <thcipriani@tin> Synchronized wmf-config/ProductionServices.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART I [[gerrit:277786]] (duration: 00m 25s) [production]
15:31 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART II [[gerrit:277786]] (duration: 00m 25s) [production]
15:30 <thcipriani@tin> Synchronized wmf-config/ProductionServices.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART I [[gerrit:277786]] (duration: 00m 26s) [production]
15:22 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Portal namespace to ne.wikipedia [[gerrit:278009]] (duration: 00m 28s) [production]
15:15 <thcipriani@tin> Synchronized portals: SWAT: Bumping portals to master [[gerrit:277964]] (duration: 00m 29s) [production]
15:14 <thcipriani@tin> Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master [[gerrit:277964]] (duration: 00m 28s) [production]
15:09 <thcipriani@tin> Synchronized wmf-config/CirrusSearch-common.php: SWAT: Enable ICU Folding on greek wikipedia PART II [[gerrit:277477]] (duration: 00m 31s) [production]
15:08 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ICU Folding on greek wikipedia PART I [[gerrit:277477]] (duration: 00m 30s) [production]
15:07 <gehel> restarting elasticsearch server elastic1014.eqiad.wmnet [production]
14:27 <gehel> restarting elasticsearch server elastic1013.eqiad.wmnet [production]
14:09 <moritzm> rolling reboot of parsoid servers in eqiad for kernel upgrade [production]
13:42 <mobrovac> restbase resrting cassandra on restbase2004 - OOM again [production]
13:36 <gehel> restarting elasticsearch server elastic1012.eqiad.wmnet [production]
13:02 <gehel> restarting elasticsearch server elastic1011.eqiad.wmnet [production]
13:00 <moritzm> rolling reboot of parsoid servers in codfw for kernel upgrade [production]
12:38 <_joe_> switching all services back to eqiad [production]
12:25 <gehel> restarting elasticsearch server elastic1010.eqiad.wmnet [production]
12:19 <akosiaris> restart pybal on lvs2001 [production]
12:16 <akosiaris> restart pybal on lvs2002 [production]