Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter

Upon upgrading elasticsearch to 2.1.0 I tried to do an export by pulling in a lot of records in one go and got the following error:

Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter

I suppose the proper fix would be to modify the code to use the scroll API, but for a quick fix, you can use the following:

 curl -XPUT http://1.2.3.4:9200/index/_settings -d '{ "index" : { "max_result_window" : 1000000}}'

Reference:

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html

elasticsearch 2.1.0 and "Unable to get effective rights from ACL: Overlapped I/O operation is in progress" error

When starting elasticsearch 2.1.0 on Windows 7 I got the following error below:

Likely root cause: java.io.IOException: Unable to get effective rights from ACL: Overlapped I/O operation is in progress.

        at sun.nio.fs.WindowsFileSystemProvider.getEffectiveAccess(WindowsFileSystemProvider.java:344)
        at sun.nio.fs.WindowsFileSystemProvider.checkAccess(WindowsFileSystemProvider.java:397)
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:246)
        at org.elasticsearch.bootstrap.Security.addPath(Security.java:227)
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:191)
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:184)
        at org.elasticsearch.bootstrap.Security.configure(Security.java:105)
        at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:196)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

When I checked the java version being used I found it was using jdk1.7.0_21

echo %JAVA_HOME%

Upgrading to jdk1.8.0_65 resolved the issue

VCFtools and Tabix installation on Linux (RHEL)

VCFtools installation

Download vcftools from here: https://github.com/vcftools/vcftools/zipball/master
 Unzip the downloaded package and proceed to install RHEL packages required to build vcftools using the following:

 

yum install autoconf

yum install automake

yum install gcc

yum group install “Development Tools”

yum install zlib

yum install zlib-devel

export PERL5LIB=/path/to/your/vcftools-directory/src/perl/

cd vcftools/
./autogen.sh
./configure
make
make install

The binary executables will be installed in the /usr/local/bin folder.

To add the executables to your path use the following:

export PATH=$PATH:/usr/local/bin

To set it permanently add the following line to the /etc/profile file just before it gets exported:

PATH=$PATH:/usr/local/bin:/usr/local/tabix

Tabix installation

(1) Go here to download the newest release.

(2) Extract the file:

tar xvjf tabix-0.2.6.tar.bz2 (3) Compile the program by typing make on the UNIX command line. (4) Export the path by adding the following line to your .bashrc file, saving your .bashrc file, and typing source on the UNIX command line.  Note: path_to_tabix is the directory where tabix is installed.

export PATH=$PATH:/path_to_tabix/tabix-0.2.6

 

References:

https://vcftools.github.io/examples.html
http://genometoolbox.blogspot.com.au/2013/11/installing-tabix-on-unix.html