Xcalar has a native S3 connector, allowing datasets to be imported directly from S3 buckets using parallel streams across your EC2 nodes.
These steps must be performed on all EC2 VMs comprising the Xcalar Cluster.
Note: Xcalar Install Home = /opt/xcalar-home in this example. Please replace as necessary.
- Install AWS CLI
/opt/xcalar-home/opt/xcalar/bin/pip2.7 install awscli --upgrade --user
- Enter AWS account credentials provided by your AWS Administrator
[ec2-user@ip-172-31-8-232 ~]$ aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: us-west-2
Default output format [None]:
- Browse all buckets visible to this access key
[ec2-user@ip-172-31-8-232 ~]$ aws s3 ls s3://
- Xcalar will now be able to browse and import data from all these S3 buckets
This wiki article describes how to manage access to Amazon S3 resources in various scenarios: