Day 89 of 90 Days of DevOps Challenge: Mounting an AWS S3 Bucket on Amazon EC2 Linux Using S3FS
Amazon S3 (Simple Storage Service) is a popular object storage service that provides developers and IT teams with secure, durable, and highly-scalable storage for their data. In this blog post, we will walk through the steps to mount an AWS S3 bucket on an Amazon EC2 Linux instance using S3FS, which allows you to treat your S3 bucket as a local filesystem.
Project Overview
This project will provide a hands-on experience with key AWS services, including EC2 and S3, while also utilizing S3FS for seamless integration. By the end of this project, you will have mounted an S3 bucket on your EC2 instance and learned how to interact with the bucket using standard file commands.
Key Components
1. AWS S3
Amazon S3 is an object storage service designed to store and retrieve any amount of data from anywhere on the web. It offers high availability, security, and performance.
2. Amazon EC2
Amazon EC2 (Elastic Compute Cloud) is a web service that provides resizable compute capacity in the cloud, allowing you to run applications and services in a flexible and scalable environment.
3. S3FS
S3FS is a FUSE (Filesystem in Userspace) based filesystem that allows you to mount an S3 bucket as a local filesystem on Linux. This provides easy access to S3 bucket files using standard file commands.
Implementation Steps
Step 1: Create IAM User and Set Policies
Create IAM User: Log in to your AWS Management Console and navigate to the IAM service to create a new user. Assign permissions that allow access to S3, such as the
AmazonS3FullAccess
policy or a custom policy that meets your project requirements.Generate Access Keys: During user creation, ensure you generate and save the access key ID and secret access key, as you will need these for authentication with the AWS CLI.
Step 2: Install AWS CLI
Connect to Your EC2 Instance: Launch an EC2 instance (preferably a Linux instance), and SSH into it:
ssh -i <your-key.pem> ec2-user@<your-ec2-instance-ip>
Install AWS CLI: Install the AWS CLI on your EC2 instance if it's not already installed:
sudo yum install -y aws-cli
Configure AWS CLI: Run the following command to configure the AWS CLI with your IAM user credentials:
aws configure
Enter your Access Key ID, Secret Access Key, default region, and output format when prompted.
Step 3: Install S3FS
Install Dependencies: Before installing S3FS, ensure you have the necessary packages:
sudo yum install -y gcc-c++ make sudo yum install -y fuse
Install S3FS: You can install S3FS from the EPEL repository or compile it from source. To install it from the EPEL repository:
sudo amazon-linux-extras install epel -y sudo yum install -y s3fs-fuse
Step 4: Mount the S3 Bucket
Create a Directory: Create a directory where you want to mount the S3 bucket:
mkdir ~/s3bucket
Create a Password File: Create a password file to store your AWS credentials for S3FS. Make sure to restrict permissions to this file:
echo "<your-access-key-id>:<your-secret-access-key>" > ~/.passwd-s3fs chmod 600 ~/.passwd-s3fs
Mount the S3 Bucket: Use the following command to mount your S3 bucket:
s3fs <your-bucket-name> ~/s3bucket -o passwd_file=~/.passwd-s3fs -o allow_other
Replace
<your-bucket-name>
with the name of your S3 bucket.Verify the Mount: You can verify if the bucket is mounted by listing the contents:
ls ~/s3bucket
Step 5: Run the Project and Share
Now that you have successfully mounted the S3 bucket on your EC2 instance, you can interact with it like any other directory on your system. You can upload, download, and manage files using standard file commands such as cp
, mv
, and rm
.
Upload a File: For example, to upload a file to your S3 bucket:
echo "Hello, S3!" > ~/s3bucket/hello.txt
Check on S3: You can log in to your AWS Management Console, navigate to S3, and check if the
hello.txt
file appears in your bucket.Share on LinkedIn: After completing the project, don’t forget to share your success on LinkedIn and showcase your newfound skills with AWS and S3FS!
Conclusion
By following the steps outlined in this blog post, you can effectively mount an AWS S3 bucket on an Amazon EC2 Linux instance using S3FS. This project illustrates the power of AWS services and how they can be integrated to create scalable and efficient storage solutions.