We were contacted by a client that operates a school web software to migrate their old SFTP system to cloud-native solution
As an experienced DevOps consulting services company, ITGix has always been committed to providing innovative solutions and streamlining processes for our clients. Today, we would like to share a use case where we partnered with a client to migrate their old SFTP system to a more efficient and cloud-native solution. Through our expertise in AWS services and our commitment to delivering reliable and scalable solutions, we aimed to address their specific needs and optimize their workflow.
Let’s outline the reasons behind this migration. Our client operates a school web software that allows schools to upload user files within the application. They expressed the need for an additional feature where users can upload files separately, which would then be processed by the application.
PREVIOUS SETUP AND CHALLENGES:
Initially, an SFTP system was installed on a CentOS server, running on top of SSH, with an AWS Elastic File System (EFS) mounted. This EFS was accessible to the applications via NFS and PVC, allowing them to read user files and write additional report files. To track user file uploads and send them to a specific API for processing, we utilized the inotifywatch service. However, we encountered several issues with the inotifywatch service, such as frequent freezing and improper functionality. Additionally, the permissions of the files written by the applications on the EFS made it difficult for the users to access them.
After using this combination of services for almost a year and a half, we decided it was time to refactor the logic and implement a more cloud-native solution. Since the applications were already hosted on AWS, we explored AWS services that could assist us in achieving a reliable and highly available SFTP system, with secure uploads, user file tracking, and seamless integration with our API.
AWS Services Selection:
After conducting thorough research, we opted to use AWS S3 as the storage service for user files. S3 offers durable storage with a plethora of useful features. To track file events, S3 provides a notification feature that can trigger AWS Lambda, a serverless managed tool designed to process these notifications and send the files to the appropriate API. Additionally, AWS offers Transfer Family, an SFTP system that utilizes S3 as its underlying storage.
However, we discovered that the Transfer Family only supports SSH-key-based authentication, which posed a limitation for our specific requirements. Although a workaround is available that involves using a custom Lambda as an Identity Provider (IdP) and managing users in AWS Secrets Manager, we decided not to pursue this setup. Instead, we continued our search for an alternative solution that could provide both key-based and password authentication to ensure a seamless transition for our users.
Discovering SFTP Gateway:
Upon further research, we discovered SFTP Gateway, developed by Thorn Technologies. This versatile SFTP system can be utilized on various cloud providers, including AWS, Azure, and GCP. SFTP Gateway serves as an SFTP server that provides real-time access to files stored in cloud storage. One notable feature of SFTP Gateway is its support for both key-based and password authentication, making it a suitable solution for our requirements. Additionally, SFTP Gateway offers a REST API, further extending its functionality and providing flexible integration capabilities with other systems and applications. This REST API allows for seamless automation and custom development, enabling us to tailor the SFTP Gateway solution to our specific needs.
Implementation and Configuration:
Given our goal of implementing a highly available and reliable SFTP system on AWS, we carefully evaluated the deployment options provided by SFTP Gateway. After assessing the requirements and considering the desired level of availability, we opted for the high-availability deployment.
The high-availability deployment of SFTP Gateway offers a robust architecture that ensures continuous operation and fault tolerance. It consists of several key components, including a Network Load Balancer (NLB), Auto Scaling Groups (ASGs), PostgreSQL, and an additional private Application Load Balancer (ALB) for the management UI.
To handle the incoming SFTP traffic and distribute it among the backend instances, we utilized a Network Load Balancer (NLB). The NLB acted as the entry point for the SFTP system, intelligently routing traffic to multiple instances for load balancing and fault tolerance. This allowed us to achieve high availability and scalability, ensuring that the SFTP system could handle increased load and seamlessly scale as needed.
To further enhance the fault tolerance and resilience of the SFTP Gateway, we leveraged Auto Scaling Groups (ASGs). The ASGs managed the EC2 instances running SFTP Gateway, automatically adjusting the number of instances based on demand. This dynamic scaling capability ensured that the SFTP system could handle varying workloads while maintaining optimal performance and availability.
For the management of the SFTP Gateway and its users, we set up an additional private Application Load Balancer (ALB). This ALB served as the entry point for accessing the SFTP Gateway’s web admin UI, providing a secure and isolated interface for managing users and folders. By separating the management UI traffic from the SFTP traffic handled by the NLB, we ensured optimized performance, enhanced security, and streamlined management of the SFTP system.
As for the storage of relevant metadata and configuration settings, we utilized PostgreSQL. PostgreSQL served as the backend database for SFTP Gateway, providing a reliable and scalable solution. By leveraging the capabilities of PostgreSQL, such as data integrity, concurrency control, and replication, we ensured the durability and consistency of the SFTP system.
By choosing the high-availability deployment of SFTP Gateway, which involved the use of a Network Load Balancer, Auto Scaling Groups, PostgreSQL, and an additional private Application Load Balancer for the management UI, we were able to achieve a resilient and scalable SFTP system. This architecture provided the necessary redundancy, fault tolerance, and secure management interface to meet our client’s requirements for a highly available and reliable solution.
As part of the migration process, we removed the old Elastic File System (EFS) that was previously used by the applications. The developers were required to re-configure the application code to utilize Amazon S3 for reading and writing data. This transition involved updating the necessary code sections to interface with S3’s APIs, ensuring a seamless transition from the old EFS setup to the more efficient and scalable S3 storage.
We successfully set up the entire workflow, integrating S3, Lambda, and SFTP Gateway, which enabled us to handle the complete lifecycle of uploading files, triggering Lambda functions, and sending files to the API.
User Migration and Distribution of New Credentials:
Our next important steps involved migrating over 4000 SFTP users from our current system to the new SFTP Gateway. This migration presented several challenges that needed to be addressed to ensure a smooth transition. Firstly, we encountered the complexity of migrating passwords as we didn’t have direct access to the actual passwords. Another crucial requirement was to ensure that usernames remained the same, with only the password and host information being changed. This was important to maintain consistency for the end users and minimize any potential disruption caused by the migration.
Furthermore, as an additional security measure, the SFTP Gateway employed IP whitelisting. To facilitate this, we needed to collect the IP addresses from which clients would connect to the SFTP system. This information was necessary to whitelist the respective IP addresses before providing clients with the new credentials for accessing the system.
To begin the migration process, we extracted all the usernames from both our SFTP server and the application database. This data was then compiled into a CSV file. We developed a bash script that would read each line from this CSV file and create new SFTP Gateway users with randomly generated passwords. These new passwords were stored in another CSV file. Following the migration, we initiated communication with our client to discuss the distribution of the new credentials to the end users. To facilitate this, we utilized a Google Data Form, which was provided to the end users. The form requested information such as the user’s school, SFTP username, and the IP address from which they would be connecting.
To ensure a smooth and automated distribution process, we developed a Python script that would automatically send the new credential information to the end users. This script streamlined the distribution process, allowing for efficient and secure delivery of the new credentials. Throughout the entire migration process, effective communication and meticulous planning were key to ensuring a smooth transition for over 4000 users. By developing custom bash scripts for user migration and utilizing automated distribution methods, we were able to streamline the process and minimize disruption for our clients.
By refactoring our system using S3, Lambda, and SFTP Gateway, we achieved a highly available and reliable SFTP solution. The implementation process involved careful planning, effective communication with the client, and the successful migration of over 4000 users. The new system worked seamlessly, facilitating secure uploads and efficient processing of user files.