Remote Software Development Environment
At Shipsy, we are seamlessly scaling development and testing environments irrespective of a developer's base location and laptop capabilities
By Aman Ruhela & Ayush Saxena
You might have often found yourself struggling to set up new projects owing to scattered information and multiple dependencies, making it difficult for a new hire (developer) to understand which version to use. Then issues like inadequate CPU power of laptops slowing down software development processes makes it difficult to build more than one project at a time. Result? Massive productivity loss.
Another significant challenge that we frequently face is driving integration testing between different services. More often than not, all the integration issues are actually discovered post-development branch mergers. This is a very reactive approach to development and we wanted to change that.
Existing cloud technologies have made it extremely easy to get powerful remote machines at the click of a button at very economical rates. Our goal was to leverage cloud-based systems to shoulder heavy compute and storage requirements and enhance developers' productivity.
Here’s How We Achieved Our Goal in 7 Easy Steps
Step 1: Selecting VMs
We selected powerful virtual machines from AWS (you can choose any cloud provider e.g. Azure, GCP, etc.) and granted developers access to the same based on their team and project.
Step 2: Creating A User on The VM
Now whenever a developer joins a team, they fill a google sheet with their Bitbucket username, public SSH key, and the projects they are working on. This triggers a script that creates a user on the virtual machine.
Then we made a folder for this user and give them access to that folder only. The script also adds the user's SSH key to the 'authorized_keys' file, enabling the user to connect with the remote machine seamlessly.
add_user_with_ssh_key() {
adduser --disabled-password --gecos "${full_name},,," --quiet ${username}
mkdir /home/${username}/.ssh
touch /home/${username}/.ssh/authorized_keys
echo "$ssh_key" > /home/${username}/.ssh/authorized_keys
chown -R ${username}:${username} /home/${username}/.ssh
}
Step 3: Restricting CPU Utilization
An imbalance in resource utilization can pose challenges when implementing this strategy. But we figured out a way to address it. To ensure one particular user does not exhaust all the resources, we created a separate cgroup for each user (cgroups in simple terms allow you to restrict how much resource users in a particular cgroup can use). This ensures that isolation between the users is maintained. In other words, there is zero interference between activities done by concurrent users of a machine.
For more detailed information on cfs_quota_us, cfs_period_us and limit_in_bytes refer resource_management_guide.
add_cgroup_configs() {
echo -e "group ${username} {
cpu {
cpu.cfs_quota_us=20000;
}
memory {
memory.limit_in_bytes = 10240m;
}
}" >> /etc/cgconfig.conf
echo -e "\n ${username} cpu,memory ${username}" >> /etc/cgrules.conf
}
restart_cgroups_services() {
systemctl restart cgrulesgend.service
systemctl restart cgconfigparser.service
}
Step 4: Cloning Repositories
Then we created the SSH key pair and gave READ access to all repositories. This allowed us to clone the repositories of all the said user's projects in the 'user folder.' Post cloning, we changed the config file in the .git folder ensuring that any future commits/pushes are done on that particular user's behalf only.
The script will further do the following things:-
Install all the dependencies of the projects
Install basic software packages like git, node, python, docker, etc
Copy the configuration files for the projects
So, now the project is ready to be used on a virtual machine. But the question is how to make it easily accessible and simple to work with for the developers? Luckily the good guys at Remote Development have developed a great VS Code plugin to solve this problem.
def setup_project(username, projectName, githubID):
try:
bitbucketCreds = bitbucket_config()
print
gitCloneCommand=f"""
git clone https://{bitbucketCreds['username']}:{bitbucketCreds['password']}@bitbucket.org/dhr00v/{projectName}.git /home/{username}/{projectName}
"""
run_system_command(gitCloneCommand, "Could not clone repository")
changeLocalRepoOwnership=f"""
chown -R {username}:{username} /home/{username}/{projectName}
"""
run_system_command(changeLocalRepoOwnership, "Unable to change local repo ownership")
gitChangeUserCommand=f"""
sudo -H -u {username} bash -c "cd /home/{username}/{projectName} && git remote remove origin && git remote add origin https://{githubID}@bitbucket.org/<yourbitbucketaccountname>/{projectName}.git"
"""
run_system_command(gitChangeUserCommand, "Unable to change email")
print("Base Project Setup Completed")
fetch_config_from_system(username, projectName)
except Exception as e:
print(e)
Step 5: Installing A Simple Plugin
You need to install the plugin, do a common palette search for "Remote SSH:Open configuration file" and enter your details.
Host <your VM’s DNS/IP>
HostName <your VM’s DNS/IP>
User <your user name on the VM>
ForwardAgent yes
IdentityFile <location of public ssh key file on your computer>
Step 6: Connecting VS Code with the VM
After that, click on the Remote Development button, the one you will see on the left panel, to connect with the machine. Now your VS Code is connected to the virtual machine on the cloud. After these steps, even though it will appear as if a developer is working on her local system, in reality, it's the powerful cloud-based virtual machine she is working on.
Step 7: Executing Integration Testing
With all your developers working on the same machine, a back-end engineer can run the server on the VS Code (which is actually running on the virtual machine) and give the port number to the corresponding front-end engineer for integration testing. One does not need to first get their branches merged and deployed. When both the engineers are satisfied, then their PRs can be merged. This will significantly reduce the development time.
In a nutshell, at Shipsy, we are seamlessly scaling development and testing environments irrespective of a developer's base location and laptop capabilities. In other words, our engineering teams' work is powered by the cloud, but from a developer's perspective, they are just working VS Codes on their own laptops.
If you love to solve complex engineering problems like this one, we are hiring. Check out the open job positions here .