By Aman Ruhela & Ayush Saxena

You might have often found yourself struggling to set up new projects owing to scattered information and multiple dependencies, making it difficult for a new hire (developer) to understand which version to use. Then issues like inadequate CPU power of laptops slowing down software development processes makes it difficult to build more than one project at a time. Result? Massive productivity loss.

Another significant challenge that we frequently face is driving integration testing between different services. More often than not, all the integration issues are actually discovered post-development branch mergers. This is a very reactive approach to development and we wanted to change that.

Existing cloud technologies have made it extremely easy to get powerful remote machines at the click of a button at very economical rates. Our goal was to leverage cloud-based systems to shoulder heavy compute and storage requirements and enhance developers' productivity.

Here’s How We Achieved Our Goal in 7 Easy Steps

Step 1: Selecting VMs

We selected powerful virtual machines from AWS (you can choose any cloud provider e.g. Azure, GCP, etc.) and granted developers access to the same based on their team and project.

Step 2: Creating A User on The VM

Now whenever a developer joins a team, they fill a google sheet with their Bitbucket username, public SSH key, and the projects they are working on. This triggers a script that creates a user on the virtual machine.

Then we made a folder for this user and give them access to that folder only. The script also adds the user's SSH key to the 'authorized_keys' file, enabling the user to connect with the remote machine seamlessly.

add_user_with_ssh_key() {
   adduser --disabled-password --gecos "${full_name},,," --quiet ${username}
   mkdir /home/${username}/.ssh
   touch /home/${username}/.ssh/authorized_keys
   echo "$ssh_key" > /home/${username}/.ssh/authorized_keys
   chown -R ${username}:${username} /home/${username}/.ssh
}

Step 3: Restricting CPU Utilization

An imbalance in resource utilization can pose challenges when implementing this strategy. But we figured out a way to address it. To ensure one particular user does not exhaust all the resources, we created a separate cgroup for each user (cgroups in simple terms allow you to restrict how much resource users in a particular cgroup can use). This ensures that isolation between the users is maintained. In other words, there is zero interference between activities done by concurrent users of a machine.

For more detailed information on cfs_quota_us, cfs_period_us and limit_in_bytes refer resource_management_guide.

add_cgroup_configs() {
   echo -e "group ${username} {
    cpu {
        cpu.cfs_quota_us=20000;
    }
    memory {
        memory.limit_in_bytes = 10240m;
    }
   }" >> /etc/cgconfig.conf
   echo -e "\n ${username} cpu,memory ${username}" >> /etc/cgrules.conf
}

restart_cgroups_services() {
   systemctl restart cgrulesgend.service
   systemctl restart cgconfigparser.service
}

Step 4: Cloning Repositories

Then we created the SSH key pair and gave READ access to all repositories. This allowed us to clone the repositories of all the said user's projects in the 'user folder.' Post cloning, we changed the config file in the .git folder ensuring that any future commits/pushes are done on that particular user's behalf only.

The script will further do the following things:-

• Install all the dependencies of the projects

• Install basic software packages like git, node, python, docker, etc

• Copy the configuration files for the projects

So, now the project is ready to be used on a virtual machine. But the question is how to make it easily accessible and simple to work with for the developers? Luckily the good guys at Remote Development have developed a great VS Code plugin to solve this problem.

def setup_project(username, projectName, githubID):
   try:
       bitbucketCreds = bitbucket_config()

       print
       gitCloneCommand=f"""
           git clone https://{bitbucketCreds['username']}:{bitbucketCreds['password']}@bitbucket.org/dhr00v/{projectName}.git /home/{username}/{projectName}
       """
       run_system_command(gitCloneCommand, "Could not clone repository")
       changeLocalRepoOwnership=f"""
           chown -R {username}:{username} /home/{username}/{projectName}
       """
       run_system_command(changeLocalRepoOwnership, "Unable to change local repo ownership")
       gitChangeUserCommand=f"""
           sudo -H -u {username} bash -c "cd /home/{username}/{projectName} && git remote remove origin && git remote add origin https://{githubID}@bitbucket.org/<yourbitbucketaccountname>/{projectName}.git"
       """
       run_system_command(gitChangeUserCommand, "Unable to change email")
       print("Base Project Setup Completed")
       fetch_config_from_system(username, projectName)

   except Exception as e:
       print(e)

Step 5: Installing A Simple Plugin

You need to install the plugin, do a common palette search for "Remote SSH:Open configuration file" and enter your details.

Host <your VM’s DNS/IP>
 HostName <your VM’s DNS/IP>
 User <your user name on the VM>
 ForwardAgent yes
 IdentityFile <location of public ssh key file on your computer>

Step 6: Connecting VS Code with the VM

After that, click on the Remote Development button, the one you will see on the left panel, to connect with the machine. Now your VS Code is connected to the virtual machine on the cloud. After these steps, even though it will appear as if a developer is working on her local system, in reality, it's the powerful cloud-based virtual machine she is working on.

Step 7: Executing Integration Testing

With all your developers working on the same machine, a back-end engineer can run the server on the VS Code (which is actually running on the virtual machine) and give the port number to the corresponding front-end engineer for integration testing. One does not need to first get their branches merged and deployed. When both the engineers are satisfied, then their PRs can be merged. This will significantly reduce the development time.

In a nutshell, at Shipsy, we are seamlessly scaling development and testing environments irrespective of a developer's base location and laptop capabilities. In other words, our engineering teams' work is powered by the cloud, but from a developer's perspective, they are just working VS Codes on their own laptops.

If you love to solve complex engineering problems like this one, we are hiring. Check out the open job positions here .