GSOC 2020 Final Assessment - CD Foundation []

GSOC 2020 had been a wonderful experience where I got to work on really interesting projects and tasks. Firstly, I would like to thank my mentors, Jithin Emmanuel and Tiffany Kyi for all the technical help, clearing roadblocks and important decision-making throughout the project, both before and during the programme. The same goes to all the PR reviewers from the organization for giving useful insights in the task.

For GSOC, there were two projects to be taken up:

  • Generate and use deploy keys Issue link: Previously, for building the projects Screwdriver either needed the repos to have public access or required the SCM Tokens to be added as a configuration. For private repositories, the SCM tokens gave user wide access that is not always desirable. In this case, Deploy Keys is a better option that would only give repository wide access. This project required the inclusion of optional generation and addition of the keys and use them for checkouts.
  • Notify SCM changes from external SCM repositories Issue link: Previously Screwdriver only supported addition of webhooks to the source repositories. This project required the addition of support to add webhooks to external repositories and trigger jobs which have specifically subscribed to them.

Work Summary#

Milestones achieved:

  • Generate and use deploy keys in pipeline build mechanism.
    • Implementation of 2 flows:
      • Manual generation of the keys by user and addtion to the pipeline and SCM Repo.
      • Automatic generation and addtion of public key to the repo with Github-API as well as addition of private key as a secret. Second level validation to activate this is setting a flag in the API config and UI.
    • Unit Tests and Feature tests for all the features.
  • Notification from external SCM repositories:
    • Refactor to add webhooks to multiple repos with custom events.
    • Add logic to fetch triggered pipelines from the datastore during subscribed hook events.
    • Add logic to select and trigger specific subscribed jobs during subscribed hook events.

Pull Requests#

Pre GSOC work:#


I feel I’ve have completed almost all of the goals mentioned in the PR. The Generation and use of Deploy Keys PRs are all merged and the feature has been published. For the Notification from external SCM repositories, all of the PRs have been made and some have been approved. The feature tests are the ones that are left in the implementation and are in WIP. But those being similar to lot of existing tests can be easily added.

Future Work#

A couple of enhancements were discussed to be added to both the products and were decided to be taken up as secondary goals. Some of them were:

  • Implement rotation of deploy keys that would require enabling of storage of the keys in the data store in a secure way.
  • Add label based selector of jobs that would subscribe to external pipelines.