In our ongoing efforts to raise the bar and improve the security and integrity of our codebase, we've implemented a robust system of GPG signature verification and code ownership all the way back to the repo root/initial commit. This post will explain the key components of our approach and its benefits and implications.

Contributors can fork the pAI-OS/paios repo (git clone https://github.com/pAI-OS/paios.git) and create pull requests (with GPG signed commits!) from there, and more active active contributors can work on feature-* branches in the paios repo itself where it is more readily accessible to others. The first time we accept a pull request we'll need to do a quick video keysigning ceremony with a maintainer (currently myself and Karsten Wade) where you show ID and confirm compliance with CONTRIBUTING.md before we sign your key and have you publish it on a GPG keyserver (i.e., keyserver.ubuntu.com with gpg --keyserver keyserver.ubuntu.com --send-keys <0xabcd1234>).

This was inspired by:

• Debian: My work as a Debian developer (where new maintainers need to have their key signed by at least two existing developers before they can be added to the Debian keyring)
• CAcert: My work as Organisation Assurance Officer for community certification authority CAcert
• Apple: Apple's recent pioneering work on Private Cloud Compute (some 15 years into the transition from products to services that we call "cloud").
• Qubes OS: Security operating system Qubes OS' own code signing policy requiring that "All contributions to the Qubes OS source code must be cryptographically signed by the author’s PGP key."

Key Components​

1. CODEOWNERS File: The CODEOWNERS file defines who is responsible for different parts of the repository. This ensures that the right people are automatically requested for review when changes are requested.
2. GPG Signature Verification Workflow: We've set up a GitHub Actions workflow (verify-gpg-signatures.yml) that runs on pull requests, pushes to the main branch, and can be manually triggered. This workflow sets up the GPG environment and runs our custom verification script.
3. GPG Signature Verification Script: Our custom verification script (verify-gpg-signatures.sh) is the heart of our verification process. It imports trusted GPG keys from the gpg-keys, verifies commit signatures, checks if signing keys are trusted or signed by trusted keys, and reports any issues with commit signatures.

Benefits of This Approach​

1. Integrity: By requiring GPG signatures on all commits, we ensure - and you can verify - that all code changes are coming from verified contributors.
2. Accountability: The CODEOWNERS file clearly defines who is responsible for different parts of the codebase. It currently contains myself and Karsten Wade, a fellow long-time open source community member.
3. Automation: The GitHub Actions workflow automatically checks signatures on new pull requests and pushes.
4. Flexibility: The workflow can be manually triggered with a flag to check all commits in the repository.
5. Transparency: Any issues with signatures are clearly reported in the GitHub interface.

Implications for Developers​

We've recently migrated from SSH to GPG signatures for all commits in our history. As a result, anyone who cloned the pAI-OS/paios repo before July 15, 2024, will need to re-clone it. This is because the commit hashes, which are generated from the contents of the commit itself, have necessarily changed due to the rewriting of history to accommodate the new GPG signatures:

> git log --reverse --pretty=raw
commit adfcc54c511955d56576fa903eb07605e3c32c1a
tree fb3a2b360e4cb9ea6100006d56ca0bab863c520a
author Sam Johnston <[email protected]> 1712456042 -0700
committer Sam Johnston <[email protected]> 1712456042 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iHUEABYKAB0WIQQCg6PrpLqfl0rHX+kYjl3CelT6JQUCZpVWaAAKCRAYjl3CelT6
 JVHUAP0QTwdTC6yWA+yYirJ2GrYqB4J+rOiJ8EvZuv7DIYmnXwEAnHHn0/pXT/Ld
 mcCGKcFfpONKAoIe+wnYgMCsIRrmaAQ=
 =9KWo
 -----END PGP SIGNATURE-----

    Initial commit

Process​

The following command was used to rebase the repository to add GPG signatures to all commits, using the 'exec' option to run a script for each commit that does a git commit with the amend and no-edit flags as well as the -S flag to sign the commit with our GPG key (and --allow-empty and --empty=keep on the parent command to preserve a few empty commits in the log):

git rebase --empty=keep -X theirs --committer-date-is-author-date --exec '../sign-commit.sh' -i --root

The git command was originally inline but it wasn't picking up the commit date from the environment variables so it looked like months of work was done in seconds; the --committer-date-is-author-date would normally have done this but not when you specify your own command wtih --exec.

By using interactive (-i) mode I was able to change pick to drop for the two commits that made it in unsigned early on (and remove the following exec lines). Telling the rebase -X theirs has it automatically accept the incoming change rather than drop to merge conflict resolution.

#!/bin/sh

COMMIT_DATE=$(git show -s --format=%ci HEAD)
AUTHOR_DATE=$(git show -s --format=%ai HEAD)

GIT_COMMITTER_DATE="$COMMIT_DATE" GIT_AUTHOR_DATE="$AUTHOR_DATE" git commit --amend --no-edit --allow-empty -S

I now know more about git than I ever wanted to know, but not enough to be confident this was the best way to achieve the project's goals. Hopefully by sharing it I can save others hours of yak shaving.

Conclusion​

By implementing these security measures, we're taking significant steps to ensure the integrity and trustworthiness of our codebase. Fortunately it was early enough on in the project's life to be able to do with minimal disruption, though obviously it would have been better to start from the initial commit. While the new process may require some adjustment for contributors, the long-term benefits in terms of security and accountability are well worth the effort.

We encourage all contributors to set up GPG signing (not SSH!) for their commits and to familiarize themselves with our new verification process. Together, we can maintain a secure and trustworthy open-source project while raising the bar for the industry in light of increasingly frequent supply chain attacks.

One of the key design decisions we made in developing pAI-OS was that the frontend/s (by default, a single page web application written in React) and backend (a Python Flask app) are separate services. This separation of concerns allows us to have a more fine-grained control over the API security, and to be able to use the tools and techniques we already know best for each layer. By being strict about not using private APIs, you can be sure that all the features and functions you see in any interface are available to all of them.

In the ever-evolving landscape of API security, ensuring that all requests and responses are compliant with predefined specifications is increasingly important in a multi-layer defense strategy. At pAI-OS, we have adopted OpenAPI and "pattern" regexps with Connexion to achieve this goal without having to rely on third-party services (though you can upload your OpenAPI spec to providers like Cloudflare who will enforce it before requests even reach your server!).

Why OpenAPI and Connexion?

OpenAPI is a powerful specification for defining APIs, allowing us to describe our API endpoints, request/response formats, and validation rules in a standardized way. Connexion is a Python framework that automates the validation of requests and responses against an OpenAPI specification. By integrating these tools, we ensure that our API adheres to strict validation rules, reducing the risk of security vulnerabilities and improving overall reliability. This can exact a small performance penalty, but given this is an administrative interface rather than one that's invovled in user requests, the trade-off is worth it.

Using "pattern" Regular Expressions for Validation

One of the key features we leverage in our OpenAPI spec is the use of "pattern" regexps. These regular expressions allow us to define precise validation rules for various parts of our API, such as path parameters, query parameters, and request bodies. By specifying patterns, we can enforce constraints on the data being sent to and from our API, ensuring it meets our security and format requirements.

Examples

Filenames​

pAI-OS is designed to be cross platform, but different platforms accept different characters in filenames (and use different path formats too, which is why use pathlib.Path internally). We also use the following regexp to ensure that filenames are valid on macOS, Windows, and Linux:

fileName:
    type: string
    description: A filename that is valid on macOS, Windows, and Linux
    example: Mistral-7B-Instruct-v0.3-Q8_0.gguf
pattern: '^[^<>:;,?"*|/]+$'

UUIDs​

For scalability and security, we don't let users define the id of objects they create (via POST) or update (via PUT). Instead, we generate UUIDs for them. This way we can ensure that the id of an object is a valid UUID, that it follows the correct format, and that it's unique even if it's generated by one of several servers.

Rather than hoping users only ever send us valid UUIDs, we can be sure that they will by using the pattern regexp:

uuid4:
    type: string
    format: uuid
    example: 7bea4732-214f-40e7-9161-4e7241a2b97e
    pattern: ^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$

Emails​

For a more complex example one need look no further than an almost 100% RFC-compliant email regex:

email:
    type: string
    format: email
    example: [email protected]
    pattern: (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)_|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])_")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]\*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

Path Traversal Example

For example, here Connexion has intercepted an attempt at executing a path traversal vulnerability before reaching the rest of our code:

Conclusion

In summary, by using OpenAPI and Connexion, we can ensure that our API is secure and compliant with the latest standards. This approach allows us to have a more fine-grained control over the API security, albeit with a small performance penalty. By being strict about not using private APIs, you can be sure that all the features and functions you see in any interface are available to all of them.