Written by:
Alexander Kjäll, Security Engineer at Schibsted
Stian Kristoffersen, Security Engineer at Schibsted
Ståle Pettersen, Head of Product and Application Security at Schibsted
There has been an increased focus on supply chain security lately, from the Solarwinds breach and Chrome extensions becoming malicious, to Google announcing the Open Source Security Foundation (OpenSSF). Supply chain security is a complex problem with many aspects to take into consideration as pointed out by Eric Rescorla’s recently published article “Notes on Addressing Supply Chain Vulnerabilities“. On Feb 9th, Alex Birsan published an article on how he managed to hack companies like Apple, Microsoft, Tesla, Uber, and many others, through publishing packages in public package repositories with the same name as internal packages inside these companies. The developers and package systems got “confused”, which made them pull in the public packages instead of the internal ones, so he coined the term “Dependency Confusion”. There are already threat actors exploiting this according to Sonatype.
Package managers like npm, NuGet, PIP, Maven, Gradle, Cocoapods, Gems and Composer can all end up in an insecure configuration where they are vulnerable to “dependency confusion”. Packages typically have full access to the system they are installed on, except Deno’s permission modeland Java’s Security Manager, making the impact severe. In addition, in many companies it is common practise to have a package repository manager (like Nexus or Artifactory) as an extra layer between the developer machine and local/remote repositories, adding an extra layer of confusion to the problem.
In this blog post, we will elaborate on how different package systems are affected by Dependency Confusion when combined with Artifactory, and how Schibsted’s Product & Application Security team, in collaboration with Schibsted’s Developer Foundations team, have tried to mitigate this risk in our internal Artifactory instance.
In order for any package system to be vulnerable, you need to have internal dependencies that are not published to the global repositories. If you do not pull in any private packages, you are not vulnerable.
We first show an example of npm; feel free to skip this section and go straight to the package management system you use.
Jump to description for each package system
Dependency Confusion and Artifactory
Artifactory serves as both a storage of packages, a cache of remote packages and a resolver of package download requests. A virtual repository can have any number of underlying local or remote repositories. This is where confusion can occur, as requesting a package from a virtual repository does not specify if the local or remote should be used. When a version is specified, Artifactory will provide the package based on the ordering configured in the virtual repository. If a version range is specified, Artifactory will provide the package with the highest version, regardless of the ordering configured in Artifactory.
In order to mitigate the Dependency Confusion attack we decided on a multi-layered defence. By making sure we own the upstream namespace or package (note that this leaks the name of your internal packages), and making sure Artifactory does not serve upstream packages that are supposed to only be available internally. Owning them upstream avoids being vulnerable when package managers decide to only resolve the specified dependency from the specified repository, and the child dependencies from the upstream repository (like previous versions of Ruby’s Gems). In addition, if no Artifactory is configured, package managers may try to fetch all packages from upstream, which potentially would fetch malicious packages.
In virtual repositories, local repositories are always resolved first in Artifactory. The order of the remotes should be in the order “most trusted” to “less trusted”. This only has effect when an exact version of the package is specified; then the ordering decides which repository that the package will be served from. If a version is not specified, ordering will not protect you, as Artifactory will serve the highest version available in any repository, regardless of ordering. So an attacker would publish version 99 (or 1.99 etc.), to make sure his package is always chosen.
Artifactory’s exclude and include patterns can be used to filter which packages should be provided by each repository. We recommend to implement exclude rules for all locally published packages, in case the upstream namespace/package is deleted. Our tool Artishock makes this process a bit more automated.
Deciding what to exclude is not straightforward. We have identified these common cases:
- Internal package without a public package with the same name
- Internal package that had the same name as a real public package
- Internal package that have been open-sourced and now exists both internally and externally
- Internal package that has been claimed upstream, but not in use
- Internal patch of open source project
We want to balance blocking external packages with the risk of breaking internal projects. This most often required manual review of each internally published package. We will now look at how to use Artishock for this.
Detailed example of npm and Artifactory
Our setup in Artifactory have three repositories for npm:
- npm-local
- npm-remote
- npm-virtual
- npm-local
- npm-remote
The local one is where our internal packages are published, the remote one is the cache of npmjs.com and the virtual one composed of the other two and is where you point your local npm configuration.
If the dependency in npm’s package.json have a version range, it will return the package with the highest version across all configured repositories, regardless of ordering in Artifactory, this is the Dependency Confusion attack.
For npm the long term solution is to migrate all our internal packages into scopes that we own in the central repository, but we had a lot of packages and needed a mitigation until that was in place. We tried to do this in two ways, register the names in the central register, and add exclude rules to our internal artifactory installation.
Example attack:
- The application “business-center-billing” depends on “babel-preset-internal”:”^0.1.1″ and “babel-preset-internal” is only available in our internal Artifactory and not registered in public npm.
- The attacker creates a malicious package: https://www.npmjs.com/package/babel-preset-internal with version 0.1.2
- Developer runs “npm update” to upgrade packages -> babel-preset-internal is upgraded to the malicious version 0.1.2 from public npm.
Artischock: an open-source tool to ease the work
The tool is available at https://github.com/schibsted/artishock.
First it’s useful to gen an overview of the npm repositories in Artifactory
$ artishock repo-ls –package-system npm
npm-virtual [npm:virtual]
– npm-local [npm:local]
– npm-remote [npm:remote:https://registry.npmjs.org/]
npm-local [npm:local]
npm-remote [npm:remote]
The goal is to exclude packages from npm-remote that we don’t want to interfere with npm-local. As this can break things it will probably come down to a manual review. We start out with getting the list of candidates to exclude then a couple of helper commands to make judgements afterwards.
$ artishock exclude-candidates –package-system npm –local npm-local
internal1-pkg
@internal1-scope/internal2-pkg
…
As you build your list of trusted packages, filter them out
$ artishock exclude-candidates –package-system npm –local npm-local –trusted trusted-remote-packages.txt –excluded excluded-remote-packages.txt
trusted-remote-packages.txt and excluded-remote-packages.txt will start out with any packages you already know about. The exclude-packages.txt will start out empty. The goal is that artishock exclude-candidates will return the empty list.
Local packages that do not exist in npmjs.com can be claimed and/or excluded locally. The following command will get the list of local npm packages, then query npmjs.com to see if they already exist. If you do not want to disclose your internal package names to npmjs.com you should not use this command (the –query-upstreamflag is to acknowledge this). For scoped packages the command checks if an organization with the same name is claimed, not if each package exists.
$ artishock not-claimed –package-system npm –local npm-local –query-upstream
internal1-pkg
@internal1-scope
…
To decide if a package should be excluded, it’s also useful to see if the remote version is in use locally. We can do this by looking for the package in the Artifactory cache for npm-remote (name npm-remote-cache). In our case these packages were most often internal patches of an upstream project, or a project that started as an internal project and then got open-sourced.
$ artishock cached –package-system npm –local npm-local –remote npm-remote
internal1-pkg
@internal1-scope/internal2-pkg
…
After you have excluded the relevant packages for npm-remote in Artifactory, we can partially verify it by checking if packages from npm-local that also exists in npmjs.com cannot be resolved by npm-remote (packages and scopes excluded that does not exist in npmjs.com will not show up)
$ artishock inferred-exclude –package-system npm –local npm-local –remote npm-remote –query-upstream
internal1-pkg
@internal1-scope/internal2-pkg
…
Dependency Confusion Forever
Keep in mind that configuring exclude/include rules and potentially claiming package names and scopes upstream is an ongoing effort. If developers publish new packages internally that are not covered by existing rules, the rules must be updated and potentially claimed upstream. Detecting new packages should be automated. The exclude-candidates commands should continue to remain empty
$ artishock exclude-candidates –package-system npm –local npm-local –trusted trusted-remote-packages.txt –excluded excluded-remote-packages.txt
Additional scenarios
In the case of Maven you might have several remote repositories rather than just one as in the npm case. To get a sense if a remote is in use you can get a summary (this is slow for anything but small repositories as it will iterate over all the files) with the following
$ artishock repo-stats –package-system maven –repo small-remote-cache
Total downloads: 4
Archive count: 4
Last downloaded: 2021-03-01T02:03:04.000Z
Last downloaded by: john.doe@example.com
Mitigations for each package system
No package system is the same, let’s go through how the different package systems are affected and how one can mitigate dependency confusion.
Node.js: NPM
Affected? Yes
NPM supports scoped (“@schibsted/example1-pkg”) and unscoped packages (“example2-pkg”). A scoped package belongs to an organisation, and only members of the organisation can publish packages to that scope. An unscoped package (“example2-pkg”) may or may not belong to an organisation.
Mitigation
If your internal packages published to Artifactory are scoped, you should make sure you also own the same scope on npmjs.com to block malicious parties from publishing packages with the same scope. If your internal scope is owned by another organisation on npmjs.com, we recommend creating a new scope on npmjs.com and migrating your internal packages to that scope internally. If your internal packages are unscoped, we recommend migrating them to a scope you own on npmjs.com. Migrating packages to a new scope internally may take some time, an option to protect yourself while working on the migration is to register the unscoped packages on npmjs.com.
Also make sure each project has a .npmrc where all internal scopes point to “npm-local” in Artifactory. Claiming the scope upstream protects against scenarios where .npmrc does not exist or is misconfigured.
Safe:
registry=https://artifactory.internal/artifactory/api/npm/npm-virtual/
@schibsted:registry=https://artifactory.internal/artifactory/api/npm/npm-local/
@another-example:registry=https://artifactory.internal/artifactory/api/npm/npm-local/
Potentially insecure npm config:
@schibsted:registry=https://artifactory.internal/artifactory/api/npm/npm-virtual/
There are several reasons why one should use “npm ci” instead of “npm install” in CI systems, but for dependency confusion, if package-lock.json is provided in the project and “npm ci” is used, your build system will not be compromised (assuming package-lock.json isn’t already in “compromised”).
Python: PIP and Poetry
Affected? Yes
PIP does not have any grouping/scoping, all packages are in a global namespace. If you have internal packages that are not reserved on pypi.org, you are vulnerable as long as “index-url” or “extra-index-url” can fetch remote packages (ie “pypi-virtual” in Artifactory).
Poetry is also affected.
Mitigation
Claim all internal package names on pypi.org. If the name is already in use, you need to rename your package to a unique name and publish that and update all of your applications to use the new package name.
In addition it is possible to set up exclude filters for pypi-remote in Artifactory to prevent pypi-virtual to fetch the remote version of internal packages. The package name should still be unique and reserved upstream to avoid things breaking in the future (ie. upstream package is removed).
Java: Maven / Gradle
Affected? Yes
If all of these conditions apply then you are vulnerable:
- You have multiple repositories configured or a virtual remote in Artifactory that can pull in remote and local packages.
- There is a remote repository configured that lets anyone claim group ids without verification or you use internal packages with a group id that’s not dns based.
- You specify your dependencies with a version range
Maven central no longer lets people register new single word group-id’s, and instead enforce that the group id should be DNS based on a domain that the user can show that they control.
For Maven there are several projects that have their own snapshot repository, and some require you to add their repository because their package is not in one of the large repositories. To prevent these smaller repositories from overreaching, the include rules in Artifactory can be used to limit the repository to some prefixes, e.g. com.example.
Please note that Dependency Confusion can also affect Gradle and Maven projects, without the use of an internal proxy like Artifactory, if you specify multiple repositories directly.
Mitigation
Migrate your internal group-ids to a domain based name for a domain that you control.
.NET: NuGet
Affected? Yes
If you pull packages from both public and private repositories, you are vulnerable if you do not own all upstream packages and/or ID prefix in NuGet Gallery for your internal packages.
Mitigation
Microsoft has published a mitigation guide for NuGet.
Ensure your nuget.config packageSources section starts with a <clear /> entry to remove any inherited configuration, and use a single <add/> entry for your private feed.
For NuGet Gallery: An ID prefix can be registered by publishers to restrict uploads to the public gallery. Packages under a registered prefix can only be uploaded by approved accounts, which also protects against public substitution attacks. This reservation should be done whether you intend to publish your packages to NuGet.org or not. Using a registered ID prefix for private packages helps ensure that an attacker cannot claim any of your names. Follow these instructions to reserve your ID prefix.
PHP: Composer
Affected? Yes
If your internal packages are published to a local repository in Artifactory and you do not own the vendor prefix on packagist.org, you are most likely vulnerable. We recommend following the mitigation steps below as a precaution. For a complete understanding of the issue, please read this article.
Mitigation
Make sure all your packages use a vendor prefix, then publish a package on packagist.org with the vendor prefix that your packages use. This makes you the owner of the prefix, and no attacker can publish packages under that vendor prefix. If the vendor prefix is already taken, migrate your packages to an available vendor prefix.
In Composer, do not use Artifactory “virtual”, but explicit specify internal and external repositories, and exclude the internal namespace in the external repository configuration:
“repositories”: {
“private-repo”: {
“url”: “https://artifactory.internal/artifactory/api/composer/php-local”
}
“packagist.org”: {
“url”: “https://artifactory.internal/artifactory/api/composer/github.com-php-remote”,
“exclude”: [“myprefix/*”]
}
}
Objective-C/Swift: Cocoapods
Affected? Yes
If you have multiple global “source” elements in your Podfile or a virtual remote in Artifactory that can pull in remote and local packages, you are vulnerable.
Mitigation
Podfile should specify source directly for all internal dependencies instead of specifying a global source:
Secure Podfile example:
source ‘https://github.com/CocoaPods/Specs.git’
pod ‘SchibstedInternal, :source => ‘git@internal-git.example:CocoaPods/Specs.git’
Artifactory does not support virtual repositories for Cocoapods, but one should exclude local packages in all the remote Cocoapods repositories in Artifactory.
In addition, all upstream package names should be claimed (no namespace concept exists for Cocoapods) as a second layer of defence.
Ruby: Gems
Affected? Yes
The security team responsible for rubygems.org did spot the packages used by the original researcher according to their blog post. If both internal and external source is specified in Gemfile you are vulnerable. Example of vulnerable config:
source ‘https://rubygems.org’
source ‘https://artifactory.internal’
Mitigation
Specify explicitly which package is fetched from each remote like this:
source “https://artifactory.internal” do
gem ‘internal-package’
end
Even with the safe specification as above, the dependencies to “internal-package” may be pulled from rubygems.org. Please upgrade to RubyGems 3.2.10 or later (released 15. Feb 2021), as it contains a fix which will try to resolve all child dependencies against the same source repository specified for the dependency (if it can’t be resolved, it will fall back to rubygems.org). Previously, dependencies to a specific source was resolved with the global source first, instead of the specified source. In Bundler 3, multiple global sources will not be allowed.
Docker
Affected? With specific Artifactory config, yes
Images not hosted on Dockerhub require FQDN as part of the image name and are therefore secure. But if Artifactory is configured as a virtual registry for Docker, it can be vulnerable since a FQDN is pointing to Artifactory’s proxy service which may pull images from Dockerhub or docker-local in Artifactory, depending on how Artifactory is configured.
Mitigation
We have not performed a full investigation how you may be affected when using the virtual Docker registry feature in Artifactory (safest is to avoid using virtual). That being said, if you use “docker-virtual” in Artifactory, make sure local has precedence over Dockerhub. This will make sure that if the image and exact version exists in local, it will be used. We recommend registering an Organization on Dockerhub for internal namespaces as a precaution.
Go: Go modules
Affected? Unlikely
Go dependencies includes FQDN to where the dependency is hosted. So in all common setups, they are not vulnerable. As a prefix of a commit is typically included in go.mod and hashes are verified against Go’s checksum database in the default setup. There could be configurations where Go is configured against a “virtual” repository in Artifactory combined with an insecure configuration (GOPROXY, checksums verification has been disabled through GONOSUMDB, GOPRIVATEand/or GOINSECURE is set) that could be vulnerable. We have not done a full investigation as we do not have a virtual for Go modules in our Artifactory instance.
Rust: Cargo
Affected? No
Artifactory doesn’t support the cargo packaging system for Rust, and as far as we know no other private repository software implements the virtual-repository feature that artifactory have. Instead you need to explicitly state the remote repository for all your private dependencies.
Dependency Confusion – wrapping up
We have looked at how the different package managers can be vulnerable to dependency confusion in combination with Artifactory and how to implement mitigations. As long as package managers do not require which repository each dependency should be fetched from, this is not a solved issue. In addition, the “virtual” features in package repository managers makes it even more difficult to mitigate without continuously keeping includes/exclude rules up to date. Sandboxing features in languages and package managers would be a welcome addition, to reduce impact of malicious packages.
We have started working on continuously monitoring packages published to our internal Artifactory, and alerts us if the namespace is not owned by us upstream. We would then need to manually evaluate if a new exclude rule should be added in Artifactory, and optionally also claim the package/namespace upstream.
Note that Artifactory 7.16.1 (to be released within a couple of weeks) and 6.23.13 adds a new repository flag named “Priority Resolution” (disabled by default), which will make it possible to not search remotes if a local repository has any version of the requested package. Enabling this feature will mitigate some of the Dependency Confusion issues discussed in the post (but not all).