A comparison of binary repository management systems
Choosing the right tools for your critical infrastructure is a daunting task. To help you make an informed choice on binary repository management systems we took it upon ourselves to explore three leading solutions. So, how do Artifactory, Nexus and ProGet stack up?
The days when we could put our release on a shared drive or public FTP and still call ourselves professional software developers are over. We are completely capable of managing our source code by using Git in a mature ecosystem of repository managers like Bitbucket or GitHub. But when it comes to our products, the output of our builds, we are stumped. Binary repository managers (BRM) help with traceability by maintaining a single source of binary truth, and can even speed up the pipeline by eliminating redundant builds.
The BRM is where we put our DLLs, JARs and container images. A BRM can help you manage security and licenses by proxying for public package managers like pip or apt-get.
Jump to the different sections of this blog:
But which one should you choose? And how do you make that decision? We set out to investigate the pros and cons of the most popular binary repository managers. In our experiment we chose to compare Artifactory by JFrog, Nexus by Sonatype and ProGet by Inedo. They are all under active development, implement multiple repository types and provide support with a license.
The experiment was conducted in the most common environment - a one-node setup with a pro or basic license. This covers most of the use cases that we see at our customers.
We decided to compare the binary repository managers on the following criteria spanning the entire spectrum from Dev to Ops.
This gave us the broadest basis from which to elect a winner.
Easy to use and obtain trial licenses are generally a good sign. If it is a good product why wouldn’t we let you try it out? First fix is free.
JFrog allows you to register for a trial license online. A 30-day trial will be sent to your email and their sales department can extend this.
To acquire a 14 day trial for Nexus we had a lengthy email conversation with their sales department. This was a tedious manual process that we had to jump through again to extend the trial license. Do not expect to just be able to try out within a week.
After installing ProGet you choose between the free license and a 45-day enterprise evaluation license. To use their Docker image we had to create a user on their website to request a trial license.
ProGet wins for the fast and easy acquisition of a trial license, followed by Artifactory. Nexus really has some catching up to do in this category.
Most companies use multiple languages and platforms. The repositories supported by our test subjects can be found in the figure right below here. X= commercial support C= community support
Artifactory comes with a tremendous amount of repositories out of the box. JFrog does not allow user provided repositories.
SonaType has provided Nexus with OSGi interfaces that enable you to build custom repository types. The community already supports seven custom types (http://exchange.sonatype.com/list). Custom-built tooling isn’t for every organization, but for those with specific needs we have to applaud Nexus for extensibility.
ProGet comes with the least amount of repository types and no option to provide your own. inedo is open to support new repository types.
Artifactory pulls out ahead on this one. The repository types covered out of the box cover a majority of use cases. But be sure to check if your needs are satisfied in the table above.
With all of your binaries in one place the sheer number of artifacts can be breathtaking. Locating the exact artifact you need can be a challenge. Some organizations have workflows that are not supported by the default repository layouts, so flexibility is a strength.
Artifactory has custom layouts that allow you to make your own. Every repository has a type (Maven, NPM, etc) and a layout. Custom layouts should be cautiously adopted. Most of the clients (eg. Gradle) do not have the feature of discovering a non-standard layout.
Artifactory has good support for search through their web-based GUI and allows you to search on name, checksum, binary type, properties, and more. For the real complex searches Artifactory provides the Artifactory Query Language. AQL allows you to answer even the most advanced searches, but is limited to JFrog CLI and the REST API.
Nexus does not provide custom layouts, but it allows you to deploy out of layout. Deploying out of layout limits resolving those Artifacts through the REST and not through tools like Maven or Gradle. Their UI search features are on par with Artifactory offering multiple ways of defining your search, both on name, attributes and checksum.
ProGet does not have any notion of a layout. You upload the required files to ProGet and it will place them in the right hierarchical order according to the repository type. This is a clean implementation that reduces complexity, but obviously limits the customization. It is not possible to search across repositories in ProGet which heavily limits the usefulness of their search functionality. If you want to find an artifact you have to know where it is. In the trial we were unable to retrieve anything useful through the search bar.
Artifactory wins because of the many ways you can retrieve artifacts and because it provides custom layouts.
A useable web interface is great for human interaction, but in a modern software development pipeline most of the daily interactions will be running in pipelines. While all the test subjects support uploads through tools like Gradle, operations like repository creation and promotions are not yet standard equipment. This means that providing us with a REST API and CLI tools helps to bridge the gap that we see in the default setups.
Artifactory has a thoroughly documented REST API. It handles almost all possible operations, from everyday tasks like uploading artifacts, to updates of users, and configuring reverse proxies. Their UI also makes heavy use of the API, which is a testimony to its maturity.
JFrog also provide “JFrog CLI”, an application which can interact with their products, including Artifactory. Amongst other things JFrog CLI supports manipulation of artifacts and builds. JFrog supply example use cases, like putting the tool beside your MSBuild project and using the commands as part of your build process to upload the resulting artifacts.
The Artifactory plugin for Jenkins is very mature.
Nexus has an interactive API browser. This is quite nice to work with but their REST API is very lacking. The documentation is not comprehensive and having to guess whether a parameter is mandatory or what the return type is does not help the user experience of working with the API. Nexus also uses the idea of a pagination token meaning you can only go to the “next” or “previous” page - you can’t fast forward e.g. five pages of items. Theres is no way of limiting the number of results to get back either.
A more extensive documentation is available on the Sonatype website.
It is worth noting that Sonatype wrote in a blog post in March 2018 that they are currently working on a proper REST API that will eventually be added to Nexus 3.
ProGet has both up and download capabilities in the Jenkins plugin, and you can set both description and title when uploading (see).
Metadata is not supported in ProGet in any way. If you want to move your artifact from one repository to another you need to do this through the REST api (see).
Artifactory has the most mature and extensive REST api alongside many different plugins to support integration of the tool. We are eager to see what Nexus is building in the future.
Automated Pipelines in a Continuous Delivery flow can output a lot of binary artifacts. Being able to control what is kept and what is cleaned up is an essential requirement for being able to maintain your stack.
Artifactory’s primary means of cleanup relies on AQL to identify obsolete artifacts and then using the CLI or REST to delete them.
This makes it a bit harder compared with the others, but much more flexible in creating rulesets not only based on time or usage. For simple use-cases a snapshot repository can be used to limit the amount of releases for a specific version. (read more)
Nexus offers scheduled cleanup tasks for maven snapshot repository that let you define a filter specifying which artifacts to keep and which to delete. It also supports an “unused in x days” task for the rest that can be run defined by a cron-like string. (read more)
ProGet has the options of deleting all but the latest X amount of releases, delete unused versions, or in a regex style pattern. Triggering can be configured to happen after the repository grows larger than a certain size in megabytes.
ProGet wins with the best customizable inline solution. However ProGet offers no way of marking a version as “Do Not Delete” and requires a fail-forward style of development.
If that’s not for you then Artifactory gives the best solution, although you need a standalone service to perform the actual cleaning.
All products have their own user data and integrate with various other servers. Integration to Active Directory and similar services are crucial to avoid problems with multiple maintenance of user accounts.
Artifactory comes with a comprehensive suite of integrations: LDAP, Active Directory, OAuth on GitHub, Google, Cloud Foundry and OpenID. On top of that Artifactory supports SAML SSO, so custom providers can be configured. This allows for password managers like LastPass.
One caveat is that the GitHub integration gives login permission to everyone, on the instance, so it should only be used with a GitHub Enterprise, as providing everyone on GitHub.com with access is probably not what you are trying to do.
Nexus supports LDAP and Atlassian Crowd.
ProGet supports LDAP and Active Directory.
Artifactory wins due to the broadest lineup of integrations. In practice most companies use LDAP so this was almost a tie. It is an interesting move by Nexus to support Atlassian Crowd rather than Active Directory.
Restricting access is a must-have feature whether your company is still working out of a garage or has grown to enterprise scale. All three products have the notion of users and groups and roles, so let’s focus on the usability.
Artifactory has a permission scheme of users, groups and permissions. The users are the central component. Users belong to groups that have roles. Roles are permission sets such as Manage, Delete/Overwrite, Deploy/Cache, Annotate, or Read. These roles are associated with repositories to implement the permission scheme.
Nexus has made sure every fathomable combination of users, repositories and permission is available. This is not praise. It is one of the most cumbersome and complex access control systems we have ever worked with. Users have roles, and roles have privileges. There are hundreds of different privileges making it an act of black magic or brute force to figure out what access configuration enables a user to do on any specific task.
Nexus allows you to disable a user’s access.
ProGet has a permission scheme of users, groups and tasks, where tasks are the same as roles in other systems. Users can be created without assigning tasks, default is “none” i.e. anonymous.
There is a small preset group of tasks to define global admin, repository admin, upload and delete, and download possibilities.
If they do not fit your needs, new ones can be configured.
ProGet gets a small win over Artifactory. They both have more or less the same thing, but ProGet allows a bit more control over aspects such as viewing scheduled task and credential management.
Using the license that we are using in our experiment Artifactory only supports local storage. Artifactory has binary deduplication by hash-value allowing for multiple instances of the same file without bloating the filestore.
Artifactory has a very simple pricing model, but that unfortunately means that we have to pay a huge increase to get cloud storage. It is just one of the many benefits from the Enterprise solution. Most of our clients use the Artifactory Pro license.
In both the OSS and Pro versions of Nexus local storage and S3 is supported. Nexus has the concept of blob stores. Stores of both types can be active at the same time. A repository can only be in one store at a time, and artifacts can only be moved between repositories in the same blob store. This means that all the repositories for an artifact’s lifecycle need to be in the same store.
Nexus does not provide any deduplication even though this feature has been requested more than a year ago.
ProGet comes with local storage out of the box and Inedo provides S3 and Azure Blob storage extensions. They also offer a seven step guide for migrating local storage repositories to the cloud, but not the other way around.
ProGet does not provide any deduplication.
ProGet wins this close race as it provides the best solution out of the box. Nexus comes close with its capabilities for spanning multiple blob stores, but it comes with its limitations. For the biggest enterprises Artifactory’s enterprise license provides the best features.
Artifactory does not have a marketplace for plugins, but it provides a GitHub repository with many examples. The documentation for these plugins are of highly varying quality and depth. Artifactory provides extensibility through their event API which allows you to do nearly everything through plugins. Unfortunately, JFrog has not taken a big stance on best practices and leaves users to figure out their own solutions.
Nexus has some plugin-ability and has a marketplace. The interface for the marketplace is not great, but it allows filtering on product and version. There are about 25 plugins that are useable in Nexus.
The installation of plugins is a manual process. The plugins are built on OSGi containers (see).
To install a plugin you need to stop the Nexus service, edit XML configuration files, and download the plugin. In some cases you even need to build the plugin too. This process is painful and should be seen as code injection rather than plugin functionality.
ProGet actually achieves a plugin marketplace with click-to-install functionality requiring only a restart. While there are only a small selection of plugins, none of which support new repository types, it is by far the best experience of the three products.
ProGet wins for actually having a mature plugin platform.
If you have read this far it should be no surprise that Artifactory is our preferred choice. Artifactory supports the most repository types, and has the best REST API, both in terms of documentation and functionality. It also has the largest ecosystem of tools and integrations.
ProGet stems from the .NET world and is much more opinionated and restricted. If this fits your software development world then ProGet has got you covered.
Nexus has two things going for it. The amount of functionality you get in their OSS is impressive and it allows you to script your own repository types. While this is a large thing to undergo having this flexibility is a nice idea.
If you have any comments, please do not hesitate to write us.
Delivering Agile Transformations
A morning seminar on June 4th in Copenhagen
Introduction to Artifactory
Learn how to interact with Artifactory and get hands-on experience
What do we need from a CI/CD system? How should we decide which one to use? In this blog we ask ourselves what a modern CI/CD system should look like and compare two commonly used build systems: Jenkins pipelines and Concourse CI.
Concourse vs. Jenkins
An in-depth comparison of two CI/CD servers
A practical guide to writing, testing and publishing custom Gradle plugins, complete with a demo repository for a running start!
How to write, test and publish a custom Gradle plugin
Kickstart your plugin development with a documented example
Two Nordic DevOps heavyweights join forces - Eficode and Praqma
We will shape the future of DevOps together
What testing steps should you include in your Continuous Delivery pipeline? Don’t just string together existing manual processes - use simple, collaborative tools to design something better!
Pipeline - The Game that Delivers!
A new card game to design Continuous Delivery pipelines
In the Accelerate book, researchers explain several metrics which they have shown will measure the performance of a DevOps organization, and crucially, drive performance of the organization as a whole. I will explain why this is important, using an analogy with your risk of a heart attack.
Winning with DevOps and Reducing Your Risk of a Heart Attack
Clinical Trials and Software Process
Your repositories are cluttered, you’re storing irrelevant builds, and your disks are full. Luckily, you’ve come to the right place for advice. Here’s how to set up automatic cleanup.
Artifactory Retention Policies
How to set up automatic Artifactory repository cleaning
In many security and safety critical environments there can be requirements for long-term reproducibility of artifacts that are created. In this talk from NDC Tech Town, Mike Long shows how to use Docker to create binary reproducible applications from a C++ application.
Binary Reproducible Builds
How to Create Binary Reproducible Builds using Native Build Tools and Docker
In distributed architectures networking is usually taken as a given. However, when it does not function as we expect it is often hard to understand the reason. This blog post outlines some general insights into networking with more specific techniques for understanding the networking infrastructure of a Kubernetes cluster.
Debugging Kubernetes Networking
How to Find a Needle in a Virtualized Network
Choosing the perfect CI/CD tool for your project can be tricky. In this post we compare two managed CI/CD services - CircleCI and Google Cloud Build. Sami Alajrami is on hand to score our exciting CI/CD matchup.
CircleCI vs Google Cloud Build
How to choose the CI/CD tool that’s right for you
Simply aligning your IT capability with your business goals isn’t enough to grow your company. To achieve real IT-enabled growth you’ll need to build on a foundation of technical excellence and thanks to DORA that’s something you can now actually measure.
Avoiding the Agile Alignment Trap
Why a new software process won’t fix your delivery effectiveness
Hear about upcoming events in Scandinavia, latest tech blogs, and training in the field of Continuous Delivery and DevOps