14 June 2019
Follow @igormakaCocoaPods 1.7.2 brings with it the final version of CDN support for the trunk specs repo.
It's been a tale of ridiculous scale, unintended consequences and free open-source plans. CocoaPods is moving to use a CDN which is located at https://cdn.cocoapods.org/ and is still considered experimental. A future release will make it the default spec source.
A CDN is a Content Delivery Network - what this means for CocoaPods is that using CocoaPods won't require you to have a local copy of all the public Podspecs on your computer. Saving you about a GB of file storage, and shaving a lot of time off pod install
s.
Scaling when free and open source
CocoaPods is a free, open-source project run by maintainers in their spare time. As such, it depends on the good graces and sponsorships of multiple tech companies. The initial decision was to store the specs in a GitHub repo, a common practice for a free, open-source project for storing such metadata.
The first hurdles of scale were encountered in 2016, when traffic to the master spec repo has caused GitHub to rate-limit all operations.
A comment from GitHub at the time stated that this repo alone was using 5 whole server CPUs and consuming terabytes upon terabytes of bandwidth.
The crisis was averted by sharding the repo to improve algorithmic performance and by limiting all first-time clones to being shallow.
This has solved GitHub's problem, but CocoaPods' users still had to deal with unsatisfactory performance.
Git as a Database
Git was invented at a time when "slow network" and "no backups" were legitimate design concerns. Running endless builds as part of continuous integration wasn't commonplace. CDN wasn't so commonly available.
There are many stories of git repos hitting the hard envelope of git's design constraints. Microsoft's Windows repo is one example. CocoaPods is another.
At the time of writing, the spec repo had:
- almost 400k commits - half of the Linux Kernel repo!
- over 400k directories
- over 300k spec files
These numbers have caused the repo to take minutes to clone, long times to update and have wasted uncountable CI time. It has also proven to be resistant to tarballing, as this sort of directory structure hits a bottleneck in tar
as well.
Directory Listing Denied
It was obvious to many that the spec repo should be put behind a CDN, but there were several constraints:
- It had to be a free CDN, as the project is free and open-source.
- It had to allow some way of obtaining directory listings, for retrieving versions of pods.
- It had to auto-update from GitHub as the source of truth.
The first implementation was a shell script, polling GitHub and piping find
into ls
into index files. This ran on a machine that was not open or free and therefore could not be the true solution. Nevertheless, this auto-updated repo was put behind a jsDelivr CDN and the client interfacing with it was released in 1.7.0 labeled "highly experimental".
Final Lap with Netlify
The final version of the CDN for CocoaPods/Specs was implemented on Netlify, a static site hosting service supporting flexible site generation. This solution ticked all the boxes: a generous open-source plan, fast CDN and continuous deployment from GitHub.
Upon each commit, Netlify runs a specialized script which generates a per-shard index for all the pods and versions in the repo. If you've ever noticed that the directory structure for our Podspecs repo was strange, this is what we call sharding. An example of a shard index can be found at https://cdn.cocoapods.org/all_pods_versions_2_2_2.txt. This would correspond to ~/.cocoapods/repos/master/Specs/2/2/2/
locally.
Additionally, we create an all_pods.txt
file which contains a list of all pods.
Finally, any other request made is redirected to GitHub's CDN.
Usage
Using the CDN for retrieving specs, while "finalized", should still be considered experimental until we switch it to be default.
To use the CDN source in your Podfile
:
If you don't have private specs:
source 'https://cdn.cocoapods.org/'
If you have private specs:
source 'https://github.com/artsy/Specs.git'
- source 'https://github.com/CocoaPods/Specs.git'
+ source 'https://cdn.cocoapods.org/'
Doing this will break your Podfile.lock
, so you are likely to need to run pod update
to see the changes (be careful, this may update your Pods also).
If you have a CI setup, it is recommended to cache the new repo dir as it is very small and would save even more time. With 1.7.2 it should be located at ~/.cocoapods/repos/cocoapods-
(yes, with a -
), but we're looking to improve the naming in an upcoming release.
In an upcoming release, we are planning to make the CDN source the default source!
P.S. Thanks to @dnkoutso for all the release work!
🚀