-
-
Save copiousfreetime/1055823 to your computer and use it in GitHub Desktop.
Revisions
-
copiousfreetime revised this gist
Jun 30, 2011 . 1 changed file with 3 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -84,7 +84,9 @@ In addition to dependency management another interesting use of DNS is to provid * Root delegates to .org name servers * .org name servers delegate to rubygems.org name servers * rubygems.org name servers can either respond to the query or delegate to another set of name servers. It'll answer in this case. * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org and all NAPTR records for 1.0.3.rails.index.rubygems.org, for example: rails.index.rubygems.org. 600 CNAME 1.0.3.rails.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://rubygems.org/rails-3.0.1.gem!i" . -
copiousfreetime revised this gist
Jun 30, 2011 . 1 changed file with 3 additions and 3 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -86,9 +86,9 @@ In addition to dependency management another interesting use of DNS is to provid * rubygems.org name servers can either respond to the query or delegate to another set of name servers. It'll answer in this case. * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org and all NAPTR records for 1.0.3.rails.index.rubygems.org, for example: rails.index.rubygems.org. 600 CNAME 1.0.3.rails.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://rubygems.org/rails-3.0.1.gem!i" . 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 20 "U" "TCP+http" "!^.*$!http://backup.rubygems.org/rails-3.0.1.gem!i" . Note that there is no need to do any complex regex translation to get the various URLs since they are mapped directly to the canonical name of the gem. -
copiousfreetime revised this gist
Jun 30, 2011 . 1 changed file with 17 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -57,6 +57,23 @@ question, which is always found as the CNAME of that gemname 0.0.1.amalgalite.index.rubygems.org 600 PTR fastercsv.index.rubygems.org 0.0.1.amalgalite.index.rubygems.org 600 PTR arrayfields.index.rubygems.org Development Dependencies ------------------------ All of the above dependencies are assumed to be runtime. If using the gem command you typed: gem install --development amalgalite Then that would install all of amalgalite's development dependencies. To facilitate this same functionality we will add an additional PTR records for all the development dependencies using 'gemname-development' as the namespace. amalgalite.index.rubygems.org 600 CNAME 0.0.1.amalgalite.index.rubygems.org 0.0.1.amalgalite-development.index.rubygems.org 600 PTR 8.0.rake.index.rubygems.org 0.0.1.amalgalite-development.index.rubygems.org 600 PTR 2.1.configuration.index.rubygems.org 0.0.1.amalgalite-development.index.rubygems.org 600 PTR 5.2.rspec.index.rubygems.org Downloads --------- -
aeden revised this gist
Jun 17, 2011 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -10,7 +10,7 @@ Dependency Resolution I'd like to propose an alternate way of indexing RubyGems: let's use DNS. Here's how this might work. For this example, I want to get the latest version of Rails, which is 3.0.1 (in this scenario): * Client sends question to local name server for ALL records at rails.index.rubygems.org * Local name server does not have the record so it sends the usual response indicating that the search should go upstream to the roots -
copiousfreetime revised this gist
Jan 23, 2011 . 1 changed file with 28 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -30,6 +30,33 @@ Here's how this might work: Note that some PTR records represent canonical gem names and others would be a CNAME pointing to the appropriate canonical version. The last record is an example of this where the CNAME record would likely resolve to something like 7.0.1.bundler.index.rubygems.org (which would be the reverse notation for bundler-1.0.7) This also allows for ~>, = and >= support, for instance in the Amalgalite 1.0.0 gem has runtime dependencies of * arrayfields ~> 4.7.4 * fastercsv ~> 1.5.4 This can be modeled with the following set of records amalgalite.index.rubygems.org 600 CNAME 0.0.1.amalgalite.index.rubygems.org 0.0.1.amalgalite.index.rubygems.org 600 PTR 5.1.fastercsv.index.rubygems.org 0.0.1.amalgalite.index.rubygems.org 600 PTR 7.4.arrayfields.index.rubygems.org It is not exactly the same, but close enough, the 5.1.fastercsv.index.rubygems.org would then be a CNAME record for the latest 1.5.x version of fastercsv. for a = dependency, they would be: amalgalite.index.rubygems.org 600 CNAME 0.0.1.amalgalite.index.rubygems.org 0.0.1.amalgalite.index.rubygems.org 600 PTR 4.5.1.fastercsv.index.rubygems.org 0.0.1.amalgalite.index.rubygems.org 600 PTR 4.7.4.arrayfields.index.rubygems.org And for a >=, they would be dependent on the most recent release of the gem in question, which is always found as the CNAME of that gemname amalgalite.index.rubygems.org 600 CNAME 0.0.1.amalgalite.index.rubygems.org 0.0.1.amalgalite.index.rubygems.org 600 PTR fastercsv.index.rubygems.org 0.0.1.amalgalite.index.rubygems.org 600 PTR arrayfields.index.rubygems.org Downloads --------- @@ -86,4 +113,4 @@ Reference * [Nick Q Blog Post](http://robots.thoughtbot.com/post/2729333530/fetching-source-index-for-http-rubygems-org) * [RFC 1035](http://tools.ietf.org/html/rfc1035) * [RFC 3401](http://tools.ietf.org/html/rfc3401) * [NAPTR Record](http://en.wikipedia.org/wiki/NAPTR_record) -
aeden revised this gist
Jan 18, 2011 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -65,8 +65,8 @@ DNS provides the tools necessary to make this a decentralized system if we desir * .org name servers delegate to rubygems.org name servers * rubygems.org name servers respond with the following NS record: rails.index.rubygems.org. 600 NS ds1.rubyonrails.org rails.index.rubygems.org. 600 NS ds2.rubyonrails.org * The question is then sent to one of the two name servers which responds with a CNAME record pointing rails.index.rubygems.org to 1.0.3.rails.index.rubyonrails.org. * The rubyonrails.org name servers would then respond as shown in the scenarios above. -
aeden revised this gist
Jan 18, 2011 . 1 changed file with 4 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -51,6 +51,10 @@ Note that there is no need to do any complex regex translation to get the variou Other Considerations -------------------- ### Platforms To support multiple platforms (i.e. jruby) the client will first try platform.z.y.x.gemname.index.rubygems.org. If this is not found then the client should use z.y.x.gemname.index.rubygems.org. If a platform gem is provided then CNAME records will also need to be provided for all of the variations, i.e platform.y.x, platform.x and platform. ### Decentralization DNS provides the tools necessary to make this a decentralized system if we desire. This would be accomplished by delegating responsibility for gem names out to different DNS servers other than the rubygems.org servers. For example, if responsibility for management of the Rails gem metadata was decrentralized then the interaction might look like this: -
aeden revised this gist
Jan 18, 2011 . 1 changed file with 3 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -5,6 +5,9 @@ Currently the RubyGems index is stored as a Gzip file that is a marshalled array Additionally the gems themselves are currently centralized since there is nothing in the meta data that indicates where the gem should be downloaded from. However in order to allow this it is important to find ways of keeping the index from being poisoned (is this an issue in the centralized system?) Dependency Resolution --------------------- I'd like to propose an alternate way of indexing RubyGems: let's use DNS. Here's how this might work: -
aeden revised this gist
Jan 18, 2011 . 1 changed file with 11 additions and 13 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -9,18 +9,14 @@ I'd like to propose an alternate way of indexing RubyGems: let's use DNS. Here's how this might work: * Client sends question to local name server for ALL records at rails.index.rubygems.org * Local name server does not have the record so it sends the usual response indicating that the search should go upstream to the roots * Root delegates to .org name servers * .org name servers delegate to rubygems.org name servers * rubygems.org name servers can either respond to the query or delegate to another set of name servers. It'll answer in this case. * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org and all PTR records for 1.0.3.rails.index.rubygems.org. For example: rails.index.rubygems.org. 600 CNAME 1.0.3.rails.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activesupport.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.actiopack.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activerecord.index.rubygems.org. @@ -29,19 +25,21 @@ Scenario 1: Client wants to get the dependencies for the gem 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.railties.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 1.bundler.index.rubygems.org. Note that some PTR records represent canonical gem names and others would be a CNAME pointing to the appropriate canonical version. The last record is an example of this where the CNAME record would likely resolve to something like 7.0.1.bundler.index.rubygems.org (which would be the reverse notation for bundler-1.0.7) Downloads --------- In addition to dependency management another interesting use of DNS is to provide references to where gems can be downloaded. Here is how this might work: * Client sends question to local name server for ALL records at rails.index.rubygems.org * Local name server does not have the record so it sends the usual response indicating that the search should go upstream to the roots * Root delegates to .org name servers * .org name servers delegate to rubygems.org name servers * rubygems.org name servers can either respond to the query or delegate to another set of name servers. It'll answer in this case. * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org and all NAPTR records for 1.0.3.rails.index.rubygems.org, for example: rails.index.rubygems.org. 600 CNAME 1.0.3.rails.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://rubygems.org/rails-3.0.1.gem!i" . 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 20 "U" "TCP+http" "!^.*$!http://backup.rubygems.org/rails-3.0.1.gem!i" . -
aeden revised this gist
Jan 18, 2011 . 1 changed file with 2 additions and 27 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -9,30 +9,7 @@ I'd like to propose an alternate way of indexing RubyGems: let's use DNS. Here's how this might work: Scenario 1: Client wants to get the dependencies for the gem ------------------------------------------------------------ * Client sends question to local name server for PTR records at rails.index.rubygems.org @@ -54,8 +31,7 @@ Scenario 2: Client wants to get the dependencies for the gem Note that the first 6 records represent canonical gem names, whereas the final record would likely be a CNAME pointing to the latest version in the 1 major release branch. That CNAME record would likely resolve to something like 7.0.1.bundler.index.rubygems.org (which would be the reverse notation for bundler-1.0.7) Scenario 2: Client wants to download a gem ------------------------------------------ * Client sends question to local name server for NAPTR records at rails.index.rubygems.org @@ -71,7 +47,6 @@ Scenario 3: Client wants to download a gem Note that there is no need to do any complex regex translation to get the various URLs since they are mapped directly to the canonical name of the gem. Other Considerations -------------------- -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -102,7 +102,7 @@ DNS does not provide a mechanism for search for records given a part of a name. Reference --------- * [Wikipedia list of DNS record types](http://en.wikipedia.org/wiki/List_of_DNS_record_types) * [Nick Q Blog Post](http://robots.thoughtbot.com/post/2729333530/fetching-source-index-for-http-rubygems-org) * [RFC 1035](http://tools.ietf.org/html/rfc1035) * [RFC 3401](http://tools.ietf.org/html/rfc3401) -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 5 additions and 5 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -102,8 +102,8 @@ DNS does not provide a mechanism for search for records given a part of a name. Reference --------- * (http://en.wikipedia.org/wiki/List_of_DNS_record_types) * [Nick Q Blog Post](http://robots.thoughtbot.com/post/2729333530/fetching-source-index-for-http-rubygems-org) * [RFC 1035](http://tools.ietf.org/html/rfc1035) * [RFC 3401](http://tools.ietf.org/html/rfc3401) * [NAPTR Record](http://en.wikipedia.org/wiki/NAPTR_record) -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 5 additions and 5 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -102,8 +102,8 @@ DNS does not provide a mechanism for search for records given a part of a name. Reference --------- * http://en.wikipedia.org/wiki/List_of_DNS_record_types * http://robots.thoughtbot.com/post/2729333530/fetching-source-index-for-http-rubygems-org * http://tools.ietf.org/html/rfc1035 * http://tools.ietf.org/html/rfc3401 * http://en.wikipedia.org/wiki/NAPTR_record -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 9 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -98,3 +98,12 @@ DNSSEC providers a means for signing DNS records so that you have verification t ### Searching DNS does not provide a mechanism for search for records given a part of a name. For example, there is no mechanism in DNS to query for the term "active" and get "activerecord", "activeresource", etc. This functionality would need to be provided using a protocol other than DNS. Reference --------- http://en.wikipedia.org/wiki/List_of_DNS_record_types http://robots.thoughtbot.com/post/2729333530/fetching-source-index-for-http-rubygems-org http://tools.ietf.org/html/rfc1035 http://tools.ietf.org/html/rfc3401 http://en.wikipedia.org/wiki/NAPTR_record -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -85,8 +85,8 @@ DNS provides the tools necessary to make this a decentralized system if we desir * .org name servers delegate to rubygems.org name servers * rubygems.org name servers respond with the following NS record: rails.index.rubygems.org. 600 NS idx1.rubyonrails.org rails.index.rubygems.org. 600 NS idx2.rubyonrails.org * The question is then sent to one of the two name servers which responds with a CNAME record pointing rails.index.rubygems.org to 1.0.3.rails.index.rubyonrails.org. * The rubyonrails.org name servers would then respond as shown in the scenarios above. -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -85,8 +85,8 @@ DNS provides the tools necessary to make this a decentralized system if we desir * .org name servers delegate to rubygems.org name servers * rubygems.org name servers respond with the following NS record: rails.index.rubygems.org. 600 NS idx1.rubyonrails.org rails.index.rubygems.org. 600 NS idx2.rubyonrails.org * The question is then sent to one of the two name servers which responds with a CNAME record pointing rails.index.rubygems.org to 1.0.3.rails.index.rubyonrails.org. * The rubyonrails.org name servers would then respond as shown in the scenarios above. -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 7 additions and 7 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -44,13 +44,13 @@ Scenario 2: Client wants to get the dependencies for the gem * rubygems.org name servers respond with all PTR records for 1.0.3.rails.index.rubygems.org, for example: 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activesupport.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.actiopack.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activerecord.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activeresource.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.actionmailer.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.railties.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 1.bundler.index.rubygems.org. Note that the first 6 records represent canonical gem names, whereas the final record would likely be a CNAME pointing to the latest version in the 1 major release branch. That CNAME record would likely resolve to something like 7.0.1.bundler.index.rubygems.org (which would be the reverse notation for bundler-1.0.7) -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 4 additions and 4 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -26,8 +26,8 @@ Scenario 1: Client wants metadata about a gem Other possible records for storing metadata: 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://www.rubyonrails.org!i" . 1.0.3.rails.index.rubygems.org. 600 RP david.loudthinking.com. david.loudthinking.com. The downside with using different record types to represent these metadata elements is that the client would likely need to send questions for each type, thus increasing the number of questions that need to be sent. Additionally, the RP record would require additional processing in the form of a question sent to david.loudthinking.com to determine what the correct email address is. Therefore I recommend we stick with convention of using TXT records for all forms of informational metadata. @@ -66,8 +66,8 @@ Scenario 3: Client wants to download a gem * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org * rubygems.org name servers respond with all NAPTR records for 1.0.3.rails.index.rubygems.org, for example: 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://rubygems.org/rails-3.0.1.gem!i" . 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 20 "U" "TCP+http" "!^.*$!http://backup.rubygems.org/rails-3.0.1.gem!i" . Note that there is no need to do any complex regex translation to get the various URLs since they are mapped directly to the canonical name of the gem. -
aeden revised this gist
Jan 16, 2011 . 1 changed file with 5 additions and 5 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -20,14 +20,14 @@ Scenario 1: Client wants metadata about a gem * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org (note: this is Rails 3.0.1 - reverse notation) * rubygems.org name servers respond with all TXT records for 1.0.3.rails.index.rubygems.org, for example: 1.0.3.rails.index.rubygems.org. 600 TXT "description:Ruby on Rails is a full-stack web framework optimized for programmer happiness and sustainable productivity. It encourages beautiful code by favoring convention over configuration." 1.0.3.rails.index.rubygems.org. 600 TXT "email:david@loudthinking.com" 1.0.3.rails.index.rubygems.org. 600 TXT "url:http://www.rubyonrails.org/" Other possible records for storing metadata: 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://www.rubyonrails.org!i" . 1.0.3.rails.index.rubygems.org. 600 RP david.loudthinking.com. david.loudthinking.com. The downside with using different record types to represent these metadata elements is that the client would likely need to send questions for each type, thus increasing the number of questions that need to be sent. Additionally, the RP record would require additional processing in the form of a question sent to david.loudthinking.com to determine what the correct email address is. Therefore I recommend we stick with convention of using TXT records for all forms of informational metadata. -
aeden created this gist
Jan 16, 2011 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,100 @@ Currently the RubyGems index is stored as a Gzip file that is a marshalled array. Whenever Bundler needs to install a gem that is not yet installed it downloads the index, gunzips it and unmarshals it. It then looks for dependencies that are described in another file that is also a gzipped and marshalled file. There are several issues that arise from this setup: * The full index must be downloaded and parsed, but does not contain dependency data, which must then be downloaded and parsed. This is a relatively time consuming process. * The index must be centralized. Additionally the gems themselves are currently centralized since there is nothing in the meta data that indicates where the gem should be downloaded from. However in order to allow this it is important to find ways of keeping the index from being poisoned (is this an issue in the centralized system?) I'd like to propose an alternate way of indexing RubyGems: let's use DNS. Here's how this might work: Scenario 1: Client wants metadata about a gem --------------------------------------------- * Client sends question to local name server for TXT records at rails.index.rubygems.org * Local name server does not have the record so it sends the usual response indicating that the search should go upstream to the roots * Root delegates to .org name servers * .org name servers delegate to rubygems.org name servers * rubygems.org name servers can either respond to the query or delegate to another set of name servers. It'll answer in this case. * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org (note: this is Rails 3.0.1 - reverse notation) * rubygems.org name servers respond with all TXT records for 1.0.3.rails.index.rubygems.org, for example: 1.0.3.rails.index.rubygems.org. 600 TXT "description:Ruby on Rails is a full-stack web framework optimized for programmer happiness and sustainable productivity. It encourages beautiful code by favoring convention over configuration." 1.0.3.rails.index.rubygems.org. 600 TXT "email:david@loudthinking.com" 1.0.3.rails.index.rubygems.org. 600 TXT "url:http://www.rubyonrails.org/" Other possible records for storing metadata: 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://www.rubyonrails.org!i" . 1.0.3.rails.index.rubygems.org. 600 RP david.loudthinking.com. david.loudthinking.com. The downside with using different record types to represent these metadata elements is that the client would likely need to send questions for each type, thus increasing the number of questions that need to be sent. Additionally, the RP record would require additional processing in the form of a question sent to david.loudthinking.com to determine what the correct email address is. Therefore I recommend we stick with convention of using TXT records for all forms of informational metadata. Scenario 2: Client wants to get the dependencies for the gem ------------------------------------------------------------ * Client sends question to local name server for PTR records at rails.index.rubygems.org * Local name server does not have the record so it sends the usual response indicating that the search should go upstream to the roots * Root delegates to .org name servers * .org name servers delegate to rubygems.org name servers * rubygems.org name servers can either respond to the query or delegate to another set of name servers. It'll answer in this case. * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org * rubygems.org name servers respond with all PTR records for 1.0.3.rails.index.rubygems.org, for example: 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activesupport.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.actiopack.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activerecord.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.activeresource.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.actionmailer.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 0.0.3.railties.index.rubygems.org. 1.0.3.rails.index.rubygems.org. 600 PTR 1.bundler.index.rubygems.org. Note that the first 6 records represent canonical gem names, whereas the final record would likely be a CNAME pointing to the latest version in the 1 major release branch. That CNAME record would likely resolve to something like 7.0.1.bundler.index.rubygems.org (which would be the reverse notation for bundler-1.0.7) Scenario 3: Client wants to download a gem ------------------------------------------ * Client sends question to local name server for NAPTR records at rails.index.rubygems.org * Local name server does not have the record so it sends the usual response indicating that the search should go upstream to the roots * Root delegates to .org name servers * .org name servers delegate to rubygems.org name servers * rubygems.org name servers can either respond to the query or delegate to another set of name servers. It'll answer in this case. * rubygems.org name servers respond with a CNAME record pointing to 1.0.3.rails.index.rubygems.org * rubygems.org name servers respond with all NAPTR records for 1.0.3.rails.index.rubygems.org, for example: 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 10 "U" "TCP+http" "!^.*$!http://rubygems.org/rails-3.0.1.gem!i" . 1.0.3.rails.index.rubygems.org. 600 NAPTR 100 20 "U" "TCP+http" "!^.*$!http://backup.rubygems.org/rails-3.0.1.gem!i" . Note that there is no need to do any complex regex translation to get the various URLs since they are mapped directly to the canonical name of the gem. Other Considerations -------------------- ### Decentralization DNS provides the tools necessary to make this a decentralized system if we desire. This would be accomplished by delegating responsibility for gem names out to different DNS servers other than the rubygems.org servers. For example, if responsibility for management of the Rails gem metadata was decrentralized then the interaction might look like this: * Client sends question to local name server for TXT records at rails.index.rubygems.org * Local name server does not have the record so it sends the usual response indicating that the search should go upstream to the roots * Root delegates to .org name servers * .org name servers delegate to rubygems.org name servers * rubygems.org name servers respond with the following NS record: rails.index.rubygems.org. 600 NS idx1.rubyonrails.org rails.index.rubygems.org. 600 NS idx2.rubyonrails.org * The question is then sent to one of the two name servers which responds with a CNAME record pointing rails.index.rubygems.org to 1.0.3.rails.index.rubyonrails.org. * The rubyonrails.org name servers would then respond as shown in the scenarios above. ### Security DNSSEC providers a means for signing DNS records so that you have verification that the name server is authoritative for the particular question. This technology is not yet widely deployed, however it does have the potential for providing layer of protection against gem poisoning when used in conjunction with and SHA signature. The SHA signature could also be stored in the name servers using a TXT or SIG record. This technology is still very experimental, but the potential exists for having a highly trusted distribution system. ### Searching DNS does not provide a mechanism for search for records given a part of a name. For example, there is no mechanism in DNS to query for the term "active" and get "activerecord", "activeresource", etc. This functionality would need to be provided using a protocol other than DNS.