-
-
Save rgould/aa1da965bd8f616443724b1bf9e3eb5c to your computer and use it in GitHub Desktop.
Revisions
-
rgould revised this gist
Feb 4, 2019 . 1 changed file with 27 additions and 15 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,83 +1,95 @@ # How to compress URL parameters (and any other string) in Ruby ## Introduction Working with web applications and microservices we often have the problem of passing long lists of settings through HTTP requests. While the canonical way of doing this would be to use a POST request and pass the values in the body of the request, this is not always an option for us. (Comment: How come? Maybe the reason we can't do it is interesting/helpful) In fact there can be cases where we need to generate URLs that can be passed to other users as a link, and reused by them to later trigger an action in the target Rails server. The only way to achieve this is for us to be able to pass these values as URL parameters in a GET request, but unfortunately these URLs exceed often the maximum length limit allowed by the browsers or web servers. (Comment: Maybe include the byte limit here) To workaround this problem we decided to compress these values using the zlib library, which turned out to be very effective for this kind of compression, especially in cases of list of settings which usually contain repetitive sequences of characters. ## Example Our client application generates a request URL to another app, appending a list of values as an array parameter. Normally the request URL would look like this: ``` http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceeds the maximum allowed length ``` This would usually trigger an error if the URL exceeds 2083 characters (comment: Characters? or bytes?) or any other browser/server specific URL limit. In order to send the values inside the my_values array we need to create a compressed string. ### Client logic and URL generation First we merge the array of values in a single comma separated string: ```ruby comma_separated_values = my_values.join(",") => "value_1,value_2,value_3" ``` We compress the values using Zlib: ```ruby compressed_values = Zlib::Deflate.deflate(comma_separated_values) => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc" ``` Then we encode the compressed values using base64: ```ruby encoded_values = Base64.strict_encode64(compressed_values) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw==" ``` Finally we encode the string to be safely passed as a string parameter: ```ruby url_safe_values = CGI.escape(encoded_values) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D" ``` Now we can append the compressed string to our URL as a parameter: ```ruby request_url = "http://serverapp.com?myvalues=#{url_safe_values}" => "http://serverapp.com?myvalues=eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D" ``` This URL can be used to issue requests to our server, both programmatically and by a user clicking on the link. In order to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client: ### Server logic and URL parameter decompression Unescape the values: ```ruby unescaped_values = CGI.unescape(params[:my_values]) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw==" ``` Decode them: ```ruby encoded_values = Base64.strict_decode64(unescaped_values) => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc" ``` Decompress them: ```ruby comma_separated_values = Zlib::Inflate.inflate(encoded_values) => "value_1,value_2,value_3" ``` And finally expand the comma separated values in to the original array: ```ruby array_of_values = comma_separated_values.split(",") => ["value_1", "value_2", "value_3"] ``` That's how we implemented an effective technique to pass a long list of values through a GET request without exceeding the maximum allowed length. -
Alessandro Caporrini revised this gist
Feb 4, 2019 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -74,7 +74,7 @@ Decompress them: comma_separated_values = Zlib::Inflate.inflate(encoded_values) => "value_1,value_2,value_3" ``` And finally expand the comma separated values in to the original array: ```ruby array_of_values = comma_separated_values.split(",") => ["value_1", "value_2", "value_3"] -
Alessandro Caporrini revised this gist
Feb 4, 2019 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,4 +1,4 @@ # How to compress url parameters (and any other string) in Ruby ## Introduction -
Alessandro Caporrini revised this gist
Feb 4, 2019 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -19,7 +19,7 @@ Our client application generate a request url to another app appending a list of http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceed the maximum allowed number ``` This would usually trigger an error in case the url excede 2083 characters or any other browser/server specific url limit. In order to send the values inside the my_values array we need to create a compressed string. -
Alessandro Caporrini revised this gist
Feb 3, 2019 . 1 changed file with 25 additions and 15 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,55 +1,63 @@ # How to compress url parameters (and any other string) in Rails ## Introduction Working with web applications and microservices we often have the problem of passing long lists of settings through http requests. While the canonical way of doing this would be to use a POST request and pass the values in the body of the request, this is not always an option for us. In fact there can be cases where we need to generate urls that can be passed to other users as link and reused by them to trigger an action in the target Rails server. The only way to achieve this is for us to be able to pass these values as url parameters in a GET request but unfortunately these urls exceed often the maximum url length limit allowed by the browsers or web servers. To workaround this problem we decided to compress these values using the zlib library, which turned out to be very effective for these kind of compression, especially in cases of list of settings which usually contain many repeated sequence of characters. ## Example Our client application generate a request url to another app appending a list of values as an array parameter, normally the request url would look like: ``` http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceed the maximum allowed number ``` This would usually trigger an error in case the url excede 2083 or any other browser/server specific url limit. In order to send the values inside the my_values array we need to create a compressed string. ### Client logic and url generation First we merge the array of values in a single comma separated string: ```ruby comma_separated_values = my_values.join(",") => "value_1,value_2,value_3" ``` We compress the values using Zlib: ```ruby compressed_values = Zlib::Deflate.deflate(comma_separated_values) => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc" ``` Then we encode in base64 the compressed values: ```ruby encoded_values = Base64.strict_encode64(compressed_values) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw==" ``` Finally we encode the string to be safely passed as a string parameter: ```ruby url_safe_values = CGI.escape(encoded_values) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D" ``` Now we can append the compressed string to our url as a parameter: ```ruby request_url = "http://serverapp.com?myvalues=#{url_safe_values}" => "http://serverapp.com?myvalues=eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D" ``` This url can be used to issue requests to our server, both programmatically and by a user clicking on the link, but in order to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client: ### Server logic and url parameter decompression Unescape the values: ```ruby @@ -71,3 +79,5 @@ And finally expand the comma separated values in the original array: array_of_values = comma_separated_values.split(",") => ["value_1", "value_2", "value_3"] ``` In this way we implemented an effective way to pass a long list of values through a GET request without exceeding the maximum allowed length. -
Alessandro Caporrini renamed this gist
Feb 3, 2019 . 1 changed file with 1 addition and 3 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,4 @@ # How to compress url parameters (and any other string) in Rails # Introduction -
Alessandro Caporrini revised this gist
Feb 3, 2019 . 1 changed file with 71 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,75 @@ # How to compress urls in ruby https://stackoverflow.com/questions/17882463/compressing-large-string-in-ruby # Introduction Working with web applications and microservices we often have the problem of passing long lists of values through http requests. While the canonical way of doing this would be to use a POST request or another HTML method and pass the values in the body of the request, this is not always a solution for us. In fact there are cases where we want to generate action urls that can be passed to other users as link and reused by them to trigger the action in the target Rails application. The only way to achieve this is for us to be able to pass these values as url parameters in a GET request. Unfortunately the url maximum lenght might be limited both by the browsers and web servers so we needed to find a workaround for this by compressing the values string with the zlib library. # Example Our client application generate a request to another app appending a list of values as an array parameter, normally the request url would look like: ``` http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exced the maximum allowed number ``` In order to be able to use this urls we take the values inside the my_values array and we create a string of comma separated values: ```ruby comma_separated_values = my_values.join(",") => "value_1,value_2,value_3" ``` Then we compress the values using Zlib: ```ruby compressed_values = Zlib::Deflate.deflate(comma_separated_values) => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc" ``` We then encode base64 the compressed values: ```ruby encoded_values = Base64.strict_encode64(compressed_values) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw==" ``` Finally we encode the string to be safelly passed as a string parameter: ```ruby url_safe_values = CGI.escape(encoded_values) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D" ``` Now we can append the compressed string to our url as a parameter: ```ruby request_url = "http://serverapp.com?myvalues=#{url_safe_values}" => "http://serverapp.com?myvalues=eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D" ``` Now we can use this url to issue requests to our server, in orded to be able to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client: Unescape the values: ```ruby unescaped_values = CGI.unescape(params[:my_values]) => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw==" ``` Decode them: ```ruby encoded_values = Base64.strict_decode64(unescaped_values) => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc" ``` Decompress them: ```ruby comma_separated_values = Zlib::Inflate.inflate(encoded_values) => "value_1,value_2,value_3" ``` And finally expand the comma separated values in the original array: ```ruby array_of_values = comma_separated_values.split(",") => ["value_1", "value_2", "value_3"] ``` -
Alessandro Caporrini created this gist
Feb 3, 2019 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,6 @@ # How to compress urls in ruby # Introduction Working with web applications and micro services we often have the problem of passing long lists of values through http requests. While the canonical way of doing this would be to use a POST request or another HTML method and pass the values in the body of the request, this is not always a solution.