Skip to content

Instantly share code, notes, and snippets.

@rgould
Forked from acaporrini/compress_urls.md
Last active February 4, 2019 13:15
Show Gist options
  • Save rgould/aa1da965bd8f616443724b1bf9e3eb5c to your computer and use it in GitHub Desktop.
Save rgould/aa1da965bd8f616443724b1bf9e3eb5c to your computer and use it in GitHub Desktop.

Revisions

  1. rgould revised this gist Feb 4, 2019. 1 changed file with 27 additions and 15 deletions.
    42 changes: 27 additions & 15 deletions compress_urls.md
    Original file line number Diff line number Diff line change
    @@ -1,83 +1,95 @@
    # How to compress url parameters (and any other string) in Ruby
    # How to compress URL parameters (and any other string) in Ruby

    ## Introduction

    Working with web applications and microservices we often have the problem of passing long lists of settings through http requests.
    While the canonical way of doing this would be to use a POST request and pass the values in the body of the request, this is not always an option for us.
    Working with web applications and microservices we often have the problem of passing long lists of settings through HTTP requests.
    While the canonical way of doing this would be to use a POST request and pass the values in the body of the request, this is not always an option for us. (Comment: How come? Maybe the reason we can't do it is interesting/helpful)

    In fact there can be cases where we need to generate urls that can be passed to other users as link and reused by them to trigger an action in the target Rails server.
    In fact there can be cases where we need to generate URLs that can be passed to other users as a link, and reused by them to later trigger an action in the target Rails server.

    The only way to achieve this is for us to be able to pass these values as url parameters in a GET request but unfortunately these urls exceed often the maximum url length limit allowed by the browsers or web servers.
    The only way to achieve this is for us to be able to pass these values as URL parameters in a GET request, but unfortunately these URLs exceed often the maximum length limit allowed by the browsers or web servers. (Comment: Maybe include the byte limit here)

    To workaround this problem we decided to compress these values using the zlib library, which turned out to be very effective for these kind of compression, especially in cases of list of settings which usually contain many repeated sequence of characters.
    To workaround this problem we decided to compress these values using the zlib library, which turned out to be very effective for this kind of compression, especially in cases of list of settings which usually contain repetitive sequences of characters.


    ## Example
    Our client application generate a request url to another app appending a list of values as an array parameter, normally the request url would look like:
    Our client application generates a request URL to another app, appending a list of values as an array parameter. Normally the request URL would look like this:

    ```
    http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceed the maximum allowed number
    http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceeds the maximum allowed length
    ```

    This would usually trigger an error in case the url excede 2083 characters or any other browser/server specific url limit.
    This would usually trigger an error if the URL exceeds 2083 characters (comment: Characters? or bytes?) or any other browser/server specific URL limit.

    In order to send the values inside the my_values array we need to create a compressed string.


    ### Client logic and url generation
    ### Client logic and URL generation

    First we merge the array of values in a single comma separated string:

    ```ruby
    comma_separated_values = my_values.join(",")
    => "value_1,value_2,value_3"
    ```

    We compress the values using Zlib:

    ```ruby
    compressed_values = Zlib::Deflate.deflate(comma_separated_values)
    => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc"
    ```
    Then we encode in base64 the compressed values:

    Then we encode the compressed values using base64:

    ```ruby
    encoded_values = Base64.strict_encode64(compressed_values)
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw=="
    ```

    Finally we encode the string to be safely passed as a string parameter:

    ```ruby
    url_safe_values = CGI.escape(encoded_values)
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
    ```
    Now we can append the compressed string to our url as a parameter:

    Now we can append the compressed string to our URL as a parameter:

    ```ruby
    request_url = "http://serverapp.com?myvalues=#{url_safe_values}"
    => "http://serverapp.com?myvalues=eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
    ```
    This url can be used to issue requests to our server, both programmatically and by a user clicking on the link, but in order to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client:
    This URL can be used to issue requests to our server, both programmatically and by a user clicking on the link. In order to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client:

    ### Server logic and url parameter decompression
    ### Server logic and URL parameter decompression

    Unescape the values:

    ```ruby
    unescaped_values = CGI.unescape(params[:my_values])
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw=="
    ```

    Decode them:

    ```ruby
    encoded_values = Base64.strict_decode64(unescaped_values)
    => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc"
    ```

    Decompress them:

    ```ruby
    comma_separated_values = Zlib::Inflate.inflate(encoded_values)
    => "value_1,value_2,value_3"
    ```

    And finally expand the comma separated values in to the original array:

    ```ruby
    array_of_values = comma_separated_values.split(",")
    => ["value_1", "value_2", "value_3"]
    ```

    In this way we implemented an effective way to pass a long list of values through a GET request without exceeding the maximum allowed length.
    That's how we implemented an effective technique to pass a long list of values through a GET request without exceeding the maximum allowed length.
  2. Alessandro Caporrini revised this gist Feb 4, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion compress_urls.md
    Original file line number Diff line number Diff line change
    @@ -74,7 +74,7 @@ Decompress them:
    comma_separated_values = Zlib::Inflate.inflate(encoded_values)
    => "value_1,value_2,value_3"
    ```
    And finally expand the comma separated values in the original array:
    And finally expand the comma separated values in to the original array:
    ```ruby
    array_of_values = comma_separated_values.split(",")
    => ["value_1", "value_2", "value_3"]
  3. Alessandro Caporrini revised this gist Feb 4, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion compress_urls.md
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    # How to compress url parameters (and any other string) in Rails
    # How to compress url parameters (and any other string) in Ruby

    ## Introduction

  4. Alessandro Caporrini revised this gist Feb 4, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion compress_urls.md
    Original file line number Diff line number Diff line change
    @@ -19,7 +19,7 @@ Our client application generate a request url to another app appending a list of
    http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceed the maximum allowed number
    ```

    This would usually trigger an error in case the url excede 2083 or any other browser/server specific url limit.
    This would usually trigger an error in case the url excede 2083 characters or any other browser/server specific url limit.

    In order to send the values inside the my_values array we need to create a compressed string.

  5. Alessandro Caporrini revised this gist Feb 3, 2019. 1 changed file with 25 additions and 15 deletions.
    40 changes: 25 additions & 15 deletions compress_urls.md
    Original file line number Diff line number Diff line change
    @@ -1,55 +1,63 @@
    # How to compress url parameters (and any other string) in Rails

    # Introduction
    ## Introduction

    Working with web applications and microservices we often have the problem of passing long lists of values through http requests.
    While the canonical way of doing this would be to use a POST request or another HTML method and pass the values in the body of the request, this is not always a solution for us.
    Working with web applications and microservices we often have the problem of passing long lists of settings through http requests.
    While the canonical way of doing this would be to use a POST request and pass the values in the body of the request, this is not always an option for us.

    In fact there are cases where we want to generate action urls that can be passed to other users as link and reused by them to trigger the action in the target Rails application.
    In fact there can be cases where we need to generate urls that can be passed to other users as link and reused by them to trigger an action in the target Rails server.

    The only way to achieve this is for us to be able to pass these values as url parameters in a GET request.
    The only way to achieve this is for us to be able to pass these values as url parameters in a GET request but unfortunately these urls exceed often the maximum url length limit allowed by the browsers or web servers.

    Unfortunately the url maximum lenght might be limited both by the browsers and web servers so we needed to find a workaround for this by compressing the values string with the zlib library.
    To workaround this problem we decided to compress these values using the zlib library, which turned out to be very effective for these kind of compression, especially in cases of list of settings which usually contain many repeated sequence of characters.

    # Example
    Our client application generate a request to another app appending a list of values as an array parameter, normally the request url would look like:

    ## Example
    Our client application generate a request url to another app appending a list of values as an array parameter, normally the request url would look like:

    ```
    http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exced the maximum allowed number
    http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceed the maximum allowed number
    ```

    In order to be able to use this urls we take the values inside the my_values array and we create a string of comma separated values:
    This would usually trigger an error in case the url excede 2083 or any other browser/server specific url limit.

    In order to send the values inside the my_values array we need to create a compressed string.


    ### Client logic and url generation

    First we merge the array of values in a single comma separated string:

    ```ruby
    comma_separated_values = my_values.join(",")
    => "value_1,value_2,value_3"
    ```
    Then we compress the values using Zlib:
    We compress the values using Zlib:

    ```ruby
    compressed_values = Zlib::Deflate.deflate(comma_separated_values)
    => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc"
    ```
    We then encode base64 the compressed values:
    Then we encode in base64 the compressed values:
    ```ruby
    encoded_values = Base64.strict_encode64(compressed_values)
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw=="
    ```
    Finally we encode the string to be safelly passed as a string parameter:
    Finally we encode the string to be safely passed as a string parameter:

    ```ruby
    url_safe_values = CGI.escape(encoded_values)
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
    ```

    Now we can append the compressed string to our url as a parameter:

    ```ruby
    request_url = "http://serverapp.com?myvalues=#{url_safe_values}"
    => "http://serverapp.com?myvalues=eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
    ```
    This url can be used to issue requests to our server, both programmatically and by a user clicking on the link, but in order to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client:

    Now we can use this url to issue requests to our server, in orded to be able to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client:
    ### Server logic and url parameter decompression

    Unescape the values:
    ```ruby
    @@ -71,3 +79,5 @@ And finally expand the comma separated values in the original array:
    array_of_values = comma_separated_values.split(",")
    => ["value_1", "value_2", "value_3"]
    ```

    In this way we implemented an effective way to pass a long list of values through a GET request without exceeding the maximum allowed length.
  6. Alessandro Caporrini renamed this gist Feb 3, 2019. 1 changed file with 1 addition and 3 deletions.
    4 changes: 1 addition & 3 deletions blog.md → compress_urls.md
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,4 @@
    # How to compress urls in ruby

    https://stackoverflow.com/questions/17882463/compressing-large-string-in-ruby
    # How to compress url parameters (and any other string) in Rails

    # Introduction

  7. Alessandro Caporrini revised this gist Feb 3, 2019. 1 changed file with 71 additions and 2 deletions.
    73 changes: 71 additions & 2 deletions blog.md
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,75 @@
    # How to compress urls in ruby

    https://stackoverflow.com/questions/17882463/compressing-large-string-in-ruby

    # Introduction

    Working with web applications and micro services we often have the problem of passing long lists of values through http requests.
    While the canonical way of doing this would be to use a POST request or another HTML method and pass the values in the body of the request, this is not always a solution.
    Working with web applications and microservices we often have the problem of passing long lists of values through http requests.
    While the canonical way of doing this would be to use a POST request or another HTML method and pass the values in the body of the request, this is not always a solution for us.

    In fact there are cases where we want to generate action urls that can be passed to other users as link and reused by them to trigger the action in the target Rails application.

    The only way to achieve this is for us to be able to pass these values as url parameters in a GET request.

    Unfortunately the url maximum lenght might be limited both by the browsers and web servers so we needed to find a workaround for this by compressing the values string with the zlib library.

    # Example
    Our client application generate a request to another app appending a list of values as an array parameter, normally the request url would look like:

    ```
    http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exced the maximum allowed number
    ```

    In order to be able to use this urls we take the values inside the my_values array and we create a string of comma separated values:

    ```ruby
    comma_separated_values = my_values.join(",")
    => "value_1,value_2,value_3"
    ```
    Then we compress the values using Zlib:

    ```ruby
    compressed_values = Zlib::Deflate.deflate(comma_separated_values)
    => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc"
    ```
    We then encode base64 the compressed values:
    ```ruby
    encoded_values = Base64.strict_encode64(compressed_values)
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw=="
    ```
    Finally we encode the string to be safelly passed as a string parameter:

    ```ruby
    url_safe_values = CGI.escape(encoded_values)
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
    ```

    Now we can append the compressed string to our url as a parameter:

    ```ruby
    request_url = "http://serverapp.com?myvalues=#{url_safe_values}"
    => "http://serverapp.com?myvalues=eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
    ```

    Now we can use this url to issue requests to our server, in orded to be able to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client:

    Unescape the values:
    ```ruby
    unescaped_values = CGI.unescape(params[:my_values])
    => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw=="
    ```
    Decode them:
    ```ruby
    encoded_values = Base64.strict_decode64(unescaped_values)
    => "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc"
    ```
    Decompress them:
    ```ruby
    comma_separated_values = Zlib::Inflate.inflate(encoded_values)
    => "value_1,value_2,value_3"
    ```
    And finally expand the comma separated values in the original array:
    ```ruby
    array_of_values = comma_separated_values.split(",")
    => ["value_1", "value_2", "value_3"]
    ```
  8. Alessandro Caporrini created this gist Feb 3, 2019.
    6 changes: 6 additions & 0 deletions blog.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,6 @@
    # How to compress urls in ruby

    # Introduction

    Working with web applications and micro services we often have the problem of passing long lists of values through http requests.
    While the canonical way of doing this would be to use a POST request or another HTML method and pass the values in the body of the request, this is not always a solution.