Skip to content

Instantly share code, notes, and snippets.

@wayne5540
Last active December 3, 2018 03:21
Show Gist options
  • Select an option

  • Save wayne5540/7a5865e74e16c89be5ed85783b908e32 to your computer and use it in GitHub Desktop.

Select an option

Save wayne5540/7a5865e74e16c89be5ed85783b908e32 to your computer and use it in GitHub Desktop.

Revisions

  1. wayne5540 revised this gist Nov 21, 2016. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions rails-json-schema-validation.md
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,5 @@
    # Validate JSON schema in Rails

    ## Topics

    1. What/Why JSON schema
  2. wayne5540 revised this gist Nov 21, 2016. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions rails-json-schema-validation.md
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@
    1. What/Why JSON schema
    2. Apply to rails model validation
    3. Test your API endpoint with schema matcher
    4. Further reading
    4. Homework for a curious reader
    5. References

    ### What/Why JSON schema
    @@ -322,7 +322,9 @@ Add schema file for user, notice we put user object definitions into `definition
    }
    ```

    Now test passed.
    In here we've used a technic called `reference`, as you can see we put `user` object's schema into `definitions` inside `app/controllers/schemas/user.json` and we use `"$ref": "#/definitions/user"` to reference it, it's a best practice to move your schema under definition key so that you can reuse it in the future (even cross file), as you see we also use `"$ref": "user.json#/definitions/user"` inside our `app/controllers/chemas/users.json`, just think it as rails view partial and you'll get it. :sunglasses:

    Let's run our test, now test passed.

    As so once again, the day is saved thanks to JSON schema. :joy:

  3. wayne5540 revised this gist Nov 21, 2016. 1 changed file with 49 additions and 6 deletions.
    55 changes: 49 additions & 6 deletions rails-json-schema-validation.md
    Original file line number Diff line number Diff line change
    @@ -8,18 +8,53 @@

    ### What/Why JSON schema

    What?
    What is JSON schema?

    * https://brandur.org/elegant-apis
    * https://spacetelescope.github.io/understanding-json-schema/index.html
    I recommend you to read [this](https://brandur.org/elegant-apis) and [that](https://spacetelescope.github.io/understanding-json-schema/index.html) to understand more about JSON schema.

    Why?
    However if you are lazy like me, just think JSON schema as the spec of your JSON data, it help you defines how your JSON data should looks like. The simplest schema is like below:

    ```json
    {
    "properties": {
    "name": {
    "type": "string"
    }
    },
    "required": ["name"],
    "type": "object"
    }
    ```

    This schema means your JSON object should have a name attribute with string type.

    for example, this will be a valid JSON:

    ```json
    {
    "name": "Wayne"
    }
    ```

    However this is not a valid JSON because name is a number but not a string

    ```json
    {
    "name": 5566
    }
    ```

    So, why we need JSON schema? What's the benefit?

    First of all, define your data properly is never a bad idea.
    And there are at least 4 benefits I can think of:

    * validate your JSON data structure, so you don't mess up your database
    * help you validate your APIs response, especially REST-like API
    * One rule, everywhere. Help your client validate their data.
    * Bonus: can integrate with Swagger (if you like)

    Cool, so now let's write some code with our beloved ruby.

    ### Apply to rails model validation

    @@ -118,7 +153,7 @@ end

    Add JSON schema file

    I prefer add json file into `app/models/schemas/report/data.json`
    I prefer add `.json` file into `app/models/schemas/report/data.json`

    ```json
    // app/models/schemas/report/data.json
    @@ -289,13 +324,21 @@ Add schema file for user, notice we put user object definitions into `definition

    Now test passed.

    ### Further reading
    As so once again, the day is saved thanks to JSON schema. :joy:

    Hope you had a good reading.

    ### Homework for a curious reader

    If this post can't fulfill your curiosity, there are more topics of JSON schema you can chasing for:

    * How to write generic JSON schema test and apply to every api endpoints?
    * How to expose your JSON schema so you can share/use it either at rails project or other repos

    ### References

    Also, you can check those reference for more details. Cheers!

    * https://brandur.org/elegant-apis
    * https://robots.thoughtbot.com/validating-json-schemas-with-an-rspec-matcher
    * https://github.com/mirego/activerecord_json_validator
  4. wayne5540 created this gist Nov 18, 2016.
    305 changes: 305 additions & 0 deletions rails-json-schema-validation.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,305 @@
    ## Topics

    1. What/Why JSON schema
    2. Apply to rails model validation
    3. Test your API endpoint with schema matcher
    4. Further reading
    5. References

    ### What/Why JSON schema

    What?

    * https://brandur.org/elegant-apis
    * https://spacetelescope.github.io/understanding-json-schema/index.html

    Why?

    * validate your JSON data structure, so you don't mess up your database
    * help you validate your APIs response, especially REST-like API
    * One rule, everywhere. Help your client validate their data.
    * Bonus: can integrate with Swagger (if you like)


    ### Apply to rails model validation

    There are 2 gems I found to help me validate JSON Schema, the first one is the ruby implementation of JSON Schema validate called [json-schema](https://github.com/ruby-json-schema/json-schema), the second one is rails validator implementation called [activerecord_json_validator](https://github.com/mirego/activerecord_json_validator) based on first one.

    Let's use [activerecord_json_validator](https://github.com/mirego/activerecord_json_validator) to integrate our model level JSON validation.

    Assume we have `User` and `Report`, we allow user to send error report to us including their system's environment and save at `data` column as JSON, migration file looks like below:

    ```rb
    # db/migrations/xxxxxxxxx_create_reports.rb
    class CreateReports < ActiveRecord::Migration
    def change
    create_table :reports do |t|
    t.references :user
    t.jsonb :data, null: false, default: "{}"
    t.timestamps null: false
    end
    end
    end
    ```

    We want our Report#data to have at least 2 keys `devise_id` and `version`, so a valid JSON should be like below:

    ```json
    {
    "devise_id": "devise-id-is-a-string",
    "version": "5.56.6"
    }
    ```

    To test our model validation, we write rspec code like below:

    ```rb
    # spec/models/report_spec.rb
    RSpec.describe Report, type: :model do
    describe 'validates data column' do
    # We use Factory girl to create fake record
    subject(:report) { create(:report, data: data) }
    let(:valid_data) do
    {
    devise_id: 'devise-id-is-a-string',
    version: '5.56.6'
    }
    end

    describe 'valid data' do
    let(:data) { valid_data }
    it 'creates report' do
    expect { report }.to change { Report.count }.by(1)
    end
    end

    describe 'invalid data' do
    context 'when missing devise_id' do
    let(:data) { valid_data.except(:devise_id) }

    it 'raise validation error' do
    expect { report }.to raise_error(ActiveRecord::RecordInvalid)
    end
    end

    context 'when missing version' do
    let(:data) { valid_data.except(:version) }

    it 'raise validation error' do
    expect { report }.to raise_error(ActiveRecord::RecordInvalid)
    end
    end
    end
    end
    end
    ```



    Install gem

    ```sh
    # Gemfile
    gem 'activerecord_json_validator'
    ```

    Add validation into `Report`

    ```rb
    # app/models/report.rb
    class Report < ActiveRecord::Base
    JSON_SCHEMA = "#{Rails.root}/app/models/schemas/report/data.json"

    belongs_to :user

    validates :data, presence: true, json: { schema: JSON_SCHEMA }
    end
    ```

    Add JSON schema file

    I prefer add json file into `app/models/schemas/report/data.json`

    ```json
    // app/models/schemas/report/data.json
    {
    "$schema": "http://yourdomain.com/somewhere/report/data",
    "type": "object",
    "properties": {
    "devise_id": {
    "type": "string"
    },
    "version": {
    "type": "string"
    }
    },
    "required": [
    "devise_id",
    "version"
    ]
    }
    ```

    Now all test passed.

    ### Test your API endpoints with schema matcher

    Now it's time to add some api endpoint response test.

    Assume we have users api `GET /users` and `GET /users/:id`, lets define our response:

    ```json
    // GET /users
    {
    "users": [
    {
    "id": 1,
    "name": "John John Slater",
    "email": "[email protected]",
    "is_good_surfer": true,
    "updated_at": "timestamp",
    "created_at": "timestamp"
    }
    ]
    }

    // GET /users/:id
    {
    "user": {
    "id": 1,
    "name": "John John Slater",
    "email": "[email protected]",
    "is_good_surfer": true,
    "updated_at": "2017-02-01T10:00:54.326+10:00",
    "created_at": "2017-02-01T10:00:54.326+10:00"
    }
    }
    ```

    Lets write some tests first

    I'm using Rspec and found out there is a gem called [json_matcher](https://github.com/thoughtbot/json_matchers), it's also based on gem [json-schema](https://github.com/ruby-json-schema/json-schema), so you can choose either one to implement your test, check [this post](https://robots.thoughtbot.com/validating-json-schemas-with-an-rspec-matcher) and you will have more idea how to do this.

    Ok, time to make our hand dirty.

    First do some setup:

    ```rb
    # Gemfile
    gem 'json_matchers'

    # spec/spec_helper.rb
    require "json_matchers/rspec"

    # spec/support/json_matchers.rb
    JsonMatchers.schema_root = "controller/schemas"
    ```

    And write tests

    ```rb
    RSpec.describe User, type: :request do
    describe 'GET /users' do
    let!(:user) { create(:user) }
    subject! { get '/users' }

    specify do
    expect(response).to be_success
    expect(response).to match_response_schema('users')
    end
    end

    describe 'GET /users/:id' do
    let(:user) { create(:user) }
    subject! { get "/users/#{user.id}" }

    specify do
    expect(response).to be_success
    expect(response).to match_response_schema('user')
    end
    end
    end
    ```

    Test should be failed because we haven't added JSON schema file yet.

    Add schema file for user, notice we put user object definitions into `definitions` so we can reuse when we define `users.json`

    ```json
    // app/controllers/schemas/user.json
    {
    "type": "object",
    "required": ["user"],
    "properties": {
    "user": {
    "$ref": "#/definitions/user"
    }
    },
    "definitions": {
    "user": {
    "type": "object",
    "required": [
    "id",
    "name",
    "email",
    "is_good_surfer",
    "updated_at",
    "created_at"
    ],
    "properties": {
    "id": {
    "type": "integer"
    },
    "name": {
    "type": "string"
    },
    "email": {
    "type": "string"
    },
    "is_good_surfer": {
    "type": "boolean"
    },
    "updated_at": {
    "type": "string"
    },
    "created_at": {
    "type": "string"
    }
    }
    }
    }
    }
    ```

    ```json
    // app/controllers/chemas/users.json
    {
    "type": "object",
    "required": ["users"],
    "properties": {
    "users": {
    "items": {
    "$ref": "user.json#/definitions/user"
    },
    "type": "array"
    }
    }
    }
    ```

    Now test passed.

    ### Further reading

    * How to write generic JSON schema test and apply to every api endpoints?
    * How to expose your JSON schema so you can share/use it either at rails project or other repos

    ### References

    * https://brandur.org/elegant-apis
    * https://robots.thoughtbot.com/validating-json-schemas-with-an-rspec-matcher
    * https://github.com/mirego/activerecord_json_validator
    * https://github.com/ruby-json-schema/json-schema
    * https://robots.thoughtbot.com/validating-the-formkeep-api
    * https://spacetelescope.github.io/understanding-json-schema/
    * http://jsonschema.net/#/