Skip to content

Instantly share code, notes, and snippets.

@bsod90

bsod90/faiss_example.py

Last active August 3, 2019 20:51
Show Gist options
  • Save bsod90/b50c8f9117f66fca12bd5df5b61b4b6c to your computer and use it in GitHub Desktop.
Save bsod90/b50c8f9117f66fca12bd5df5b61b4b6c to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. bsod90 revised this gist Aug 3, 2019. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions faiss_example.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,5 @@
    import faiss
    from encoder import UniversalEncoder

    # Don't forget to start the encoder container locally and open the 8501 port

  2. bsod90 revised this gist Aug 3, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion faiss_example.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,6 @@
    import faiss

    # Don't forget to start an encoder container locally and open the 8501 port
    # Don't forget to start the encoder container locally and open the 8501 port

    encoder = UniversalEncoder(host='localhost', port=8501)
    data = [
  3. bsod90 created this gist Aug 3, 2019.
    62 changes: 62 additions & 0 deletions faiss_example.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,62 @@
    import faiss

    # Don't forget to start an encoder container locally and open the 8501 port

    encoder = UniversalEncoder(host='localhost', port=8501)
    data = [
    'What color is chameleon?',
    'When is the festival of colors?',
    'When is the next music festival?',
    'How far is the moon?',
    'How far is the sun?',
    'What happens when the sun goes down?',
    'What we do in the shadows?',
    'What is the meaning of all this?',
    'What is the meaning of Russel\'s paradox?',
    'How are you doing?'
    ]
    encoded_data = encoder.encode(data)

    # We're going to use a regular "Flat Inner Product" index here
    # as we're not storing millions of rows.
    # We'll also wrap it in the IndexIDMap just to demonstrate
    # how you can store actual document IDs right in the index
    # and get them back along with the found vectors when searching

    index = faiss.IndexIDMap(faiss.IndexFlatIP(encoder.FEATURE_SIZE))
    index.add_with_ids(encoded_data, np.array(range(0, len(data))))

    def search(query):
    query_vector = encoder.encode([query])
    k = 1
    top_k = index.search(query_vector, k)
    return [
    data[_id] for _id in top_k[1].tolist()[0]
    ]

    # Examples:

    print(search("When is Holi?"))

    # >>> ['When is the festival of colors?']

    print(search("How far is the Earth satelite?"))

    # >>> ['How far is the moon?']

    print(search("How far is the shiny yellow thing?"))

    # >>> ['How far is the sun?']

    print(search("What is the meaning of life?"))

    # >>> ['What is the meaning of all this?']

    print(search("Are we vampires?"))

    # >>> ['What we do in the shadows?']

    print(search("как оно?"))

    # >>> ['How are you doing?']