Skip to content

Instantly share code, notes, and snippets.

@arun-gupta

arun-gupta/readme.md

Last active September 18, 2024 17:21
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save arun-gupta/5f02b5a57030ba8f975a4c328178ffe8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save arun-gupta/5f02b5a57030ba8f975a4c328178ffe8 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    # More OPEA Examples on AWS and Docker Compose
    # More OPEA Examples using Docker Compose

    - Spin up Ubuntu 24.04 VM and install Docker following the instructions at https://gist.github.com/arun-gupta/7e9f080feff664fbab878b26d13d83d7

  2. arun-gupta revised this gist Sep 18, 2024. No changes.
  3. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 35 additions and 4 deletions.
    39 changes: 35 additions & 4 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -147,10 +147,41 @@
    ```
    ubuntu@ip-172-31-54-128:~$ sudo docker container ls
    ```
    Check the logs:
    ```
    2024-09-18T17:04:48.079544Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
    2024-09-18T17:04:58.095135Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
    2024-09-18T17:05:08.110327Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
    2024-09-18T17:05:10.114867Z INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-0
    2024-09-18T17:05:10.213727Z INFO shard-manager: text_generation_launcher: Shard ready in 422.780717591s rank=0
    2024-09-18T17:05:10.298830Z INFO text_generation_launcher: Starting Webserver
    2024-09-18T17:05:10.440064Z INFO text_generation_router_v3: backends/v3/src/lib.rs:90: Warming up model
    2024-09-18T17:05:39.868187Z INFO text_generation_launcher: Cuda Graphs are disabled (CUDA_GRAPHS=None).
    2024-09-18T17:05:39.868785Z INFO text_generation_router_v3: backends/v3/src/lib.rs:102: Setting max batch total tokens to 45136
    2024-09-18T17:05:39.869908Z INFO text_generation_router_v3: backends/v3/src/lib.rs:126: Using backend V3
    2024-09-18T17:05:39.876810Z INFO text_generation_router::server: router/src/server.rs:1651: Using the Hugging Face API
    2024-09-18T17:05:39.895544Z INFO hf_hub: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hf-hub-0.3.2/src/lib.rs:55: Token file not found "/root/.cache/huggingface/token"
    2024-09-18T17:05:40.482470Z INFO text_generation_router::server: router/src/server.rs:2349: Serving revision 822086d00e1e61b0c3f99bea3577a916b4360001 of model haoranxu/ALMA-13B
    2024-09-18T17:05:40.483950Z INFO text_generation_router::server: router/src/server.rs:1781: Using config Some(Llama)
    2024-09-18T17:05:40.483965Z WARN text_generation_router::server: router/src/server.rs:1783: Could not find a fast tokenizer implementation for haoranxu/ALMA-13B
    2024-09-18T17:05:40.483967Z WARN text_generation_router::server: router/src/server.rs:1784: Rust input length validation and truncation is disabled
    2024-09-18T17:05:40.483989Z WARN text_generation_router::server: router/src/server.rs:1928: Invalid hostname, defaulting to 0.0.0.0
    2024-09-18T17:05:40.490701Z INFO text_generation_router::server: router/src/server.rs:2311: Connected
    ```
    - Access the service using `cURL` command:
    ```
    curl http://${host_ip}:8888/v1/docsum \
    -H "Content-Type: application/json" \
    -d '{"messages": "Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5."}'
    ubuntu@ip-172-31-49-59:~$ curl http://${host_ip}:8888/v1/translation -H "Content-Type: application/json" -d '{
    "language_from": "Hindi","language_to": "English","source_language": "आप कैसे हो "}'
    data: b' How'
    data: b' are'
    data: b' you'
    data: b'?'
    data: b'</s>'
    data: [DONE]
    ```
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/835.

  4. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -127,7 +127,7 @@
    Instructions should be updated to use pre-built images: https://github.com/opea-project/GenAIExamples/issues/836.
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.54.128"
    export host_ip="172.31.49.59" #private IP address
    export LLM_MODEL_ID="haoranxu/ALMA-13B"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
    export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
  5. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 36 additions and 1 deletion.
    37 changes: 36 additions & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -116,6 +116,41 @@
    -H "Content-Type: application/json" \
    -d '{"messages": "Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5."}'
    ```
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/824
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/835.

    ## Translation

    - Pull the Docker image:
    ```
    sudo docker pull opea/translation:latest
    ```
    Instructions should be updated to use pre-built images: https://github.com/opea-project/GenAIExamples/issues/836.
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.54.128"
    export LLM_MODEL_ID="haoranxu/ALMA-13B"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
    export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8888/v1/translation"
    ```
    - Download Docker Compose file:
    ```
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/Translation/docker_compose/intel/cpu/xeon/compose.yaml
    ```
    - Start the application:
    ```
    sudo docker compose -f compose.yaml up -d
    ```
    - Verify the list of containers:
    ```
    ubuntu@ip-172-31-54-128:~$ sudo docker container ls
    ```
    - Access the service using `cURL` command:
    ```
    curl http://${host_ip}:8888/v1/docsum \
    -H "Content-Type: application/json" \
    -d '{"messages": "Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5."}'
    ```
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/835.
  6. arun-gupta revised this gist Sep 18, 2024. No changes.
  7. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 6 additions and 5 deletions.
    11 changes: 6 additions & 5 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -103,11 +103,12 @@
    ```
    - Verify the list of containers:
    ```
    ubuntu@ip-172-31-50-223:~$ sudo docker container ls
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    ba99bf66e45b opea/codegen-ui:latest "docker-entrypoint.s…" 7 minutes ago Up 7 minutes 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp codegen-xeon-ui-server
    31a19966946b opea/codegen:latest "python codegen.py" 7 minutes ago Up 7 minutes 0.0.0.0:7778->7778/tcp, :::7778->7778/tcp codegen-xeon-backend-server
    1c1649d31187 opea/llm-tgi:latest "bash entrypoint.sh" 7 minutes ago Up 7 minutes 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp llm-tgi-server
    ubuntu@ip-172-31-54-128:~$ sudo docker container ls
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    68ca3c32ecdd opea/docsum-ui:latest "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp docsum-xeon-ui-server
    26b0d896b3c7 opea/docsum:latest "python docsum.py" About a minute ago Up About a minute 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp docsum-xeon-backend-server
    bd0606afb0fd opea/llm-docsum-tgi:latest "bash entrypoint.sh" About a minute ago Up About a minute 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp llm-docsum-server
    06d4446bd9b1 ghcr.io/huggingface/text-generation-inference:sha-e4201f4-intel-cpu "text-generation-lau…" 2 minutes ago Up About a minute 0.0.0.0:8008->80/tcp, [::]:8008->80/tcp tgi-service
    ```
    - Access the service using `cURL` command:
    ```
  8. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 18 additions and 3 deletions.
    21 changes: 18 additions & 3 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -86,7 +86,7 @@
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.54.128"
    export HUGGINGFACEHUB_API_TOKEN="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"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    export LLM_MODEL_ID="Intel/neural-chat-7b-v3-3"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    @@ -95,11 +95,26 @@
    ```
    - Download Docker Compose file:
    ```
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/CodeTrans/docker_compose/intel/cpu/xeon/compose.yaml
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/DocSum/docker_compose/intel/cpu/xeon/compose.yaml
    ```
    - Start the application:
    ```
    sudo docker compose -f compose.yaml up -d
    ```
    This is causing https://github.com/opea-project/GenAIExamples/issues/830
    - Verify the list of containers:
    ```
    ubuntu@ip-172-31-50-223:~$ sudo docker container ls
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    ba99bf66e45b opea/codegen-ui:latest "docker-entrypoint.s…" 7 minutes ago Up 7 minutes 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp codegen-xeon-ui-server
    31a19966946b opea/codegen:latest "python codegen.py" 7 minutes ago Up 7 minutes 0.0.0.0:7778->7778/tcp, :::7778->7778/tcp codegen-xeon-backend-server
    1c1649d31187 opea/llm-tgi:latest "bash entrypoint.sh" 7 minutes ago Up 7 minutes 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp llm-tgi-server
    ```
    - Access the service using `cURL` command:
    ```
    curl http://${host_ip}:8888/v1/docsum \
    -H "Content-Type: application/json" \
    -d '{"messages": "Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5."}'
    ```
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/824


  9. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 27 additions and 1 deletion.
    28 changes: 27 additions & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -75,5 +75,31 @@
    ```
    sudo docker compose -f compose.yaml up -d
    ```
    This is causing
    This is causing https://github.com/opea-project/GenAIExamples/issues/830

    ## DocSum

    - Pull the Docker image:
    ```
    sudo docker pull opea/docsum:latest
    ```
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.54.128"
    export HUGGINGFACEHUB_API_TOKEN="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"
    export LLM_MODEL_ID="Intel/neural-chat-7b-v3-3"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8888/v1/docsum"
    ```
    - Download Docker Compose file:
    ```
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/CodeTrans/docker_compose/intel/cpu/xeon/compose.yaml
    ```
    - Start the application:
    ```
    sudo docker compose -f compose.yaml up -d
    ```
    This is causing https://github.com/opea-project/GenAIExamples/issues/830

  10. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 2 additions and 15 deletions.
    17 changes: 2 additions & 15 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -75,18 +75,5 @@
    ```
    sudo docker compose -f compose.yaml up -d
    ```
    - Verify the list of containers:
    ```
    ubuntu@ip-172-31-50-223:~$ sudo docker container ls
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    ba99bf66e45b opea/codegen-ui:latest "docker-entrypoint.s…" 7 minutes ago Up 7 minutes 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp codegen-xeon-ui-server
    31a19966946b opea/codegen:latest "python codegen.py" 7 minutes ago Up 7 minutes 0.0.0.0:7778->7778/tcp, :::7778->7778/tcp codegen-xeon-backend-server
    1c1649d31187 opea/llm-tgi:latest "bash entrypoint.sh" 7 minutes ago Up 7 minutes 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp llm-tgi-server
    ```
    - Access the service using `cURL` command:
    ```
    curl http://${host_ip}:7778/v1/codegen \
    -H "Content-Type: application/json" \
    -d '{"messages": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}'
    ```
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/824
    This is causing

  11. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 14 additions and 14 deletions.
    28 changes: 14 additions & 14 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -52,20 +52,20 @@
    ```
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="External_Public_IP"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    # Example: NGINX_PORT=80
    export NGINX_PORT=${your_nginx_port}
    export LLM_MODEL_ID="HuggingFaceH4/mistral-7b-grok"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7777/v1/codetrans"
    export FRONTEND_SERVICE_IP=${host_ip}
    export FRONTEND_SERVICE_PORT=5173
    export BACKEND_SERVICE_NAME=codetrans
    export BACKEND_SERVICE_IP=${host_ip}
    export BACKEND_SERVICE_PORT=7777
    export host_ip="External_Public_IP"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    # Example: NGINX_PORT=80
    export NGINX_PORT=${your_nginx_port}
    export LLM_MODEL_ID="HuggingFaceH4/mistral-7b-grok"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7777/v1/codetrans"
    export FRONTEND_SERVICE_IP=${host_ip}
    export FRONTEND_SERVICE_PORT=5173
    export BACKEND_SERVICE_NAME=codetrans
    export BACKEND_SERVICE_IP=${host_ip}
    export BACKEND_SERVICE_PORT=7777
    ```
    - Download Docker Compose file:
    ```
  12. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 19 additions and 8 deletions.
    27 changes: 19 additions & 8 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -46,19 +46,30 @@

    ## CodeTrans

    - Pull the Docker image:
    ```
    sudo docker pull opea/codetrans:latest
    ```
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.50.223"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    export LLM_MODEL_ID="deepseek-ai/deepseek-coder-6.7b-instruct"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
    export host_ip="External_Public_IP"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    # Example: NGINX_PORT=80
    export NGINX_PORT=${your_nginx_port}
    export LLM_MODEL_ID="HuggingFaceH4/mistral-7b-grok"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7777/v1/codetrans"
    export FRONTEND_SERVICE_IP=${host_ip}
    export FRONTEND_SERVICE_PORT=5173
    export BACKEND_SERVICE_NAME=codetrans
    export BACKEND_SERVICE_IP=${host_ip}
    export BACKEND_SERVICE_PORT=7777
    ```
    - Download Docker Compose file:
    ```
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/CodeGen/docker_compose/intel/cpu/xeon/compose.yaml
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/CodeTrans/docker_compose/intel/cpu/xeon/compose.yaml
    ```
    - Start the application:
    ```
  13. arun-gupta revised this gist Sep 18, 2024. No changes.
  14. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 47 additions and 2 deletions.
    49 changes: 47 additions & 2 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,51 @@
    # CodeGen using OPEA on AWS and Docker Compose
    # More OPEA Examples on AWS and Docker Compose

    - Spin up Ubuntu 24.04 VM and install Docker following the instructions at https://gist.github.com/arun-gupta/7e9f080feff664fbab878b26d13d83d7


    ## CodeGen

    - Pull the Docker image:
    ```
    sudo docker pull opea/codegen:latest
    ```
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.50.223"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    export LLM_MODEL_ID="deepseek-ai/deepseek-coder-6.7b-instruct"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
    ```
    - Download Docker Compose file:
    ```
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/CodeGen/docker_compose/intel/cpu/xeon/compose.yaml
    ```
    - Start the application:
    ```
    sudo docker compose -f compose.yaml up -d
    ```
    - Verify the list of containers:
    ```
    ubuntu@ip-172-31-50-223:~$ sudo docker container ls
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    ba99bf66e45b opea/codegen-ui:latest "docker-entrypoint.s…" 7 minutes ago Up 7 minutes 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp codegen-xeon-ui-server
    31a19966946b opea/codegen:latest "python codegen.py" 7 minutes ago Up 7 minutes 0.0.0.0:7778->7778/tcp, :::7778->7778/tcp codegen-xeon-backend-server
    1c1649d31187 opea/llm-tgi:latest "bash entrypoint.sh" 7 minutes ago Up 7 minutes 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp llm-tgi-server
    ```
    - Access the service using `cURL` command:
    ```
    curl http://${host_ip}:7778/v1/codegen \
    -H "Content-Type: application/json" \
    -d '{"messages": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}'
    ```
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/824


    ## CodeTrans

    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.50.223"
    @@ -33,4 +78,4 @@
    -H "Content-Type: application/json" \
    -d '{"messages": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}'
    ```
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/824
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/824
  15. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -33,4 +33,4 @@
    -H "Content-Type: application/json" \
    -d '{"messages": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}'
    ```
    This is currently causing
    This is currently causing https://github.com/opea-project/GenAIExamples/issues/824
  16. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 20 additions and 1 deletion.
    21 changes: 20 additions & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -14,4 +14,23 @@
    - Download Docker Compose file:
    ```
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/CodeGen/docker_compose/intel/cpu/xeon/compose.yaml
    ```
    ```
    - Start the application:
    ```
    sudo docker compose -f compose.yaml up -d
    ```
    - Verify the list of containers:
    ```
    ubuntu@ip-172-31-50-223:~$ sudo docker container ls
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    ba99bf66e45b opea/codegen-ui:latest "docker-entrypoint.s…" 7 minutes ago Up 7 minutes 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp codegen-xeon-ui-server
    31a19966946b opea/codegen:latest "python codegen.py" 7 minutes ago Up 7 minutes 0.0.0.0:7778->7778/tcp, :::7778->7778/tcp codegen-xeon-backend-server
    1c1649d31187 opea/llm-tgi:latest "bash entrypoint.sh" 7 minutes ago Up 7 minutes 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp llm-tgi-server
    ```
    - Access the service using `cURL` command:
    ```
    curl http://${host_ip}:7778/v1/codegen \
    -H "Content-Type: application/json" \
    -d '{"messages": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}'
    ```
    This is currently causing
  17. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,4 +10,8 @@
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
    ```
    - Download Docker Compose file:
    ```
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/main/CodeGen/docker_compose/intel/cpu/xeon/compose.yaml
    ```
  18. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    # CodeTrans using OPEA on AWS and Docker Compose
    # CodeGen using OPEA on AWS and Docker Compose

    - Spin up Ubuntu 24.04 VM and install Docker following the instructions at https://gist.github.com/arun-gupta/7e9f080feff664fbab878b26d13d83d7
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
  19. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,7 +5,7 @@
    ```
    export host_ip="172.31.50.223"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    export LLM_MODEL_ID="meta-llama/CodeLlama-7b-hf"
    export LLM_MODEL_ID="deepseek-ai/deepseek-coder-6.7b-instruct"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
  20. arun-gupta revised this gist Sep 18, 2024. 1 changed file with 10 additions and 1 deletion.
    11 changes: 10 additions & 1 deletion readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,13 @@
    # CodeTrans using OPEA on AWS and Docker Compose

    - Spin up Ubuntu 24.04 VM and install Docker following the instructions at https://gist.github.com/arun-gupta/7e9f080feff664fbab878b26d13d83d7
    -
    - Replace HuggingFace API token and private IP address of the host below and copy the contents in a file named `.env`:
    ```
    export host_ip="172.31.50.223"
    export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
    export LLM_MODEL_ID="meta-llama/CodeLlama-7b-hf"
    export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
    export MEGA_SERVICE_HOST_IP=${host_ip}
    export LLM_SERVICE_HOST_IP=${host_ip}
    export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
    ```
  21. arun-gupta created this gist Sep 18, 2024.
    4 changes: 4 additions & 0 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,4 @@
    # CodeTrans using OPEA on AWS and Docker Compose

    - Spin up Ubuntu 24.04 VM and install Docker following the instructions at https://gist.github.com/arun-gupta/7e9f080feff664fbab878b26d13d83d7
    -