Skip to content

Instantly share code, notes, and snippets.

@sroecker

sroecker/debug_pytorch_rocm.py

Last active May 15, 2024 19:51
Show Gist options
  • Save sroecker/f52646023d45ab4e281ddee05d6ef2a5 to your computer and use it in GitHub Desktop.
Save sroecker/f52646023d45ab4e281ddee05d6ef2a5 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. sroecker revised this gist Dec 21, 2023. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions magic_variables.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,9 +1,11 @@
    # first test if it works without hip
    # hide all hip devices
    HIP_VISIBLE_DEVICES=
    # then hide all other than first card
    python debug_pytorch_rocm.py

    # some programs don't work with multi gpu setups, to hide all other than first card
    HIP_VISIBLE_DEVICES=0
    # some programs don't work with multi gpu setups
    python debug_pytorch_rocm.py

    # find out the architecture of your card
    amdgpu-arch
  2. sroecker revised this gist Dec 21, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions magic_variables.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -14,6 +14,7 @@ amdgpu-arch
    # pytorch was only compiled for certain supported archs though
    # https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html
    # so we tell it to use a compatible supported architecture
    # PLEASE MAKE SURE TO CHOOSE A COMPATIBLE ARCH
    export PYTORCH_ROCM_ARCH="gfx1030"
    export HSA_OVERRIDE_GFX_VERSION=10.3.0

  3. sroecker revised this gist Dec 19, 2023. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions magic_variables.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -7,19 +7,19 @@ HIP_VISIBLE_DEVICES=0

    # find out the architecture of your card
    amdgpu-arch
    # architecture of my card, 6600 XT
    # architecture of my card, 6600 M (same for XT) is gfx1032
    #export PYTORCH_ROCM_ARCH="gfx1032"
    #export HSA_OVERRIDE_GFX_VERSION=10.3.2

    # pytorch was only compiled for this supported arch though
    # so we tell it to use this compatible architecture
    # pytorch was only compiled for certain supported archs though
    # https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html
    # so we tell it to use a compatible supported architecture
    export PYTORCH_ROCM_ARCH="gfx1030"
    export HSA_OVERRIDE_GFX_VERSION=10.3.0
    # set compiler target, HSA_OVERRIDE_GFX_VERSION should set that too
    # export HCC_AMDGPU_TARGET=gfx1030
    # for supported architectures see
    # https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html

    # NOT NEEDED set compiler target, HSA_OVERRIDE_GFX_VERSION should set that too
    # export HCC_AMDGPU_TARGET=gfx1030

    python debug_pytorch_rocm.py
    # this should work now :D
    # if not, increase log level from 1 to 3
  4. sroecker revised this gist Dec 19, 2023. 2 changed files with 34 additions and 17 deletions.
    26 changes: 17 additions & 9 deletions debug_pytorch_rocm.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,15 +1,23 @@
    import torch

    print(torch.__version__)
    print(torch.cuda.is_available())
    print(torch.cuda.device_count())
    print(torch.cuda.current_device())
    print(torch.cuda.get_device_name(torch.cuda.current_device()))
    print("pytorch version: " + torch.__version__)
    print("CUDA available: " + str(torch.cuda.is_available()))
    print("device count: " + str(torch.cuda.device_count()))
    print("current device: " + str(torch.cuda.current_device()))
    print("device name: " + torch.cuda.get_device_name(torch.cuda.current_device()))
    print("backend:")
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(device)

    print("testing")
    res = torch.rand(3, 3).to(device)
    print("small matmul")
    a = torch.rand(3, 3).to(device)
    b = torch.rand(3, 3).to(device)
    res = torch.matmul(a, b)
    print(res)
    res = torch.rand(1000, 1000).to(device)
    print(res)
    print(res.size())
    print("larger matmul")
    a = torch.rand(1280, 1280).to(device)
    b = torch.rand(1280, 1280).to(device)
    res = torch.matmul(a, b)
    print(res)
    print(res.size())
    25 changes: 17 additions & 8 deletions magic_variables.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,17 +1,26 @@
    # first test if it works without hip
    # hide all hip devices
    HIP_VISIBLE_DEVICES=
    # hide all other than first card
    # then hide all other than first card
    HIP_VISIBLE_DEVICES=0
    # some programs don't work with multi gpu setups

    # find out the architecture of your card
    amdgpu-arch
    # architecture of my card, 6600 XT
    export PYTORCH_ROCM_ARCH="gfx1032"
    export HSA_OVERRIDE_GFX_VERSION=10.3.2
    #export PYTORCH_ROCM_ARCH="gfx1032"
    #export HSA_OVERRIDE_GFX_VERSION=10.3.2

    # pytorch was only compiled for this supported arch though
    # so we tell it to use this compatible architecture
    export PYTORCH_ROCM_ARCH="gfx1030"
    export HSA_OVERRIDE_GFX_VERSION=10.3.0
    # set compiler target, HSA_OVERRIDE_GFX_VERSION should set that too
    # export HCC_AMDGPU_TARGET=gfx1030
    # for supported architectures see
    # https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html

    # to debug, verbosity level 1-3
    AMD_LOG_LEVEL=3

    # set compiler target, HSA_OVERRIDE should set that too
    export HCC_AMDGPU_TARGET=gfx1030
    python debug_pytorch_rocm.py
    # this should work now :D
    # if not, increase log level from 1 to 3
    AMD_LOG_LEVEL=3
  5. sroecker renamed this gist Dec 18, 2023. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → magic_variables.sh
    View file
    File renamed without changes.
  6. sroecker created this gist Dec 18, 2023.
    15 changes: 15 additions & 0 deletions debug_pytorch_rocm.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,15 @@
    import torch

    print(torch.__version__)
    print(torch.cuda.is_available())
    print(torch.cuda.device_count())
    print(torch.cuda.current_device())
    print(torch.cuda.get_device_name(torch.cuda.current_device()))
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(device)

    print("testing")
    res = torch.rand(3, 3).to(device)
    print(res)
    res = torch.rand(1000, 1000).to(device)
    print(res)
    17 changes: 17 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,17 @@
    # hide all hip devices
    HIP_VISIBLE_DEVICES=
    # hide all other than first card
    HIP_VISIBLE_DEVICES=0

    # architecture of my card, 6600 XT
    export PYTORCH_ROCM_ARCH="gfx1032"
    export HSA_OVERRIDE_GFX_VERSION=10.3.2
    # pytorch was only compiled for this supported arch though
    export PYTORCH_ROCM_ARCH="gfx1030"
    export HSA_OVERRIDE_GFX_VERSION=10.3.0

    # to debug, verbosity level 1-3
    AMD_LOG_LEVEL=3

    # set compiler target, HSA_OVERRIDE should set that too
    export HCC_AMDGPU_TARGET=gfx1030