Selection of the main GPU

[olegk0]

Hello
I don't understand how I can select the main GPU in a multi-GPU system.
I assumed the maingpu option was responsible for this, but nothing changes.

For clarity, here's a snippet of the log:

Welcome to KoboldCpp - Version 1.110
...
Using Vulkan backend
ggml_vulkan: Found 2 Vulkan devices:

ggml_vulkan: 0 = Intel(R) Arc(tm) A380 Graphics (DG2) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none

ggml_vulkan: 1 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

Vulkan: Using device 0

I would like to set "AMD Radeon" as my primary GPU, but I couldn't do it.

I might be wrong, but in the end I would like the Radeon GPU memory to be used first.

[henk717]

There is no such thing as "memory used first" its tied to which device a layer is on.
What you can try is the tensor_split option which lets you define the ratio for each device.

[LostRuins]

That log seems to be coming from image generation, correct? Currently image gen is configured to use the first device to store loaded image tensors.

If you're running single gpu mode, then --usevulkan X would work to select the device, or you can do --usevulkan --device Vulkan0 for example. But for a multigpu setup image weights currently cannot be specifically assigned to a GPU.

Unless you're not using images?

For text generation, you can indeed specify --maingpu, this is passed to the llama.cpp backend and is used to pick a GPU for intermediate calculations e.g. temp GPU buffers.

[LostRuins]

Might be possible to pipe the maingpu flag here tho. @wbruna do you think its a good idea?

koboldcpp/otherarch/sdcpp/stable-diffusion.cpp

Lines 199 to 254 in 9ba8c7a

	void init_backend() {
	#ifdef SD_USE_CUDA
	LOG_DEBUG("Using CUDA backend");
	backend = ggml_backend_cuda_init(0);
	#endif
	#ifdef SD_USE_METAL
	LOG_DEBUG("Using Metal backend");
	backend = ggml_backend_metal_init();
	#endif
	#ifdef SD_USE_VULKAN
	LOG_DEBUG("Using Vulkan backend");
	size_t device = 0;
	const int device_count = ggml_backend_vk_get_device_count();
	if (device_count) {
	const char* SD_VK_DEVICE = getenv("SD_VK_DEVICE");
	if (SD_VK_DEVICE != nullptr) {
	std::string sd_vk_device_str = SD_VK_DEVICE;
	try {
	device = std::stoull(sd_vk_device_str);
	} catch (const std::invalid_argument&) {
	LOG_WARN("SD_VK_DEVICE environment variable is not a valid integer (%s). Falling back to device 0.", SD_VK_DEVICE);
	device = 0;
	} catch (const std::out_of_range&) {
	LOG_WARN("SD_VK_DEVICE environment variable value is out of range for `unsigned long long` type (%s). Falling back to device 0.", SD_VK_DEVICE);
	device = 0;
	}
	if (device >= device_count) {
	LOG_WARN("Cannot find targeted vulkan device (%llu). Falling back to device 0.", device);
	device = 0;
	}
	}
	LOG_INFO("Vulkan: Using device %llu", device);
	backend = ggml_backend_vk_init(device);
	}
	if (!backend) {
	LOG_WARN("Failed to initialize Vulkan backend");
	}
	#endif
	#ifdef SD_USE_OPENCL
	LOG_DEBUG("Using OpenCL backend");
	// ggml_log_set(ggml_log_callback_default, nullptr); // Optional ggml logs
	backend = ggml_backend_opencl_init();
	if (!backend) {
	LOG_WARN("Failed to initialize OpenCL backend");
	}
	#endif
	#ifdef SD_USE_SYCL
	LOG_DEBUG("Using SYCL backend");
	backend = ggml_backend_sycl_init(0);
	#endif
	if (!backend) {
	LOG_DEBUG("Using CPU backend");
	backend = ggml_backend_cpu_init();
	}
	}

[wbruna]

for a multigpu setup image weights currently cannot be specifically assigned to a GPU

Not by a config option, but the GGML_VK_VISIBLE_DEVICES environment variable can reorder GPUs. So GGML_VK_VISIBLE_DEVICES=1,0 could be used to swap the GPUs for both images and text; then, if llama.cpp still prefers avoiding the iGPU, Koboldcpp normal config options should be able to change it.

Oh, looking at that code snippet, I remembered about another one: SD_VK_DEVICE=1 would probably work, too.

[olegk0]

There is no such thing as "memory used first" its tied to which device a layer is on. What you can try is the tensor_split option which lets you define the ratio for each device.

Yes, I probably expressed myself incorrectly, but I tried tensor_split and it had no visible effect.

[olegk0]

That log seems to be coming from image generation, correct? Currently image gen is configured to use the first device to store loaded image tensors.

If you're running single gpu mode, then --usevulkan X would work to select the device, or you can do --usevulkan --device Vulkan0 for example. But for a multigpu setup image weights currently cannot be specifically assigned to a GPU.

Unless you're not using images?

For text generation, you can indeed specify --maingpu, this is passed to the llama.cpp backend and is used to pick a GPU for intermediate calculations e.g. temp GPU buffers.

Thanks, I forgot to mention that this is image generation, and I'm specifically interested in multi-GPU mode.

[olegk0]

for a multigpu setup image weights currently cannot be specifically assigned to a GPU

Not by a config option, but the GGML_VK_VISIBLE_DEVICES environment variable can reorder GPUs. So GGML_VK_VISIBLE_DEVICES=1,0 could be used to swap the GPUs for both images and text; then, if llama.cpp still prefers avoiding the iGPU, Koboldcpp normal config options should be able to change it.

Oh, looking at that code snippet, I remembered about another one: SD_VK_DEVICE=1 would probably work, too.

Thank you very much, this is exactly what I wanted.
Both options worked.

[LostRuins]

@wbruna I added --sdmaingpu allowing image models to be independently placed on any gpu
seems like a good QoL thing to have

it should work for cuda too, though i'd have to try

fdfb713

[wbruna]

@LostRuins , hm, ok but... that should probably be "SD_CUDA_DEVICE" or something like that.

We may also get much more flexibility eventually: leejet/stable-diffusion.cpp#1184

[LostRuins]

maybe, though the current way kobo is built doesn't allow for mixing backends because the dependencies are only built for that specific target (i.e. nocuda build literally does not include cuda at all)

[wbruna]

maybe, though the current way kobo is built doesn't allow for mixing backends because the dependencies are only built for that specific target (i.e. nocuda build literally does not include cuda at all)

"backend" there is more specific than "CUDA" or "Vulkan": it allows e.g. put CLIP on Vulkan2, run diffusion on Vulkan1, VAE on Vulkan0.

[LostRuins]

Ah cool. Could be good to have then eventually