Clash with vcpkg exported package using a preset.

[adamryczkowski]

Hi!

We have a project with large dependencies produced by vcpkg export. We use that with CMake presets, defined as

{
  "version": 3,
  "configurePresets": [
    {
      "name": "vcpkg",
      "cacheVariables": {
        "CMAKE_TOOLCHAIN_FILE": "$env{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake",
        "VCPKG_MANIFEST_MODE": "true"
      }
    },
    {
      "name": "exported",
      "cacheVariables": {
        "CMAKE_TOOLCHAIN_FILE": "dep/scripts/buildsystems/vcpkg.cmake",
        "VCPKG_MANIFEST_MODE": "false"
      }
    }

  ]
}

We build with cmake .. --preset exported. Here's the CMakeLists.txt:

cmake_minimum_required(VERSION 3.29)

include(FetchContent)

FetchContent_Declare(CMakeExtraUtils
        GIT_REPOSITORY https://github.com/LecrisUT/CMakeExtraUtils
        GIT_TAG v0.4.1)
FetchContent_MakeAvailable(CMakeExtraUtils) # <-- Here's where the error is thrown

include(DynamicVersion)
dynamic_version(PROJECT_PREFIX MyPOC_)

project(MyPOC VERSION ${PROJECT_VERSION} LANGUAGES CXX)

# Nothing below matters, as the error is thrown before this point.

We get the following error on Ubuntu 22.04:

/usr/bin/cmake --preset exported -S <repo root> -B <repo root>/cmake-build-debug
Preset CMake variables:

  CMAKE_TOOLCHAIN_FILE="dep/scripts/buildsystems/vcpkg.cmake"
  VCPKG_MANIFEST_MODE="false"

-- Found Git: /usr/bin/git (found version "2.34.1")
CMake Error at /usr/share/cmake-3.29/Modules/CMakeDetermineSystem.cmake:152 (message):
  Could not find toolchain file: dep/scripts/buildsystems/vcpkg.cmake
Call Stack (most recent call first):
  cmake-build-debug/_deps/cmakeextrautils-src/CMakeLists.txt:22 (project)

[LecrisUT]

Thanks for the issue. As far as I can read, the issue is that dep/scripts/buildsystems/vcpkg.cmake does not exist, which would make any project() command fail (CMakeExtraUtils is being hit because it also includes a project() in it and it's first in line). Can you try with your project and static version, and let me know if it works by itself?

vcpkg might be a bit tricky overall and I am still figuring out my way around it. I might need to package this there in order to be available via find_package or FetchContent(FIND_PACKAGE_ARGS)

[adamryczkowski]

Many thanks for such a quick reply!! :-)

As far as I can read, the issue is that dep/scripts/buildsystems/vcpkg.cmake does not exist, which would make any project()

Of course it exists, it is just relative path to the repo root. CMake claims it is legal to put relative paths there: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html, :

toolchainFile

An optional string representing the path to the toolchain file. This field supports [macro expansion](https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html#macro-expansion). If a relative path is specified, it is calculated relative to the build directory, and if not found, relative to the source directory. This field takes precedence over any [CMAKE_TOOLCHAIN_FILE](https://cmake.org/cmake/help/latest/variable/CMAKE_TOOLCHAIN_FILE.html#variable:CMAKE_TOOLCHAIN_FILE) value. It is allowed in preset files specifying version 3 or above.

[adamryczkowski]

After putting the full path into the CMakePresets.json, I get another error:

/usr/bin/cmake --preset exported -G Ninja -S <repo root> -B <repo root>/cmake-build-debug
[0/7] Performing update step for 'cmakeextrautils-populate'
error: pathspec 'master' did not match any file(s) known to git
CMake Error at <repo root>/_deps/cmakeextrautils-subbuild/cmakeextrautils-populate-prefix/tmp/cmakeextrautils-populate-gitupdate.cmake:188 (execute_process):
  execute_process failed command indexes:

    1: "Child return code: 1"



FAILED: cmakeextrautils-populate-prefix/src/cmakeextrautils-populate-stamp/cmakeextrautils-populate-update <repo root>/cmake-build-debug/_deps/cmakeextrautils-subbuild/cmakeextrautils-populate-prefix/src/cmakeextrautils-populate-stamp/cmakeextrautils-populate-update 
cd <repo root>/cmake-build-debug/_deps/cmakeextrautils-src && /usr/bin/cmake -Dcan_fetch=YES -P <repo root>/cmake-build-debug/_deps/cmakeextrautils-subbuild/cmakeextrautils-populate-prefix/tmp/cmakeextrautils-populate-gitupdate.cmake
ninja: build stopped: subcommand failed.
Preset CMake variables:

  CMAKE_TOOLCHAIN_FILE="<repo root>/dep/scripts/buildsystems/vcpkg.cmake"
  VCPKG_MANIFEST_MODE="false"


CMake Error at /usr/share/cmake-3.29/Modules/FetchContent.cmake:1714 (message):
  Build step for cmakeextrautils failed: 1
Call Stack (most recent call first):
  /usr/share/cmake-3.29/Modules/FetchContent.cmake:1854:EVAL:2 (__FetchContent_directPopulate)
  /usr/share/cmake-3.29/Modules/FetchContent.cmake:1854 (cmake_language)
  /usr/share/cmake-3.29/Modules/FetchContent.cmake:2081 (FetchContent_Populate)
  CMakeLists.txt:10 (FetchContent_MakeAvailable)


-- Configuring incomplete, errors occurred!

[adamryczkowski]

Please note, that the CMakeExtraUtils cloned repo runs through CMake successfully, in particular it detects the git without any issues.

[adamryczkowski]

I didn't run the CMake of the CMakeExtraUtils/test/DynamicVersion folder though.

EDIT: correction, in fact I did it implicitly, because the CMAKEEXTRAUTILS_TESTS variable is "ON" by default.

[adamryczkowski]

Here're a file that will let you replicate the issue. Put it in the repo root.

vcpkg.json. The actual package should not matter, but here's what I've used:

{
  "name": "mypoc",
  "version-string": "0.0.1",
  "license": "MIT",
  "dependencies": [
    {
      "name": "google-cloud-cpp",
      "version>=": "2.26.0#1"
    },
    {
      "name": "protobuf",
      "version>=": "4.25.1"
    }
  ],
  "builtin-baseline": "5312e9f976e89b256954f571433e34f783dd2d12"
}

If you shallow-clone vcpkg, don't use my builtin-baseline as that would not be avilable locally. Just use sha-1 of the existing commit of the main vcpkg repo.

[adamryczkowski]

First time you will need to use the vcpkg preset to build the repo. Afterwards, type

vcpkg export --zip --output-dir=. --triplet x64-linux to get the zip file with the dependencies. Extract it into the <repo root>/dep folder in such way, that the following file exists: dep/scripts/buildsystems/vcpkg.cmake.

I hope it helps.

[LecrisUT]

Of course it exists, it is just relative path to the repo root. CMake claims it is legal to put relative paths there

Ah, yes it is, but the path will be relative to the cwd. You can use ${sourceDir} to take it relative to the top-level project.

[0/7] Performing update step for 'cmakeextrautils-populate'
error: pathspec 'master' did not match any file(s) known to git

Interesting, I do not enforce master branch checking and I don't use that convention. My suspicion is that vcpkg.cmake contains a Dependency provider for vcpkg that overrides the FetchContnet logic (it should still fallthrough if it fails though). Can you share the vcpkg.cmake file?

I didn't run the CMake of the CMakeExtraUtils/test/DynamicVersion folder though.

I am using tmt and beakerlib so it's a bit tricky to run them, mostly you can follow the script there as a bash script to test manually.

EDIT: correction, in fact I did it implicitly, because the CMAKEEXTRAUTILS_TESTS variable is "ON" by default.

I only have packaging test enabled there, so you can safely disable it for now. You should not be enabling the third-party project tests, those are primarily for upstream to run.

Here're a file that will let you replicate the issue. Put it in the repo root.

Can you walk me through a bit more? I need to add that to a new project? Can you make a demo repo/gist so I can navigate through it? If you can setup a github action using lukka/run-cmake, lukka/run-vcpkg that will help a lot since I don't have a vcpkg environment and I don't know the steps to get started with that.

[adamryczkowski]

Let me start with written instructions.

1. Get into a Ubuntu (22.04) system, with current CMake and other obvious dependencies. I used https://apt.kitware.com/ to install cmake.
2. Install vcpkg following https://learn.microsoft.com/en-us/vcpkg/get_started/get-started?pivots=shell-bash#1---set-up-vcpkg
3. Start with a new repo.
4. Make the dep folder.
5. Put the CMakeLists.txt, vcpkg.json (or one with a simplier dependency. Google-cpp-cloud takes about 30 min to compile, and many GB of hard drive space) and the CMakePresets.json in the root. You may not bother about the actual C++ sources/targets, because the issue is triggered before it starts to matter.
6. Install the vcpkg. Make sure it is in the path and that the VCPKG_ROOT env variable points to the installation dir (which is a local vcpkg repo root if you followed the official installation in https://learn.microsoft.com/en-us/vcpkg/get_started/get-started?pivots=shell-bash#1---set-up-vcpkg )
7. Make a build dir, cd into it, and type cmake .. --preset vcpkg. Wait for the dependencies to build during the configure time.
8. vcpkg export --zip --output-dir=. --triplet x64-linux
9. Grab the new file and extract it back onto the dep folder in such way, that dep/scripts/buildsystems/vcpkg.cmake exists.
10. clear the build folder
11. Run cmake again with cmake .. --preset exported and observe the error.

[LecrisUT]

@adamryczkowski Can you help me a bit more to understand. Here is a dockerfile so we can have a common environment to work from:

Dockerfile

FROM fedora:latest

RUN dnf install -y \
    # Basic build system, compiler, and other tools
    git cmake ninja-build gcc-c++ neovim less yq \
    # vcpkg stuff
    vcpkg curl zip unzip tar

## Prepare vcpkg environmnet
RUN source /etc/profile.d/vcpkg.sh && git clone https://github.com/microsoft/vcpkg $VCPKG_ROOT

## Do we use a specific repo or make a new one?
#RUN git clone https://github.com/LecrisUT/CMakeExtraUtils \
#    /project

## Assuming a new clean project here
RUN mkdir /project && git init /project

WORKDIR /project

## Configure git to run git commands
RUN git config --global user.email "nobody@example.com" && \
    git config --global user.name "Example"


## Edit anything here to show what's the issue
RUN source /etc/profile.d/vcpkg.sh && \
    vcpkg new --name=test --version=0.1.0 && \
    # Add random dependency of `fmt`
    yq -iP -o json '. += {"dependencies": ["fmt"]}' vcpkg.json

COPY <<EOF CMakeLists.txt
cmake_minimum_required(VERSION 3.25...3.30)

include(FetchContent)

FetchContent_Declare(CMakeExtraUtils
        GIT_REPOSITORY https://github.com/LecrisUT/CMakeExtraUtils
        GIT_TAG v0.4.1)
FetchContent_MakeAvailable(CMakeExtraUtils)

include(DynamicVersion)
dynamic_version(PROJECT_PREFIX MyPOC_
        FALLBACK_VERSION 0.1.0
)

project(MyPOC VERSION ${PROJECT_VERSION} LANGUAGES CXX)
EOF

COPY <<"EOF" CMakePresets.json
{
  "version": 7,
  "configurePresets": [
    {
      "name": "vcpkg",
      "binaryDir": "cmake-build-vcpkg",
      "generator": "Ninja",
      "cacheVariables": {
        "CMAKE_TOOLCHAIN_FILE": "$env{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake",
        "VCPKG_MANIFEST_MODE": true
      }
    },
    {
      "name": "exported",
      "binaryDir": "cmake-build-exported",
      "generator": "Ninja",
      "cacheVariables": {
        "CMAKE_TOOLCHAIN_FILE": "dep/scripts/buildsystems/vcpkg.cmake",
        "VCPKG_MANIFEST_MODE": false
      }
    }
  ]
}
EOF

## The following should be added to work properly for production,
## but for now we can use FALLBACK_VERSION to ignore issues?
#COPY <<EOF .git_archival.txt
#node: $Format:%H$
#node-date: $Format:%cI$
#describe-name: $Format:%(describe:tags=true,match=?[0-9.]*)$
#EOF
#COPY <<EOF .gitattributes
#.git_archival.txt  export-subst
#EOF
#RUN git add * && \
#    git commit -m "Initial commit" && \
#    git tag v0.1.0

## Try to run outside of vcpkg environmnet
RUN cmake -B cmake-build-local && \
    cmake --build cmake-build-local

## Now try to build with vcpkg
RUN source /etc/profile.d/vcpkg.sh && \
    cmake --preset vcpkg && \
    cmake --build cmake-build-vcpkg

## What next, what to do to get the `dep` folder?

If you're not familiar here is how to build and then get a shell environment inside it (podman and docker should be equivalent here)

$ podman build -t cmake-extra-utils:issue-39 -f Dockerfile
$ podman run -ti cmake-extra-utils:issue-39

Try to make a reproducible from there and add on top of the dockerfile there. I am still a bit lost for the dep folder part

---

PS: when using presets, do not create a build directory because that is handled by binaryDir. It would otherwise make build/test presets unusable. cmake .. --preset vcpkg is equivalent with cmake -S .. -B . --preset vcpkg but that is very fragile.