Can we add stakers to this network? New validators were not discovered

[john-risk-adigium]

When network was up using the preset validators and stable and producing blocks, I added a new validator and tried to add it to the network. However, it was not detected as a new validator to stake.

• I run the network and made sure it is working with 2 geth nodes, 2 beacon nodes, and 1 validator node connected with the first beacon node.
• I created another validator key using the deposit tool deposit new-mnemonic starting from index 0
• I called my webmethod that makes the deposit, passing it my filled geth account public and private keys, and the deposit key generated by the deposit tool

echo {\"keys\":$(cat `ls -rt /root/validator_keys1/deposit_data* | tail -n 1`), \"address\":\"0xF359C69a1738F74C044b4d3c2dEd36c576A34d9f\", \"privateKey\": \"0x28fb2da825b6ad656a8301783032ef05052a2899a81371c46ae98965a6ecbbaf\"} > /root/validator_keys1/payload.txt
curl -X POST -H "Content-Type: application/json" -d @/root/validator_keys1/payload.txt http://localhost:8005/api/account/stake

• My web method calls the deposit contract's deposit method

export const deposit = async(depositKeys, account, pvtKey) => {
    const provider = new ethers.providers.JsonRpcProvider(process.env.RPC_URL); // connect to the network

	// Getting the contract
    const signer = new ethers.Wallet(pvtKey, provider); // create a wallet with the provided private key
    const contract = new ethers.Contract(process.env.DEPOSIT_CONTRACT_ADDRESS, contractAbi, signer); // get the deposit contract

    for (let key of depositKeys) {
        const { pubkey, withdrawal_credentials, signature, deposit_data_root } = key; // get the relevant data from the raw depositData

        // call the deposit function on the contract (and prepend 0x to the deposit data)
        const tx = await contract.deposit(prefix0X(pubkey), prefix0X(withdrawal_credentials), prefix0X(signature), prefix0X(deposit_data_root),
            {
                chainId: parseInt(process.env.CHAIN_ID), // add the chain ID to the transaction
                value: ethers.utils.parseEther('32') // the transaction value
            }
        );

        const w = await tx.wait();

        // stop on failure
        if (w.status == 0) throw new Error('TRANSACTION FAILURE: ' + JSON.stringify(w));

        console.log('>>> TX', tx);
        console.log('>>> W', w);
    }
}

• The deposit function returned json with the transaction info and status=1
• The deposit transaction was logged in geth to my contract address 0x4242...
  Submitted transaction hash=0x944ce52c06d9518c436847dcda2bd144552daf96257b132afe16e188e824af6e from=0xF359C69a1738F74C044b4d3c2dEd36c576A34d9f nonce=0 recipient=0x4242424242424242424242424242424242424242 value=32,000,000,000,000,000,000
• Started a new validator node connecting to the second beacon node:

clients/lodestar validator \
	  --dataDir "./data/consensus/1" \
	  --beaconNodes "http://127.0.0.1:9597" \
	  --suggestedFeeRecipient "0xF359C69a1738F74C044b4d3c2dEd36c576A34d9f" \
	  --graffiti "YOLO MERGEDNET GETH LODESTAR" \
	  --paramsFile "./consensus/config.yaml" \
	  --importKeystores "/root/validator_keys1" \
	  --importKeystoresPassword "/root/validator_keys1/password.txt" \
	  --logLevel debug

• The new validator is not discovered (Count=0)

Lodestar network=mergednet, version=v1.2.1/e76d4c9, commit=e76d4c9cda242f5f3465f5168d8b36a2c3031d53
Connecting to LevelDB database path=data/consensus/1/validator-db
1 local keystores
0xa6f3a9bc85181812cd003d0cc476c591e7ef5f278f5ca5450d3a07e1f32463ac29e92caaeb5c17ade8f56f9836031f9c
Genesis fetched from the beacon node
Verified connected beacon node and validator have same the config
Persisted genesisValidatorRoot 0xe63460dc044e056f26ca8f7406a18867d31f1ec195322f428b5918d4b0153050
Persisted genesisTime 1672854611
Verified connected beacon node and validator have the same genesisValidatorRoot
Discovered new validators count=0

All the logs are here

Is there something I am doing wrong?

• We tried setting ETH1_FOLLOW_DISTANCE: 5
• We also tried setting SLOTS_PER_EPOCH: 6 and EPOCHS_PER_ETH1_VOTING_PERIOD: 2, but these values were reverted back to the large values from mainnet when we queried http://127.0.0.1:9596/eth/v1/config/spec
• curl http://127.0.0.1:9596/eth/v1/beacon/states/head/validators shows 8 validators, we hoped it would be 9 as we added our validator.
• we waited half a day and still the same results

[q9f]

Everything looks good. After the deposit it takes 8-12 hours usually to activate the validator. Maybe wait a bit longer?

There is one thing:

Discovered new validators count=0

It looks like your new validator does not correctly load the keys. Can you check that?

[john-risk-adigium]

I generated many validator_key groups.
A group I put in the consensus/mnemonic.yaml file is discovered when I run the validator.
But a group that I don't put in mnemonic.yaml is not discovered.

What else can I do?

• Should we create a wallet for these validators? I imported them with the same commands without adding a wallet or menmonic.
• Would the GENESIS_FORK_VERSION need update? we selected mainnet while running the deposit tool. Should we change it to 0 to match the mainnet configuration?