Hanging connection in handshake using large certificates

[javibc3]

I am doing experiments with post-quantum cryptography in MsQuic.

I managed to integrate the OpenQuantumSafe provider into OpenSSL. Starting with the tests, initially it was going well testing key exchange algorithms and some signature algorithms.

But in signature algorithms that create very large certificates (+16KB) I realized that the server was hanging performing the handshake. The server recieves the Client Hello message and tries to respond, but it never sends the packets in the network and gets hangs up.

I leave here the logs of the server and the client.
server.log

client.log

[nibanks]

If you look at crypto.c, I believe we cap crypto payload to 32k:

    Status =
        QuicRecvBufferInitialize(
            &Crypto->RecvBuffer,
            InitialRecvBufferLength,
            QUIC_DEFAULT_STREAM_FC_WINDOW_SIZE / 2,  <==== MAX buffer size allowed
            QUIC_RECV_BUF_MODE_SINGLE,
            &Connection->Worker->AppBufferChunkPool,
            NULL);

I wonder if you're hitting that (I haven't looked at the logs).

[javibc3]

Yes, it was the buffer size. I also changed the send buffer macros QUIC_MAX_TLS_CLIENT_SEND_BUFFER and QUIC_MAX_TLS_CLIENT_SEND_BUFFER to 64KB and now it runs perfectly.

Thanks for the help!

[nibanks]

Ok. Great. So, what is a reasonable maximum buffer size to use in this new PQ world? Please also consider that the bigger this buffer, the bigger the attack surface here. This buffer handles reordering received data, so they can send 1 byte of TLS payload at the last offset and we will be forced to allocate it all and wait.

[javibc3]

Im not really sure what a reasonable maximun buffer size could be, but i can set you in context.

With the current configuration in MsQuic, ML-DSA runs perfectly on all three security levels, ML-DSA44, ML-DSA65 and ML-DSA86. With ML-DSA the certificates sizes ranges from 4 to 10 KB and signature ranging between 2.5 to 4.5KB.

The problem starts with the other NIST signature standard, SLH-DSA. Here we are talking of certificates sizes between 10KB all the way up to 70KB and signature sizes between 8KB and 50KB.

But in reality, i don't think no one is taking into consideration using SLH-DSA in TLS really. Besides the enormous certificates sizes, it also performs really poorly. I thinks it will be used in things like code signing or other offline signature, but it keeps being a NIST standard after all.