How to perform opposite operation of FFmpeg’s “hevc_mp4toannexb” filter?

FFmpeg has a bitstream filter called hevc_mp4toannexb to extract the annex-b formatted video from any container like MP4, MKV, etc.

ffmpeg -i INPUT.mkv -codec copy -bsf:v hevc_mp4toannexb OUTPUT.hevc

But I couldn’t find any filter or method in the docs to do the opposite of it, that is, mux an existing annex-b formatted video into an MKV container.

I was hoping something like this would work, but it results in an error:

ffmpeg -f hevc -i INPUT.hevc -codec copy OUTPUT.mkv

Result:

[matroska @ 0xa56c40a00] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[matroska @ 0xa56c40a00] Can't write packet with unknown timestamp

Adding -fflags +genpts before the input also results in the same error.

ffmpeg -fflags +genpts -f hevc -i INPUT.hevc -codec copy OUTPUT.mkv

Any idea on how to do this?