How to navigate history using arrow keys with zsh-autocomplete

I’m using marlonrichert’s zsh-autocomplete on top of oh-my-zsh my mac and I’m running into the following issue:

Whenever I press the app arrow, only the last command is loaded and up/down arrows don’t do anything.

I saw on the keybindings section, that the table has a main column that showcases some commands, besides the emacs and vicmd layouts. Only that column mentions actions about the up/down arrow keys, but I could not remap those at all. I tried adding bindkey commands for those keys at any point in my .zshrc file but with no effect

bindkey '^[[A' up-line-or-search                                                
bindkey '^[[B' down-line-or-search

For example, the above tries to bind the up arrow to an action that can be bound to any other combination of keys, but it’s just the up/down arrows that misbehave.

Running the following command confirms the keybind

-> % bindkey | grep up-line-or-search
"^P" up-line-or-search
"^[[A" up-line-or-search

Ctrl+P works as intended, but the up arrow key does not.

Is there a way to navigate history using the up/down arrow keys alone with zsh-autocomplete?

Note that using iterm or not does not have bearing on this behavior.