The Omnia has awesome support for many file formats. However, it may have trouble playing high-resolution videos without some noticeable lag.
Badak is a free application for Windows that allows you to convert a wide variety of videos to a format supported by the Omnia. I started using it to convert videos for my iriver clix II player, which only supports 320x240 videos. Badak's flexibility allows one to define the exact output format. Conversion speed is among the fastest of the free encoders I've tried.
Badak uses a slightly older version of mencoder. You can easily download a newer version and overwrite the one in Badak's installation folder, set to C:\Badak by default.
Here are some recommended settings:
Video bitrate: 500
Resize : 400 x 240