So I was testing out my remote today and had what felt like a few loss of connection (no acceleration, no braking). I began debugging the connection and found that it worked perfectly. The problem wasn't the transmitting part, but the trigger button. It seems that the tactile button I am using has a "dead-zone" after it "clicks".
I will have to make some adjustments to the trigger part, so the button doesn't release accidentally.