I have had the same issue with a 200kv racestar 5045 motor.
I have done a ton of research and I finally got my sensors to work. I though the sensors were bad and considered replacing them all together but I found the solution before it came to that.
What it came down to was the correct wiring of the sensors to the vesc. The sensors were perfectly fine and had no issues.
The black and red wires from the sensor turned out to follow convention (red = positive, black = ground) even though the wiring diagram online says otherwise.
I swapped the wires around until I read the following from my multimeter. The ground of the multimeter was placed on the ground pin for the hall sensor.
5v pin should be approx. 5v
temp pin is skipped
hall 3 pin was approx. 0v
hall 2 and 1 pin were approx. 3v
You will know everything is wired correctly when one hall sensor is zero volts and the other two are around 3v. When I received UNKNOWN HALL ERROR: 255, each hall sensor pin was reading ~4.3 volts which is incorrect.
If you don’t have a multimeter, go pick one up. They are fairly cheap, and will help you resolve this issue.
Running the motors in sensored mode is well worth the extra effort. No more cogging, and I have excellent start up torque.
Edit: Just swap the red and black wires around and the other hall sensor wires that ARE NOT white. The white wire (middle wire) is almost certainly hall 2. Swap the wires around until you get the correct reading with the multimeter.