Hmm. Both will work. However, to me the second form (with the infinitive) seems to suggest the motion of
off.
We started off trying the widget in the thingamajig = our first attempt was to try the widget in the thingamajig.
We started off to try the widget in the thingamajig = we left the area in order to go and try the widget in the thingamajig.
There may be other opinions.