Labuan town is very small and crowded, in fact, to get a cab in this city is easy, but if you first came to this island, you might get a little difficult to get a taxi. Most of the island's taxi company is privately owned but registered them with the taxi associations Labuan and have a valid permit.
You suggested just taking a taxi have a valid permit. Where taxi terminal in Labuan? Through my observations, you can get a taxi service in these areas.
Here is a taxi terminal area, in front of the building. If you see this place, go to the ticket counter and tell them where you are going and make payment and ticketing staff will show you a taxi for a ride.