The delivery fee covers the driver's time, truck operation, insurance, and fixed costs that apply even for a short trip. The rental fee is just for the physical unit; the delivery fee is for the service of bringing it to you and taking it away.
A single delivery requires the driver to complete a round trip. The driver loads the unit at the yard, drives to your site, unloads and positions it, then returns later to pick it up. That driver is paid for their time on the clock. The truck burns fuel and requires maintenance. The company carries insurance for the vehicle and the driver. These are fixed costs per job, regardless of whether they are delivering one unit or ten.
Distance does factor in, but for a rural field, the mileage is likely higher. Even with good truck access, the time commitment and vehicle wear are the same. For a one-day event, the rental period is short, so the delivery fee becomes a large portion of the total because the fixed costs of the two trips are spread over only one day of rental revenue. A longer rental would make the delivery fee a smaller percentage of the total cost.
The weekend rate for the standard portable toilet itself is often a promotional price to be competitive, which can make the delivery fee appear large by comparison.