Yeh, the best that you can do is not to display them. All the site wide and per post options control whether the pingback/trackback will be stored in the DB, but the piece of code handling pingbacks from reception till checking if to put them in DB will always be executed.
This is probably not a big thing as I don’t remember exploits utilizing pingbacks, but it annoys me esthetically that an option described “Allow link notifications from other blogs (pingbacks and trackbacks.) ” doesn’t do what can be interpreted from the description (what it does is to set the default of the ping option on the post edit page).