To delete a song from your library:

1. Enter song selection mode
2. Select the song(s) you want to delete
3. Press the delete button

As demonstrated in this animation:

To delete a set list, the process is exactly the same, except you enter selection mode in your set list library.

Alternatively, if you want to delete a single song or set list you can swipe it to the left, and press the delete button.