BUY BOOKS IN ADVANCE!!! Whatever you do, don't buy them from the bookstores. They will jack up the price to a ridiculous amount.
When buying books, I'd suggest this site: Slugbooks
Just type in the ISBN number on the book that you need and it will search for the book on a bunch of different databases and find you the cheapest price. It's saved me so much money!
When it comes to selling back your books, I'd suggest selling them on Ebay, Amazon, etc. The school bookstore will give you practically nothing for them. I can get all of my money back from selling them on Ebay over the summer.
Want to rent books? This is probably the best option for big textbooks, but at the same time, you aren't going to get ANY money back when you return them. I even buy my bigger textbooks instead of renting them so that I at least get SOME money back at the end of the year.
It's really about preference. Rent or buy your books, whichever you prefer, but DON'T fall for the scam of buying books from the bookstore. It's such a waste of money. If the school tells you that you HAVE to get them new from the bookstore, contact your prof and see if you will even use the code. 90% of the time, they don't even use the codes, and if they do, you can usually buy them from the book publisher company and buy the book separately.