There are so many factors, that it's hard to know where to begin.
My advice would be to make changes based on data (not hunches), and to stick to Google's webmaster guidelines (a lot of people don't, and then wonder why Google won't throw more search traffic their way).
You hit the nail on the head with one very important metric though: Bounce rate.
This is an indication that the content on your website doesn't draw people in enough, and so they are navigating away from your site once they arrive.
Some websites will naturally have a high bounce rate, because their purpose is to serve out quick information with little engagement. Take a dictionary, for example.
But if your aim is to encourage users to explore your site and navigate on to a second or third pageview, then there are three key things you can do:
- Earn people's trust. A site heavy in ads or with a poor design will spark security concerns in many web users' minds. First impressions count, which is why it is worth spending some time considering your design.
- Have attractive call to actions. Make relevant and interesting pages easy to find, and highlight them in a way that is clickable.
- Focus on content. If you provide relevant, well-written and useful information on your site, then users will stick around - and Google will reward you with more traffic.
Sure, there are little tricks you can do to work your way up search ranking, but once other indicators such as bounce rate drop - this gives Google's algorithm cause for concern, and your ranking will fall accordingly.