Very often, poverty stays in families. This might have something to do with the fact that poor people often need to quit school in order to get a job so they can eat and have a roof over their heads, so they can't get a good paying job because they lack a good education. Also, poor people tend to live in districts with poor schools (as much of a school's income is typically tied to local property taxes, and not surprisingly, this means that there is much less funding for poor areas), even if they don't have to quit school to get a job. Additionally, even if they make it through high school, mommy and daddy are not going to be able to pay for college (obviously, because they are poor), and, because they went to a bad high school, they are less likely to be able to get a scholarship to a college, so their chances of going to college will be small.
If we look at the other end of the spectrum, people who have rich parents can afford to go to the best schools, and do not generally need to drop out of school in order to eat. They also can afford to hire tutors if they are a bit slow, so they are likely to be able to pass, even if they are not the best students. This means that they can get high paying jobs, if they want them. Of course, if one has money, one invests the money, and then one can work much less, or not at all.
Simply put, people do not start at the same position, so it is hardly surprising if they do not end up with the same results.
It is as if we were to have people run a race of a mile, with some people starting half-way to the finish line, some at 3/4 of the way there, and some who have to run the whole way. The ones who start 3/4 of the way have the best running shoes. Who do you think will win the race? Unless the one at the beginning is exceptional, while the others are very bad runners, there will simply be no contest.
Also, you seem to think that someone who is poor should work 24/7, and have no leisure at all. People do not work efficiently that way; everyone needs breaks, and consequently, even poor people may want to watch a little TV upon occasion, without it being a sign of laziness.