One in seven homeowners aged 70 and over are still paying off their mortgage, with the average debt a hefty £40,000.
New figures from the Saga Equity Release Advice Service also show that one in three over 50s still have a mortgage.
They need to find £50,000 on average to own their home outright and be mortgage free.
Shopping around for a new mortgage could help people pay it off quicker but around one in three over 50s say they have never tried to remortgage.
Many haven’t shopped around because they assume they won’t be able to get one because of their age.
One in 10 over 50s are concerned about their lender’s maximum borrowing age.
They may be right to be worried as one in 14 saying they have been prevented from moving their mortgage to a more competitive deal because of their age.
While those in their early 50s still have years of working life to chip away at their debt, the over-70s are having to use their weekly pension to service the debt.
Alex Edmans, head of retirement at Saga Personal Finance, said: “Millions of older homeowners have found themselves abandoned by mortgage lenders and stuck in uncompetitive deals because of the unfair age restrictions that many lenders have in place.
“If these people had access to a better deal they wouldn’t have to pay as much back each month which would leave them with more money to enjoy their retirement.
“For those in retirement struggling to meet their monthly mortgage costs it may be worth considering a lifetime mortgage to help ease the burden of the monthly repayments.
“This may not be suitable for all, so it is well worth speaking with a specialist adviser, who would consider all alternatives and review whether any state benefits could help provide some relief.”