Only 20% of rail passengers are still wearing masks at train stations in England now that it is no longer compulsory, according to Network Rail.
That compares with 80% before restrictions were lifted on 19 July.
On buses and Tubes, Transport for London has retained mask-wearing as a condition of carriage in the capital.
There is no legal requirement to wear one on public transport in England, although it remains the case in Scotland, Wales and Northern Ireland.
Several unions that represent transport workers say they are concerned that there has been a drop in mask-wearing, even on TfL services.
The unions fear it would be difficult to reintroduce mask-wearing if the government decided to do so.
A government statement said: "The guidance is clear that people are expected and recommended to wear a mask when they come into contact with people they don't normally meet in enclosed and crowded spaces."
"It is open to transport operators to decide if they want to implement their own policies, working within their particular environment."
'Serious issues over enforcement'
Mike Lynch, general secretary of the RMT union, said his members had noticed a marked policy drift since the government changed the rules.
"Even on London Transport, where it is supposed to be a condition of carriage, just like paying a fare, the policy is coming apart at the seams and as more people see others failing to comply, the situation will escalate quickly over the autumn," he said.
Transport for London currently has more than 500 uniformed officers who are undertaking compliance activity on networks across London and non-exempt customers who fail to comply may be refused travel.
Siwan Hayward, director of policing and compliance for TfL, said: "Covid is still with us and we all have a role to play in doing the right thing and keeping each other safe."
Mr Lynch explained that there is a "serious issue over enforcement", meaning workers were "potentially put into the front line" and were at risk of abuse and violence from people seeking confrontation.
"With the government already making compulsory mask-wearing on transport a contingency if cases escalate in the coming weeks, there is a real danger they won't be able to get the genie back into the bottle."
Mick Whelan, general secretary of train drivers' union Aslef, said his members had also noticed a change.
"[Mask-wearing] is mandatory - but not enforced - on TfL and we have noticed a decline, in recent days, in the number of passengers wearing masks," said Mr Whelan.
"We believe mask-wearing should be mandatory on trains - to protect passengers and staff and to control the Covid-19 pandemic."
A spokesperson for Unite, which represents bus drivers, said: "Our members are reporting that mask-wearing on buses is collapsing and has got much worse in recent weeks.
"In some cases, there are very few passengers wearing masks on a bus, with those who tend to still wear a mask being older.
"For our members, while they are relatively safe when actually driving the bus because of the sealed cabs, there is growing concern that they are being placed in danger when they are required to board a bus at a depot to be taken to a rendezvous point to collect the bus they are allocated to drive.
"Due to buses getting busier, they can't socially distance and feel their health is being placed at risk by non-mask-wearers."
However, a spokesperson for the Rail Delivery Group said most people are "doing the right thing" and wearing face coverings, especially when carriages are busy.
"Travelling by train is as safe, if not safer, than other indoor activities, with ventilation that refreshes the air in carriages every six to nine minutes," a spokesperson added.
"In addition to expecting people to wear face coverings in crowded spaces, train operators are continuing with extra cleaning and we've improved information about the busier and quieter times to travel. This all means that people can continue to travel with confidence."