Why didn't Smiley arrest Polyakov?

Polyakov is a diplomat, with a role as a cultural attaché. As such he has diplomatic immunity.

Arresting a diplomat is an incredible breach of international norms and would put your own diplomats worldwide at risk. Article 29 of the Vienna Convention on Diplomatic Relations states:

The person of a diplomatic agent shall be inviolable. He shall not be liable to any form of arrest or detention. The receiving State shall treat him with due respect and shall take all appropriate steps to prevent any attack on his person, freedom or dignity.

Theoretically the UK could order Moscow to withdraw Polyakov from the country, but even that is a sanction reserved for very serious breakdown in relations between countries.

Expelling diplomats does happen, but it's a decision made at a political level in a country, not something that an agent like Smiley can do.

It's known that some diplomats are spies for their country, and the way you deal with that is by having counter espionage operations against them, which is what we are seeing in the movie.