They bathe outdoors even in the winter.
No worries — this doesn’t lead to hypothermia. There are many geothermal areas in Iceland and swimming in a steaming river or a pool can be just as nice as taking a hot bath. In fact, it might be especially enjoyable in the winter when you get an opportunity to see the northern lights and the stars. The hardest temptation to resist is not to add bath foam into the hot water!