Why do most countries dislike borrowing from the International Monetary Fund (IMF)?

They say it will badly affect the public sector eg loss of jobs etc. Why is this, and why do they tend to dislike the International Monetary Fund (IMF) specifically and only use it as a last resort