Google will only show the correct language version of your site if each language has its own unique URL. Without separate URLs, Google cannot index the English and Swedish versions independently, so it cannot show different versions to users in different countries.
You don’t need to make big changes. Your URLs just need to be unique in some way, for example by adding something small such as a language folder or a language parameter. This allows Google to understand that one page is Swedish and the other is English.
Once you have unique URLs, you can add hreflang tags. These tell Google that the Swedish page is meant for users in Sweden and the English page is meant for everyone else. After that, Google will show the Swedish result on google.se and the English result elsewhere.
If you want proper multilingual SEO setup without breaking anything, Tech Savy Crew can help you get it right.