It doesn’t have any negative effect on your site. When you migrate there is an option to export seo data from yoast to RankMath and you are good to go.
I have done this. So I can say it with confidence that you can migrate to RankMath without any worries.