Thanks to everyone who helped me figure out this issue. Special thanks to @Cuong Phan who helped put me on the right track.

The solution I ended up using was this:

I modified the Service Portal Header and included the following:

Server script

data.lang = gs.getSession().getLanguage(); //returns the current user's language

Client contoller

var lang = $scope.data.lang;

if (lang != 'en') {
	document.body.style.direction = 'rtl';
}

This modifies the CSS using DOM manipulation to change the direction to right-to-left.

You can put this in a widget too and place the widget on the pages individually, but I felt it was easier to place it in the header as that way, it'll be automatically applied to every page on that portal.