You can define a sysparm_query parameter for the normal Table REST API which will add the "Dynamic" search query.
Build whatever query you want on the Table REST API request in a list view - I just pulled up a list and selected "Assignment Group is (dynamic) one of my groups" but you can use whatever query you want:
You can then right-click on the filter breadcrumbs and click "Copy Query" to grab the encoded query:
Mine came up as "assignment_groupDYNAMICd6435e965f510100a9ad2572f2b47744". I believe that the dynamic query SysID is common to all instances, so that query string *should* work in almost any instance as a REST query as below, however if it doesn't work, you should be able to copy the encoded query using the method above.
Here's the "Assigned to me" REST API call that returns incidents where I'm in the assignment group:
https://<INSTANCENAME>.service-now.com/api/now/table/incident?sysparm_query=assignment_groupDYNAMICd6435e965f510100a9ad2572f2b47744
You will obviously also need to pass Authentication for the user that you're interested in, and that user will need to have READ ACL access to the table you're querying.
