79439962

Date: 2025-02-14 16:21:41
Score: 0.5
Natty:
Report link

Assuming this type of pagination object in your payload:

pagination info in your payload

I implemented it using a foreach that iterates through a paging array. Here is the general structure:

example pipeline

Let's discuss each activity:

  1. get oauth token - retrieve the bearer token required for API authentication.

  2. first call for paging info - call the api specifically to retrieve the first pagination object.

  3. create paging array - for the Name field: create a Pipeline variable of type Array called paging_array. For the Value field: use the @range function to build the array, representing the number of calls that need to be made to the API based on the limit and total number of records to be retrieved:

@range(
    1,
    add(
        div(activity('first call for paging info').output.pagination.total, activity('first call for paging info').output.pagination.limit), 
        if(
            equals(mod(activity('first call for paging info').output.pagination.total, activity('first call for paging info').output.pagination.limit), 0),
            0,
            1
        )
    )
)
  1. foreach paging array item - the Items field simply references the paging_array created earlier. Consider setting a Batch count if your API has throttling requirements.

foreach activity Items

  1. call using item from paging array (Web activity within Foreach) - I've used a Web activity here as an example, but you may perhaps use a Copy Activity to integrate into your data warehouse or move to a storage account, etc. The important thing is that the source URL specifies the offset using the current Foreach item:
https://your.excellent.api/v1/users?limit=100,offset=@{item()} 

Watch your batch count so your API is not overwhelmed, especially in the scenario where your doing an initial full load and then subsequent deltas...

Thank you!

Douglas

Reasons:
  • Blacklisted phrase (0.5): Thank you
  • Long answer (-1):
  • Has code block (-0.5):
  • User mentioned (1): @range
  • Low reputation (0.5):
Posted by: Douglas Wiley