To handle a list of records from pagination-based Rest-API, we can implement with Standard Connectors in the MuleSoft application.
Introduction:
Normally, we can collect records from an external system and insert them into our target system using many approaches but if the system stored their records inside pages, then we need to go for an alternative approach.
- Why do we need to go for VM listener and queue approach?
Some approaches such as recursive flow reference may affect application performance and stability.
- What happens if we go with VM listener and its queue approach?
VM listener has its own queue to store the message; once it processed the records, it will clear all the memory and run as a fresh one. So, in our pagination, we can clear the memory for each page.
How does this application work?
First, it publishes page properties–such as current page and last page into VM queue ex: page_property_queue; then, the queue will consume the page property.
So, now the queue has the current page and last page value based on this current page number it passes to external system API. Paginated Rest API response contains records; so, it sends a list of records into the batch job.
The batch job will process records. In the aggregator component, it creates Salesforce records. In the On Complete phase, it will pass the next page properties into the queue, and this operation continues until the last page.
Finally, page iteration and creation of record part completed without resource and performance stability issues.
Considerations:
- Salesforce callout per iteration problem may occur. To handle this, Mule applications have a temp folder and database connection options. First, collect all records until all pages are over and then store data into a temp folder or database. Batch job source data can be collected from the temp folder or database table.
- Use reusable data weave code and configuration property for the application; so, data future requests will be handled better way.
Application Design screenshot:
- Scheduler main flow:
- Pagination handler sub flow:
Mule Application XML Code:
<?xml version=”1.0″ encoding=”UTF-8″?>
<mule xmlns:salesforce=”http://www.mulesoft.org/schema/mule/salesforce”
xmlns:batch=”http://www.mulesoft.org/schema/mule/batch”
xmlns:http=”http://www.mulesoft.org/schema/mule/http”Â
xmlns:ee=”http://www.mulesoft.org/schema/mule/ee/core”Â
xmlns:vm=”http://www.mulesoft.org/schema/mule/vm”
xmlns=”http://www.mulesoft.org/schema/mule/core” xmlns:doc=”http://www.mulesoft.org/
schema/mule/documentation” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”Â
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/vm http://www.mulesoft.org/schema/mule/vm/current/mule-vm.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd
http://www.mulesoft.org/schema/mule/salesforce http://www.mulesoft.org/schema/mule/salesforce/current/mule-salesforce.xsd”>
<vm:config name=”VM_Config” doc:name=”VM Config” doc:id=”577b072a-5fc1-4239-bc6d-e918e049eabf” >
<vm:connection />
<vm:queues >
<vm:queue queueName=”pagination_parameter_queue” queueType=”PERSISTENT” />
</vm:queues>
</vm:config>
<http:request-config name=”HTTP_Request_configuration” doc:name=”HTTP Request configuration” doc:id=”1175ced3-1a0e-473f-bd1d-92b6fed31f42″ >
<http:request-connection host=”${host}” port=”${port}” />
</http:request-config>
<salesforce:sfdc-config name=”Salesforce_Config” doc:name=”Salesforce Config” doc:id=”4d5ab0e8-66b4-4820-8136-9330e7e431d7″ >
<salesforce:basic-connection username=”{username}” password=”{password}” securityToken=”{securitytoken}” url=”{url}” />
</salesforce:sfdc-config>
<flow name=”pagination_rest_api_mule_appFlow” doc:id=”2f75bdc8-3e10-4bed-a60d-fa662dcd48fb” >
<scheduler doc:name=”Schedule app to run specified time” doc:id=”37e2aa60-8289-464c-96e6-61b4dee2c2af” >
<scheduling-strategy >
<fixed-frequency frequency=”1″ timeUnit=”DAYS”/>
</scheduling-strategy>
</scheduler>
<logger level=”INFO” doc:name=”debug log before request to rest api” doc:id=”ee98c188-f028-4ad3-968a-5cecc32aa430″ message=”application started its execution”/>
<set-variable value=”#[{currentpage:0,lastPage:1}]” doc:name=”page_properties currentpage=>0 and lastPage=>1″ doc:id=”d905dee5-ced1-4cc4-be2e-9cdbd05a35ed” variableName=”page_properties”/>
<vm:publish-consume queueName=”pagination_parameter_queue” doc:name=”publish page properties to the page handler listener” doc:id=”3212c473-2adc-430e-bcc7-b8d317cda3bf” config-ref=”VM_Config” timeout=”1″ timeoutUnit=”HOURS”/>
</flow>
<flow name=”RequestperPage_subflow” doc:id=”1cddc50a-eb49-48d9-b037-559c93a75d58″>
<vm:listener queueName=”pagination_parameter_queue” doc:name=”Page handling listener” doc:id=”e1190c3f-886f-40ac-b2d5-44df627bb1e6″ config-ref=”VM_Config” timeout=”1″ timeoutUnit=”HOURS”>
<vm:response>
<vm:content><![CDATA[#[{
“currentPage”:payload.”currentPage”,
“lastPage”:payload.”lastPage”
}]]]></vm:content>
</vm:response>
</vm:listener>
<ee:transform doc:name=”page properties current page and last page” doc:id=”0bbf99b6-8a96-41d8-a4cc-b81127d25cc2″>
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/json
—
{
“currentPage”:payload.”currentPage”,
“lastPage”:payload.”lastPage”
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<choice doc:name=”Choice to route until last page” doc:id=”b5b8d044-3168-4f2f-ab7a-a6865ae5fca0″ >
<when expression=”#[payload.lastPage >= payload.currentPage]”>
<logger level=”INFO” doc:name=”current page not yet reached last page” doc:id=”ac325cec-5b34-4ddc-be8f-729458fd9c78″ />
<http:request method=”GET” doc:name=”send request to rest api with page no” doc:id=”c64f326a-5ec9-4359-8abc-19fa98a5693a” config-ref=”HTTP_Request_configuration” url=”url”>
<http:query-params ><![CDATA[#[output application/java
—
{
page : payload.currentpage
}]]]></http:query-params>
</http:request>
<set-variable value=”#[{lastpage:payload.lastpage,currentpage:payload.currentpage}]” doc:name=”update page property variable” doc:id=”054094eb-b867-45d4-a842-bcffdc7d5cc9″ variableName=”page_property” />
<ee:transform doc:name=”json response contains record and page properties” doc:id=”95f91d72-f4f3-4c73-a10f-3a81683698f7″ >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
—
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<set-payload value=”#[payload]” doc:name=”pass list of records into batch job” doc:id=”a403c24c-cc42-4566-bb24-a312eafbe04d” />
<batch:job jobName=”pagination_rest_api_salesforce_record_batch_job” doc:id=”ae3a563a-ff25-4c22-84af-6aa889ef75b2″ maxFailedRecords=”-1″>
<batch:process-records>
<batch:step name=”Batch_Step_to_handle_each_record” doc:id=”4d457eab-a0d9-4c0e-811e-b5e0c33cb7c4″>
<ee:transform doc:name=”Transform record mapping to salesforce fields” doc:id=”aedd6612-e83f-40fa-8d47-a176294f6421″>
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/java
—
{
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<batch:aggregator doc:name=”Batch Aggregator to collect list of records” doc:id=”a7edcffc-7a85-4c28-ad54-5a13c48f1bc1″ size=”150″>
<ee:transform doc:name=”record list to create records in salesforce” doc:id=”4450d4fb-6847-468e-993d-0717ef96a9a2″>
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/java
—
{
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<salesforce:create type=”Account” doc:name=”insert records into salesforce ” doc:id=”75616818-478c-4e78-b3aa-fcabb44c00fc” config-ref=”Salesforce_Config” />
</batch:aggregator>
</batch:step>
</batch:process-records>
<batch:on-complete>
<logger level=”INFO” doc:name=”print log once batch excecuted” doc:id=”57d68d47-4655-4c10-86bd-a4c44823014c” />
<vm:publish-consume queueName=”pagination_parameter_queue” doc:name=”for next iteration send page properties to page handler queue” doc:id=”deedefdf-89ab-458d-98d9-f087cdc6370d” config-ref=”VM_Config”>
<vm:content ><![CDATA[#[{currentpage:vars.page_property.currentpage,lastpage:vars.page_property.lastpage}]]]></vm:content>
</vm:publish-consume>
</batch:on-complete>
</batch:job>
</when>
<when expression=”#[payload.currentPage > payload.lastPage]”>
<logger level=”INFO” doc:name=”current page reached last page” doc:id=”37808230-6132-4d15-89bb-f2cc3b4d859a” />
</when>
<otherwise >
<logger level=”INFO” doc:name=”default log” doc:id=”5db8092b-61cf-48e1-b773-6b5b10a6d14d” message=”default page handler logger”/>
</otherwise>
</choice>
<logger level=”INFO” doc:name=”listener action executing logger” doc:id=”91d730d0-b9d8-4dd9-9d6e-fad1f2e80772″ message=”listener asynchronous action execution is inprogress”/>
<error-handler ref=”appError_Handler”/>
</flow>
</mule>
