Secure Payments Pagination of API Transactions

General

LoanPro utilizes AWS Elaticsearch to power up our API search options and results. One of the constraints of Elasticsearch is that it will only return up to 10,000 results for a query. For some searches, such as transactions, customers often need Secure Payments to return more than 10,000 results. This article goes over how to get Secure Payments to return larger data sets.

Get Transactions

To get transaction data from Secure Payments, send a POST request to:

https://pciwallet.simnang.com/api/search/transaction

The payload will be an Elasticsearch query object that looks something like this:

{  
"search":{
"query":{
"bool":{
"must":[
{
"bool":{
"should":{
"query_string":{
"query":"*", //keyword query
"fields":[
"_id",
"id",
"amount",
"customer_name",
"processor",
"metadata",
"batch"
],
"default_operator":"AND"
}
}
}
},
{
"match":{
"batch.keyword":"59899f74-b3b4-4b67-8376-55fcd70e2708" //Batch ID
}
}
]
}
},
"aggregations":{
"aggs":{
"terms":{
"field":"batch.keyword",
"size":100
}
}
},
"size":10
}
}

The object will let you specify the number of results to return, with a cap at 10,000.

Getting More Results

A maximum of 10,000 results can be returned with a standard query. For speed purposes we don't recommend a single query that returns 10,000 results. Instead, multiple queries should be sent. Each new query should get a set of results starting after the last transaction from the previous query.

For example, if you sent a query that returned results with transaction IDs that ranged from 3569 to 4126, the next query should return results that start with the transaction with an ID of (4127 assuming that transaction exists).

In order to accomplish this, add the "search_after" property to your payload. Your payload may look something like this:

{
"search":{
"size":10000,
"query":{
"bool":{
"should":[
{
"term":{
"batch":{
"value":"2351eb99-d45e-4c33-b28b-64e75a748932",
"boost":"1.0"
}
}
}
],
"adjust_pure_negative":"true",
"boost":"1.0"
}
},
"search_after":[
"10755605"
],
"sort":[
{
"id":"ASC"
}
]
}
}

search_after is an array, and to get the results you want, you should enter the transaction ID for the previous results set as a single value in the array.

Real-Life Use

The way customers use search_after to get larger data sets is typically this:

  • Run a query to get the batches generated that day
  • Check the aggregations for each batch to see how many transactions each batch included
  • If any batches contain more than 10,000 transactions, send multiple requests to get all transactions in the batch. The "search_after" value for each query should have a value of the last transaction ID (tranId) from the previous query.


How did we do?


Powered by HelpDocs (opens in a new tab)