Importing through the API

What is an import, and how does it work in the API?

Imports let you move a large body of data into LoanPro much faster than the regular user interface would allow (and with a lot less work, as well.) The basic process involves creating a .csv file with your data, then moving it into LoanPro. It might be helpful to read through our main article on The Import Process first. It explains everything that happens along the way, and shows how to add your import file from within the UI. This article, on the other hand, will show how to go through that same process through the API.

The Overall Process in the API

Just like in the UI, the process here involves four steps:

1. You fill out and upload your .csv file.
2. LoanPro validates the file to make sure all the data is compatible.
3. LoanPro ingests your data.
4. LoanPro verifies your data and returns a report detailing the results. This is available on Customer and Loan Setup imports, but we'll soon offer it on all imports.

Each step will involve sending a POST or PUT request to an endpoint that specifies both the step of the process and the specific import you're working on. This article will include those endpoints, as well as the JSON payloads you'll need.

Step 1: The CSV File

Before you do anything in a REST client, you'll need to actually compile your data into a comma-separated value (.csv) file. The main Import Process article explains where you can find templates and what software you can use to edit them.

Once you've put all your data into a .csv, you're ready to send it through the API. This is actually done with two separate requests: First, a POST request that primes the system to receive your file and generates a pre-signed upload URL, and then a PUT request including the file itself.

Primer

Send a request to this endpoint:

POST https://loanpro.simnang.com/api/public/api/1/data/import/upload/{OdataEntity}

You'll want to replace {Odata Entity} with the specific type of file you're importing. This tells LoanPro what data to expect, and where to put it.

Here's a Full List of Odata Parameters

Just replace {OdataEntity} in that sample endpoint with the item below. For instance, if you wanted to import a Customer Info .csv, you'd send it to https://loanpro.simnang.com/api/public/api/1/data/import/upload/Customers

Many of these parameters are just the name of the category without spaces, but there's a few oddballs in there. Here’s the list:

Customers

• Customer Info – Customers
• Payment Profiles – PaymentAccounts
• References – References
• Notes – Notes
• Custom Fields – CustomFields
• Tool Actions (OFAC check, USPS verify, etc.) – CustomerTools
• Access – Loans

Loan Account

• Loan Setup – LoanSetup
• Settings – LoanSettings
• AutoPay – Autopays
• Checklist – Checklist
• Collateral – Collateral
• Custom Fields – CustomFieldValues
• Insurance – Insurance
• Notes – Notes
• Promises – Promises
• Enroll Recurring Charge – LoanRecurrentCharges
• Enroll Loan Settings Rules Applied – LoanRuleAppliedSettings
• Enroll Credits Rules Applied – LoanRuleAppliedChargeOff
• Enroll APD Reset Rules Applied – LoanRuleAppliedAPDReset
• Enroll Loan Checklist Rules Applied – LoanRuleAppliedChecklist
• Enroll Change Due Date Rules Applied – LoanRuleAppliedChangeDueDate
• Enroll Stop Interest Date Rules Applied – LoanRuleAppliedStopInterest
• Enroll Account Tools Rules Applied – RuleAppliedAccountTools
• Enroll Customer Tools Rules Applied – RuleAppliedCustomerTools
• Actions And Results – ActionResultsNotes
• Bankruptcy – Bankruptcies
• Simple Roll Schedule – ScheduleRolls
• Tools
• Link Portfolios – LoanPortfolio
• Tool Actions (i.e. archive, resurrect, delete, activate) - LoanToolActions

Loan Transactions

• Payment Custom Fields – PaymentCustomFieldValues
• Payments – Payments
• Recovery Payments – RecoveryPayments
• Charges – Charges
• Credits – LoanCredits
• Suspend/Resume Interest – StopInterestDates
• Change Due Date – LoanDueDateChanges
• Escrow Settings – EscrowSubsetOptions
• Escrow Transactions – EscrowTransactions
• Escrow Disbursement – Disbursements
• Cash Drawer Transactions – CashDrawerTransactions
• Charge Off Transactions – ChargeOff
• Amount Past Due Adjustments – APDAdjustments
• Days Past Due Adjustments – DPDAdjustments

Company Settings

• Create Source Companies – SourceCompanies
• Create Portfolios – Portfolios
• Create Sub Portfolio – SubPortfolios

Here's a sample JSON payload:

{	"fileName": "loanSetup123.csv",	"importProgression": "manual",	"validationType": "lineRejection"}

• "fileName" – The name of your file. Include .csv at the end.
• "importProgression" – Set to "manual" or "auto". Manual means that you'll need to send requests to start the next steps of validating, ingesting, and verifying the data. With automatic, you'll just need to send the file and your work is done.
• "validationType" – Set to "fileRejection" or "lineRejection". This determines what happens if there are errors in your file. Line rejection will import all the good rows and give you a file with the rejected rows; file rejection ditches the whole thing if there's an error, but will tell you where those errors are.

When you send the request, you'll get a response that includes a lot of information about your import.

Here's a sample 200 response.
{   "d":{      "__metadata":{         "uri":"http://loanpro.simnang.com/api/public/api/1/odata.svc/DataImport(id=221)",         "type":"Entity.DataImport"      },      "id":221,      "entity":"Entity.Customer",      "status":"dataimport.status.uploaded",      "description":"Customer Data Import",      "fileName":"dataImport/prod/tenants/234/files/dataImport/Customer/customers1234_609a.csv",      "reverseFileName":null,      "statsNew":0,      "statsUpd":0,      "statsDel":0,      "originalNew":0,      "originalUpd":0,      "originalDel":0,      "statsErr":0,      "errors":0,      "modified":"/Date(1620764559)/",      "byPassWarnings":1,      "skipCalc":0,      "numRows":0,      "totalBlocks":0,      "verifiedBlocks":0,      "importedBlocks":0,      "created":"/Date(1620764559)/",      "deleted":0,      "createdBy":1938,      "validationType":"lineRejection",      "importProgression":"manual",      "rejectionFile":null,      "coverSheet":null,      "deltasFile":null,      "downloadURL":null,      "rejectionFileUrl":null,      "coverSheetUrl":null,      "deltasFileUrl":null,      "User":{         "__deferred":{            "uri":"DataImport(221)/User"         }      },      "url":"{THIS IS THE URL FOR YOUR NEXT REQUEST}"   }}

You'll need two of those fields in your next requests, "url" and "id". The "url" field provides a pre-signed S3 URL, which specifies the location where the file will be uploaded. This URL will expire in 15 minutes, so it should only be created for immediate use.

The File

This step is unique for three reasons: It's a PUT request instead of a POST, it doesn't have the authorization and token in it's header, it's a Multipart Form instead of a JSON payload, and it's sent to the endpoint that you got in the response from the last request. Here's a sample payload, but your REST client might have a more user-friendly interface that just let's you drag and drop the file. Once you submit, you should get a 200 response with no body.

-----------------------------9867132174374996981330441183Content-Disposition: form-data; name="upload"; filename="(file name goes here)"Content-Type: "text/csv"(file binary goes here)-----------------------------9867132174374996981330441183--
Make sure you submit your .csv as a binary file. If you submit a non-binary file, you'll still get a 200 response at this point, but LoanPro will not be able to ingest the data later on. We support binary files because they're smaller than base64, which means imports (especially large imports) will integrate into the system much faster.

Half Step: Checking an Import's Progress

If you're using manual progression, you'll need to send POST requests to begin the next three steps of validating, ingesting, and verifying the data; however, each step must be complete before the next begins. You can check on the status in the UI (Settings > Company > Imports > Uploads & History) or through a GET request to an endpoint specific to your import. The endpoint will look something like this:

GET https://loanpro.simnang.com/api/public/api/1/data/import/details/{OdataEntity}({import id})

The endpoint will include the same Odata Entity that you used for the first request, and the import ID from that requests' response. For example, if you're doing a Loan Setup import and the ID is 211, it'll look like this:

https://loanpro.simnang.com/api/public/api/1/data/import/details/LoanSetup(211)

The response will give you a couple of URLs (this is where you'll get your reports at the end of the process) and the "status" field will tell you whether a step is in progress or complete.

Here's a sample 200 response
{  "d": {    "id": "218",    "fileUrl": "URL FOR YOUR CSV FILE",    "rejectionFileUrl": null,    "coverSheetUrl": null,    "verificationFileUrl": null,    "status": "dataimport.status.impinprogress"  }}

Step 2: Validation

If you've selected automatic progression, LoanPro takes care of this for you. If you selected manual progression, however, you'll need to send a POST request to kick start this step. The endpoint will include the same Odata Entity that you used for the first request, and the import ID from that requests' response. Here's the endpoint and payload:

POST https://loanpro.simnang.com/api/public/api/1/data/import/validate/{OdataEntity}({import id})
{	"\$byPassWarnings": 1}

Step 3: Ingestion

Again, if you selected automatic progression you can just sit back and let LoanPro do it's thing. If you selected manual progression, you'll need to send another POST request to kick this step off. First thing's first, make sure that the data is validated with a GET request. (See the Half Step above for details. When Step 2 is finished, the "status" value should be "dataimport.status.validated".)

When that's done, you're ready to send the next request. It's a POST (with no payload body) to this endpoint:

POST https://loanpro.simnang.com/api/public/api/1/data/import/{OdataEntity}({import id})

Like the other endpoints in this process, you'll need to put in the your same Odata Entity and import ID.

You'll get a 200 response with no body if it's successful. Now you just wait for LoanPro to ingest your data. Imports are processed in the order they come in, so even a small import won't be ingested until earlier imports are finished, which can delay you here. Once it's ingested, though, your import is done. The last step double-checks everything and gives you reports on how it went, but by this point all of your import data is integrated with the existing information in LoanPro.

Step 4: Verification

Verification is a new feature we've just added to Customer and Loan Setup imports, but it'll soon be available for imports of any kind. Here, LoanPro compares your import file's data against everything that is now in the system. Like the last few steps, this one happens on it's own if you chose automatic progression. If you chose manual progression, you'll want to start with a GET request to make sure LoanPro is done ingested the file (See the Half Step for details). Then, you'll start the verification process send a POST request with no body to this endpoint:

POST https://loanpro.simnang.com/api/public/api/1/data/import/verify/{OdataEntity}({import id})

And yet again, you'll need to put in the your same Odata Entity and import ID.

When this step is complete, your import is finished, and LoanPro will generate several reports going over the results — you can get more info about individual reports with our article on Import Reports.

You can access these reports through the API with a GET request (the same GET request, in fact, that you've been using to check the import's progress). The endpoint will change with the Odata Entity and import ID, but here's the template:

GET https://loanpro.simnang.com/api/public/api/1/data/import/details/{OdataEntity}({import id})

You'll get a response with URLs that you can follow for your reports. It'll look like this:

{  "d": {    "id": "218",    "fileUrl": "ORIGINAL FILE URL",    "rejectionFileUrl": "REJECTION FILE URL",    "coverSheetUrl": "COVER SHEET URL",    "verificationFileUrl": "VERIFICATION FILE URL",    "status": "dataimport.status.imported"  }}