API - Transaction Warnings Overview

General

The purpose of transaction warnings it to help users avoid creating duplicate data within the system. The decision to return a transaction warning is based on a comparison of key pieces of data. If data you submit through the API looks like it might be redundant, LoanPro will return a transaction warning.

Since loans and customers are completely separate in LoanPro, different checks are made within the system for each. When a new customer is submitted, LoanPro checks existing customers to see if matches exist for:

  • Social Security/Social Insurance Number
  • Email Address
  • Phone Number
  • Primary Address
  • Mailing Address

When a new loan is submitted to the API, the system will check the display ID (the loan ID designated by your company) to see if it already exists in your account. There is also a warning for collateral if a duplicate VIN is submitted.

Catch the Warning

To catch a warning, you need to wait for a response. Always wait for a response! There have been countless times when we got frantic calls about missing data, only to find that the customer had submitted date, but the submission failed and no one was waiting for a failed response! Not waiting for a response is like clicking the “Place your order” button on amazon and then immediately turning off your computer and cancelling your internet. Did your purchase go through? You don’t know. Did it ship yet? You don’t know. Will the package come? Maybe. Can you check? No. This may seem like an overly-dramatic example, but when we do important things in our personal life, we always wait for a response. We want to know, did we get what we want or do we have to try again? Did payment go through or was my credit card declined? Is the check valid or did it bounce? Did they like my gift or did they think it was awful? The same principle applies to computing; always wait for a response!

Now that you are waiting for a response (and if your code isn’t, go do that right now, we’ll wait), the next question is how do you tell if the response is a transaction warning? Well first, we’ll start with a different question. How do I know if my request was successful? We only really care if it was a transaction warning or an error if it didn’t work.

Detecting A Success

There are two requirements for any response to be considered a success. Both requirements must be met. They are:

  1. The HTTP response code is 200
  2. A “d” object exists in the root

Succesful Response:

{
"d": {
// All data goes here
}
}

If the HTTP response code is not 200 or there is not a “d” object in the root, then it is not a successful response. Only if both requirements are met do we consider it a success.

Detecting a Transaction Warning

Once we know what a success looks like, it’s easy to know what a failure looks like; i.e. everything that isn’t a success! So how do we distinguish a transaction warning? Transaction warnings will have a data object with a nested error object, and in that error object there will be a type field with the value TransactionWarningException. An example is shown below:

Transaction Warnings Response:

{
"data": {
"error": {
"message": "Transaction warnings found.",
"type": "TransactionWarningException", // type field
"code": 500
},
...
},
...
}

When coding to recognize a transaction warning, first test the server response to see if it is successful. If the response is not successful, check it to see if it contains a “data” object with a nested “error” object which includes a “type” of “TransactionWarningException”.

Here is a pseudo-code example for recognizing a transaction warning:

response = sendRequestToServer()
if not_successful(response) then:
// problem with request
if exists_in(response, "data") and exists_in(response.data, "error") and
response.data.error.type equals "TransactionWarningException" then:
// Handle transaction warning
else:
// Some other error
end if
else:
// it was successful
end if

Handling the Warning

There are several options on how to handle transaction warnings. We will only cover a few here.

  1. Ignore Warnings

Transaction warnings are just that, warnings. They can be overridden simply by adding "__ignoreWarnings": true to every object in the payload (including the nested ones).

Here is an example of a payload that will ignore transaction warnings:

{
"displayId": "Hi",
"LoanSetup": {
"loanClass": "loan.class.carLoan",
"loanType": "loan.type.installment",
"__ignoreWarnings": true
},
"__ignoreWarnings": true
}

That’s it! Your data will be saved, so you won’t lose anything. The only downside is you may enter duplicate data. If you’re going to do this, you won’t get warnings at all, but you should still wait for a response since there may still be errors.

  1. Analyze Duplicates (Recommended)

This is the recommended solution. It’s also the hardest to implement. For this method, whenever you get a transaction warning you will look at the reason for the warning and then either use the information it provides or show it to the user.

We’ll use the example above. Here’s the important stuff for reference:

  • Customer transaction warnings :
{
"data": {
"error": {
"message": "Transaction warnings found.",
"type": "TransactionWarningException",
"code": 500
},
"warnings": [
"Email jd@none.com already exists in the system (Customer ID 1: John Doe).",
"Email address jd@none.com may not be a valid email account.",
"Ssn 012345678 already exists in the system (Customer ID 1: John Doe).",
"(Phones) Phone '2524732111' already found in the system (Customer ID 1: John Doe).",
"(PrimaryAddress) Address '123 OAK LANE, SCHENECTADY, NY, 123456789, USA' already found in the system (Customer ID 1: John Doe).",
"(MailAddress) Address '123 OAK LANE, SCHENECTADY, NY, 123456789, USA' already found in the system (Customer ID 1: John Doe)."
]
},
...
}

We’ll also talk about the following warnings for loan creation:

  • Loan Transaction warnings :
 ...
"warnings": [
"A duplicate collateral with a VIN number of 'JTHGL46F975057216' was found on Loan identified by 'JK-1002541111' and created on '2017-12-07 21:08:26' UTC.",
"A duplicate loan with an ID of 'LN-5450-343' was found."
]
...

How to use regex :

regex = get_regex_to_test()
for each warnings as warning do:
if warning.matches(regex) then:
// Regex matched!
else:
// Regex didn’t match.
end if
loop
Customer Transaction Warnings

For customer transaction warnings, we’ll use the following regexes:

  • /ID (\d+)/
  • This will grab the customer IDs from the warnings as capture groups.
  • Capture groups allow you to get only the part of your string that matches the portion of the regex that is between the parentheses.
  • /may not be a valid email/
  • This checks if the problem is a possibly-invalid email address (i.e. the email address fails LoanPro’s verification).

In the case of warnings about duplicate customers, we will use capture groups to grab the IDs of the associated customers. Most often, when getting a regex match, you’ll get an array or iterable where the first element is the text that matched the full regex and the rest are matches for the capture groups. In this case, the ID of the original or duplicate customer would be the second element of the regex match. If the user decides to use the original customer instead of proceeding to create a new one, the ID can be used to reference the customer or link the customer to a loan.

We’ll look deeper into handling duplication warnings in the next three lessons, because to properly handle these warnings we need to understand how to link a loan and a customer. For now, if you get a possible-duplicate warning don’t create the customer. We’ll handle this case by giving the user the option to proceed with the creation of a new customer or link the existing customer to the loan.

If you get a match on the second regex (invalid email), then we know that the email may not be valid (LoanPro performs a lot of checks to make sure an email is valid). If this is the case, you can simply display the warning to the user and prompt the user to enter a valid email address.

Loan Transaction Warnings

Loan transaction warnings include a duplicate display ID warning and a duplicate VIN warning for the collateral entity. The collateral is a nested entity within a loan. Here are the regexes to use to pull the loan IDs from each type of warning:

  • /duplicate collateral .* found on Loan identified by '([^']+)'/
  • This will capture the loan ID of the loan associated with the collateral entity that contains the duplicate VIN.
  • /duplicate loan .* ID of '([^']+)'/
  • This will detect a duplicate loan display ID warning and capture the loan ID.

If the first regex matches, that means you are submitting collateral with the same VIN as another piece of collateral that already exists in your account. Currently, there is not a way to link an existing collateral entity to a loan, so we will simply ignore the warning. If a warning matches the second regex, the display ID you are submitting is already being used by another loan. The best practice is to create and submit a different display ID.

As an example, suppose we get the messages:

"A duplicate collateral with a VIN number of 'JTHGL46F975057216' was found on Loan identified by 'JK-1002541111' and created on '2017-12-07 21:08:26' UTC."

"A duplicate loan with an ID of 'LN-5450-343' was found."

We can then handle these warnings as follows:

  • How to handle transaction warnings :
// Sample data
// In this case it's just a VIN and display id
// but a real scenario would have the entire loan info
var loan_to_create = {
Collateral: {
vin: 'JTHGL46F975057216'
},
displayId: 'LN-5450-343'
}

// get the sample warnings
// In a real scenario, this would be done in an error handler
var warnings = get_warnings();

// A list of objects describing how to handle loan warnings
var loan_warning_handlers = [
{
regex: /duplicate collateral .* found on Loan identified by '([^']+)'/,
resolver: fix_collateral
},
{
regex: / duplicate loan.* ID of '([^']+)'/,
resolver: new_display_id
}
]

// Handle warnings
handle_warnings(warnings, loan_to_create, loan_warning_handlers)
.then(fixed_loan => {
// fixed_loan holds the correct Collateral info
// here is where you'd resubmit your loan
console.log(fixed_loan);
})

function handle_warnings(warnings, entity_to_fix, warning_handlers) {
// for each warning
var promises = warnings.map(warning => {
// test the warning against each reged
return Promise.all(warning_handlers.map(function (handler) {
var matches;
if ((matches = warning.match(handler.regex))) {
return handler.resolver(matches, entity_to_fix)
}
return Promise.resolve(null)
})).then(values => {
return values.filter(v => v);
}).then(values => {
// one match means return the result (will be merged into the entity object)
if(values.length === 1)
return values[0]
// no match means return an empty object (won't change the result)
else if (values.length === 0)
return {}

// If we get multiple matches that's a problem
// The regexes should be designed to have at most one match for any given warning
throw new Error("Too many regex matches")
})
})

return Promise.all(promises).then(values => {
// Merge all values into one object (allows for each handler to update a subset of the entity)
return values.reduce(((a, c) => Object.assign(a, c)), entity_to_fix)
})
}

function fix_collateral(regex_matches, loan_to_fix) {
// simply ignore warnings
return Promise.resolve({
"__ignoreWarnings": true,
Collateral: Object.assign(loan_to_fix.Collateral, {"__ignoreWarnings": true})
})
}

function new_display_id(){
// You'll want to replace this with a way to generate a unique id
return Promise.resolve({displayId:'Some-new-id'})
}

// This would get the warnings from the returned message in your code
// It only returns the sample warning for purposes of this demonstration

function get_warnings() {
return [
"A duplicate collateral with a VIN number of 'JTHGL46F975057216' was found on Loan identified by 'JK-1002541111' and created on '2017-12-07 21:08:26' UTC.",
"A duplicate loan with an ID of 'LN-5450-343' was found."
];
}

The above example handles the transaction warnings for creating a loan. It does not generate a new display ID. The reason for this is most companies are very particular about the elements display IDs contain and how they are formatted. Best practice is to work with your company to come up with a formula to create display IDs that conform with your company’s established convention.

Avoiding Duplicates

One of the best methods for dealing with Transaction Warnings is to implement measures to avoid ever submitting duplicate data. We’re going to discuss a couple of techniques here, but not all of them.

Before we get started, first a word of warning. None of these techniques defy the laws of physics. If you aren’t sure why that’s important, learn the theory of general relativity. If you didn’t just read that article, let me tell you the most important part of that theory: Events that occur at the same time for one observer could occur at different times for another observer. This great law of physics applies to all network communications, and using any cloud-based API or database is rooted in network communications. In short, when you get a response from a server, the server state could be different when you receive the response than when it was sent.

This most commonly happens when the server is serving multiple clients. In that case, as soon as the server finishes one request it starts on another request. Consequently, by the time you’re ready to send another request, the server’s state is already different. The reason I bring this up is we’ve had tons of customers who’ve ran into issues because they neglected this fact. Keep in mind that any response you get back reflects the server’s state at the time the response was sent, not the time you got the response. If you want a more detailed explanation, see our Advanced Transaction Warnings course.

Query the Server

This is the most common technique I’ve seen. It’s also the one that most commonly highlights the discrepancy for how programmers think about computing and how networks work, which is why I described the situation in the disclaimer above.

The idea here is to query the LoanPro API to see if an entity already exists (e.g. if there’s a loan with a matching display ID or a customer with a matching address). If no such entity exists, then you can create the entity. Otherwise, you handle the situation like you would a transaction warning.

We won't go into too much detail on how to do this as it's not our recommended method.

Keep an In-Memory Cache

This is slightly more complicated but is also effective. Keep an in-memory cache of the hashed version of every piece of data you process which could throw a transaction warning. Basically, whenever you create a loan, cache a hash of the VIN and Display ID used. Whenever you create a customer, cache a hash of their email, address, SSN, and phone number.

One thing to note: cache only a cryptographic hash of the SSN and other sensitive data! A person’s identity can be stolen if attackers get their SSN or other sensitive data. A cryptographic hash (such as SHA-2 or BCrypt) will make it harder for attackers to get that information.

There are a few downsides to this approach. One is that in-memory caches take up RAM and potentially a lot of it. Data is also not persisted (if the machine or caching program restart the cache is lost). Another is that, unless you use a distributed cache like Redis, the cache is only available on one machine. This means you either have a central caching server or a separate cache per server. If you do choose a distributed cache, be careful as they can take a while to propagate changes.

Keep a Database

This is very similar to the in-memory cache, the only difference being you use a dedicated database instead. While the difference seems subtle, it has a few advantages. The first being that it uses hard drive space instead of RAM - which is cheaper. Databases also persist their data, so you can restart the database without losing data.

However, you still need to cache only a cryptographic hash of the data! This is more important with a database since all data is persisted so hackers can potentially access more data and have more time to access it.


How did we do?


Powered by HelpDocs (opens in a new tab)