LoanPro’s data files solution helps you store and analyze historical data from across our platform in your data lake, including loan, line of credit, payment, and customer data. It’s ideal for large scale reporting and for combining LoanPro data with external sources.

Unlike a traditional database, which is optimized for daily operations, a data lake is optimized for scale. By sending your data in Parquet files (a highly efficient, column-based format), you can store your data more efficiently and retrieve it faster.

You can store your data in its original format, access it in the structure that makes sense for your business, and query it using tools like Amazon Athena or Snowflake. Whether you’re creating dashboards, running audits, or building predictive models, our data files solution gives your team the flexibility to work with your data however you need.

Sign up and configuration

Before receiving your data files, you’ll need to reach out to your LoanPro contact to get started. We’ll work with you to define the right setup based on your environment, data needs, and update frequency. From there, we’ll confirm which datasets will be shared, how they’ll be delivered, and outline everything in your contract to make sure we’re aligned before implementation begins. 

Data lake architecture

The files we deliver follow the Medallion Architecture, which organizes data into three layers. Each layer is built to support different use cases, depending on how much processing and transformation you want applied to your data before analysis:

• Bronze layer (raw data). Stores your data in directed, untouched copies of tables from your source databases. It’s useful when you need a complete, original record for audits, debugging, detailed portfolio performance analysis, or accounting workflows.
• Silver layer (cleaned data). Data in this layer is joined into more comprehensive, standardized tables. We remove duplicates, align formats, and mask sensitive fields. This layer is better for consistent reporting or when you need to join data across tables for a broader analysis.
• Gold layer (aggregated data). This layer’s data is grouped, summarized, or calculated for clients with specific business use cases, like portfolio performance, payment behavior, or collections insights. It’s ready for use in dashboards or advanced analysis.

Because Silver and Gold layers involve additional processing and curation, they can add time to your implementation. This layered structure lets you work with your data at the level of detail that fits your workflow.

Data collection and prep

LoanPro’s data lake is powered by a centralized data pipeline. This pipeline pulls data from your LoanPro environment, prepares it for analysis, and stores it in Amazon S3 in a format that’s optimized for large-scale use. 

Here's how it works:

This structure makes it easier to find what you need, track how your data changes over time, and refer back to specific points in time for audit or compliance purposes.

Implementation and data sharing details

How you format your data files depends on a few things, like the platform you’re using and how often you need updates. We’ll help guide you through the setup to make sure it works well with your system.

Here’s a breakdown of key implementation variables:

• Data platform. If you’re using a platform that supports AWS natively (like Snowflake or Databricks), a setup is usually straightforward. If you’re using Azure or GCP, a VPN is required. 
• Data scope. You can request specific tables or columns to limit the amount of data shared and improve performance.
• Report types. If you need specific or custom reports, let us know up front so we can account for any extra setup.
• Update frequency. By default, data is extracted once per day, after our daily maintenance window. If you need a different frequency for your use case, we can discuss possible adjustments.

All of these details will be confirmed in your contract, including which data will be shared, how often, and in what format. 

How data sharing works

Data sharing depends on the cloud environment where your data platform is hosted. The setup process is simplest if you're on AWS, but we also support other environments with a few extra steps. Here are two common examples.

Data storage in S3

Your data is stored in Amazon S3 using a consistent folder structure. This structure makes it easy to locate specific tables, track data over time, and manage historical files.

The exact setup depends on your hosting environment:

For both Saas and VPC, initial data extractions will be labeled as “full,” and all following extractions will be “delta” snapshots that include only changed records since the previous pull.

This structure makes it easier to organize large volumes of data, maintain a record of changes, and retrieve historical snapshots when needed.

Customizing your dataset

LoanPro provides a standard set of datasets based on commonly requested tables. If you need additional tables, specific columns, or something more custom, we can work with you to define a dataset that fits your requirements.

To request a custom dataset, reach out to your regular LoanPro contact.