Data Lake Export
Automated daily delivery of immutable, incremental Parquet files to S3 for enterprise warehousing and reporting
The LoanPro Data Lake Export is a scheduled daily batch extraction pipeline. It periodically extracts modified operational data across LoanPro services (i.e., LMS, Connections, and Secure Payments) and delivers compressed, Apache Parquet files directly to an Amazon S3 landing zone.
Unlike standard databases built for real-time application operations, Data Lake Export creates a permanent, unchangeable history of your data. It is specifically designed to feed enterprise data warehouses (like Snowflake, Databricks, and BigQuery), power cross-system reporting, and meet regulatory compliance needs.
Additional data tools
While Data Lake Export is ideal for large-scale analytics, LoanPro provides several complementary tools suited for specific operational needs:
- UI Reports: Best for ad hoc reporting and teams without the technical resources needed to query a database directly.
- API: Integrates LoanPro with the rest of your operations, allowing you to pull information and make updates in real-time.
- Relational database service (RDS): A direct database connection that lets you query all of your data in real time using SQL.
- Data on Demand: A managed query service for teams that need custom database extracts without writing SQL. LoanPro configures recurring queries and delivers the data directly to a secure location of your choice.
Together, these tools provide flexible data access and integration options for lenders of any size.
Extraction pipeline and S3 directory layout
Extraction schedule
- Daily batch extractions execute automatically post-maintenance between 6:30–7:30 a.m. Central Standard Time.
Snapshot methodology
- Baseline Seed (
/full/): When a tenant or table is first initialized, LoanPro performs a complete extraction to establish a point-in-time baseline snapshot. - Daily Incremental Deltas (
/YYYY-MM-DD/): All ongoing daily exports contain incremental deltas instead of full table drops.- Contains: Only rows modified or inserted between 00:00:00 Coordinated Universal Time (UTC) of the previous calendar day and the current day's execution time are included.
- Folder Path: Folders are stored under the date that matches the extraction run date (e.g.,
/2026-09-04/). - Empty Runs: Delta folders are event-driven, so if a table has no updates during a 24-hour cycle, no folder is created for that day.
S3 landing directory structure
Data is delivered to dedicated S3 bucket paths that are organized by tenant, source application, and table entity.
| Path Type | Path Format | Description |
|---|---|---|
| Bucket Name | <bucket_name> | Environment-specific S3 bucket target. |
| S3 Key/File Path | <tenant_id>/<application>/<table_name>/<delta_extraction_date|full>/<parquet_files> | Relative S3 key structure. |
| Full S3 URI | s3://<bucket_name>/<tenant_id>/<application>/<table_name>/<delta_extraction_date|full>/<parquet_files> | Qualified S3 path. |
Refer to the table below for detailed descriptions and expected values for each dynamic path variable.
| Variable | Definition | Allowed Values/Formats |
|---|---|---|
| <bucket_name> | Environment-specific S3 bucket target. | SaaS: VPC: |
| <tenant_id> | Unique LoanPro tenant identifier. | Numeric/string ID. |
| <application> | Source engine generating the data. | lms, secure-payments, connections |
| <table_name> | Extracted database entity. | Target entity name (e.g., loan_entity, customer_entity, loan_tx). |
| <delta_extraction_date|full> | Storage path indicating the export type. |
|
| <parquet_files> | Apache Parquet data files. | Written in parallel by distributed Spark executors. |
Destination support & cloud integration
LoanPro supports automated data lake delivery across major data warehouses, cloud platforms, and storage destinations. Delivery mechanics are structured around client environment architecture, security compliance, and cloud provider capabilities.
AWS & cloud warehouse destinations
For AWS environments and cloud-agnostic data platforms operating on AWS infrastructure, delivery is fully automated via native AWS security primitives. There are two delivery options:
1. Standard Tier (Direct S3 Write)
- Architecture: An AWS Glue extraction job writes data directly from LoanPro's pipeline to your designated S3 target bucket.
- Encryption: SSE-S3 (AWS’ default server-side encryption).
- Ideal Use Case: Lenders seeking a standard, automated data feed without custom key-management overhead.
2. Enterprise Tier (S3 Cross-Account Replication)
- Architecture: Data lands in a LoanPro-hosted bucket first, then syncs automatically to your target S3 bucket via AWS S3 Cross-Account Replication (CRR/SRR).
- Encryption: SSE-KMS (server-side encryption with AWS Key Management Service Keys).
- Ideal Use Case: Enterprise lenders with strict security or compliance mandates requiring full ownership, management, and rotation of their own data encryption keys at rest.
Note for Non-AWS environments (e.g., Azure or Google Cloud Platform): If your data platform operates on a non-native AWS cloud provider, please contact your LoanPro representative for the latest technical integration patterns and supported architectural options.
Legacy SFTP (platform-agnostic file drops)
Use this option if you need to deliver files to an SFTP server without using cloud storage infrastructure.
- Architecture: Parquet files are pushed directly to an SFTP server that you host and maintain.
- Encryption: Standard SFTP in-transit encryption (SSH).
- Ideal Use Case: Lenders requiring a fallback option when primary delivery methods (i.e., Standard Tier and Enterprise Tier) are not viable.
Integration FAQ & troubleshooting
Why am I seeing duplicate entity records (i.e., multiple rows with the same ID) across my historical data?
Answer: Our daily incremental delta files deliver the latest state of modified records for a specific extraction run.
- Same-day Changes: If a loan only updates multiple times in a single day prior to the extraction job, the daily delta file will only contain 1 row representing its final state at extraction time.
- Multi-day Updates: If a loan updates on subsequent days, a new record with the
last_updatedtimestamp will be delivered in each corresponding daily folder.
How do Primary Keys work in raw data lake files?
Answer: The id column represents the Business Entity Primary Key (i.e., the loan or customer a row belongs to). Since raw S3 files track historical state changes over time, the unique physical row identifier in the raw landing zone is the composite key (id + last_updated). Once merged downstream into your target warehouse, id returns to a strictly unique entity primary key.
What happens if there is a processing error or a missed delivery?
Answer: LoanPro actively monitors data lake pipeline deliveries. In the rare event of a processing or system error, our Engineering Team immediately escalates the matter internally. Our primary goal is to always resolve the issue and re-deliver files within the same business day, while keeping your team updated through our standard support channels.
If a technical interruption prevents us from delivering a file within the same day, our pipeline safely preserves the checkpoint of your last successful run. In the event of a multi-day service disruption, missing data is recovered via an ad-hoc backfill execution—independent of the daily scheduled pipeline. This out-of-band run performs a catch-up extract, which generates a consolidated Parquet file containing all modified records across the outage window.
To maintain strict ingestion alignment, this backfill file is stored in the S3 folder named after the specified execution date instead of retroactive historical folders. On the day of recovery, two distinct Parquet files will land in that date’s folder:
1. Ad-hoc backfill file
2. Standard daily incremental file
This checkpoint-and-catch-up system ensures your target environment never misses a single update, even if our daily delivery cadence is temporarily disrupted.
What happens if a data lake gets out of sync with LoanPro?
Answer: If you discover that your local data lake has fallen out of sync or experienced data drift for any reason, you can request a manual resynchronization on a per-table basis. LoanPro can then target the specific table(s) that require correction, so you do not have to reset your entire data lake—unless you want to.
How it works:
- Request: Submit a support ticket through your standard LoanPro communication channels, specifying which table(s) need a reset.
- Process: LoanPro’s Engineering Team will coordinate directly with you to schedule a new full extract.
- If you own the S3 landing bucket, your engineering team is responsible for clearing or archiving the existing data in your target S3 destination before the resync to prevent the mixing of new and old snapshot files.
- If LoanPro owns the S3 landing bucket, LoanPro will coordinate the cleanup of the target S3 path before generating the new snapshot, avoiding unintended data deletion or duplicate file collisions.
- Delivery: Once scheduled, LoanPro will upload a brand-new, comprehensive baseline snapshot into your /full/ path for the required table(s), allowing you to completely overwrite your local copy and reestablish a perfect baseline before your daily deltas resume.