Hi,
I am facing a unique challenge, problem is to process a million flatfiles for ELT.
Using TPT to leverage parallelism and optimized loading. However, acquisition phase is taking longer than expected.
I am using batch directory scan, with 2 file reader instances. The process picks up well and ends reading ~1000 files in less than 30 secs, however, it slows down once crosses ~3000 files.
Avg file size is 500KB.
Need inputs as to how this can be optimized further hence reducing the overall time.
↧