Export Processor Limitation on Processing Large Amount of Records
Release Date: 5/13/2021
It said it exported 1,621,017 records which seems about right I guess. In the actual exported file, when it hit record 1,242,387, it started repeating the same exact row of data until the end of the file. See sample screen below.
It also takes a very long time to close the Export Processor window after clicking the X button. I don't know if some cleanup is happening or what, but it took much longer than expected.
A - I think the problem you experienced is because the system ran out of memory.
In the first phase of Export Processor, you will use “get records” in the Process tab. The Process tab will read all qualified invoice records and put them in the “List View.” The handling of the List View memory is actually done by the OS, not by us. All we do is just keep on pushing records to the List View and let the OS obtain more memory to handle our request. Elliott is a 32-bit application, and at some point, the OS will run out of memory for a 32-bit application. I know the maximum memory that a 32-bit application can access is 2GB. I don’t know if the OS will further limit that in a List View.
If you like, you can use Export Processor and choose “get records” again, to see how the Elliott memory grows in the Task Manager and where it stops. That will give you some clue. My test shows each 100,000 records takes about 100MB of memory. I did not test to go beyond a million rows. But you can see my sample screen below to get an idea:
Version: 8.0 & Up
Q - I exported all invoice header records from our
production company using Export Processor. No filters were used. As you can imagine, this
took a long time. No errors were encountered while it was processing either
the process or export jobs.
It said it exported 1,621,017 records which seems about right I guess. In the actual exported file, when it hit record 1,242,387, it started repeating the same exact row of data until the end of the file. See sample screen below.
It also takes a very long time to close the Export Processor window after clicking the X button. I don't know if some cleanup is happening or what, but it took much longer than expected.
A - I think the problem you experienced is because the system ran out of memory.
In the first phase of Export Processor, you will use “get records” in the Process tab. The Process tab will read all qualified invoice records and put them in the “List View.” The handling of the List View memory is actually done by the OS, not by us. All we do is just keep on pushing records to the List View and let the OS obtain more memory to handle our request. Elliott is a 32-bit application, and at some point, the OS will run out of memory for a 32-bit application. I know the maximum memory that a 32-bit application can access is 2GB. I don’t know if the OS will further limit that in a List View.
If you like, you can use Export Processor and choose “get records” again, to see how the Elliott memory grows in the Task Manager and where it stops. That will give you some clue. My test shows each 100,000 records takes about 100MB of memory. I did not test to go beyond a million rows. But you can see my sample screen below to get an idea:
When you try to exit Export Processor, it needs to release all the memory used by the application, so it takes a long time. This is a normal behavior.
If this limitation is because of memory, I think the strategy for you is to break it up into two batches to export and append the file when done.
Just FYI, Excel has a limitation of 1,048,576 rows.
If this limitation is because of memory, I think the strategy for you is to break it up into two batches to export and append the file when done.
Just FYI, Excel has a limitation of 1,048,576 rows.
EMK
Related Articles
CP0000 Customer Order Processing Overview
Overview How To Use This Manual When most people receive a new product, they want to use it right away. Unfortunately, this is not quite possible when dealing with computer hardware and software. This manual is intended to serve as a reference guide ...CPEXPMNU Customer Order Processing Sales Order Export
Sales Order Export Application Overview The export module is designed to export COP Sales Orders, Invoices and Invoice history data to a user-defined formatted ASCII file. The application can be used for EDI or any other applications that you may ...Using the Export Processor to Export Data
Q: It has been a while since we used the Export Processor to Export Data from Elliott. I remember how to select the category and choose the search data, but I can't remember how to select all the fields that need to be processed for export. Can you ...Feature - Support 12 Months History in Export Processor
Release Date - 7/13/17 Note: This feature is only available in version 8.2 and above. The Export Processer now supports the 12 Month History Summary file for the following tables and values: Customers: Total sales for the customer by period Total ...Elliott Physical Count Processing
Q. How can we process physical count with Elliott using a spreadsheet export and import method? We are prefer to have a variance report before posting to inventory. A. Elliott provides a verify efficient tool to create a worksheet with a spreadsheet ...