RE: Persist Archiver in Map/Reduce
I am creating pdf’s of daily invoices (5000+) in the map stage, and passing the file id’s to the reduce stage to zip them into a single file. However, the compress module’s archiver requires loading the file which uses 10 governance points, so I hit the reduce stage limit around 500 invoice files. Is there a way to reopen the archiver or keep it open between reduce instances or map/reduce tasks?
I’m not 100% certain this will work but…
Are you able to pass the file contents to reduce?
Then file.create() cots you nothing and I don’t think you need to actually save the file in oder to add this to the archive???
Yes! Thank you.
I was able to zip 10,000 invoice pdfs in a single reduce instance with b64 file contents passed from map stage instances.
| reduce |
|
I was probably very close to the 100M of instructions limit (zip file is 47mb, decompressed is 71mb)
Only down side: once converted to base64 (file.getContents), NetSuite doesn’t convert it back to binary at file.Create so the file size is ≈ 150%
When you call file.create({...}, are you setting the fileType option to file.Type.PDF?
