Batch Execution in Drive Explorer

When the number of files needs to be listed is enormous, the process is batched. It can happen in both the methods of listing.

Google Sheet

As mentioned in the apps script guide, maximum execution time of any add-on script is 6 minutes for Gmail & Google Workplace (G Suite) users. If you have to process a lot of files with so many nested sub folders, there are chances that Drive Explorer will exceed the max execution time.

In that case, Drive Explorer batches the process so that it can process as many files as it can. Batching is a workaround for the limitation Google has enforced. Under the hood, Drive Explorer takes a break for some seconds before getting back to listing your files. When batching happens, you will get to see a notification like the one below.

Batch execution is a workaround rather than a solution to the problem. Use listing from the Drive method when you can for a proper solution.

drive explorer batch process in google sheet

You can check the status of batch execution anytime by selecting, Extensions → Drive Explorer → Batch Execution Status.

Google Drive

Google Drive method can list any number of files without any issues. But, when it fetches a huge amout of data, the browser runs out of memory and can crash. To avoid this, Drive Explorer start to stream the data to a local CSV file.

Once the total size of the data fetched reaches ~30MB, a file will start to download or you will get a download request (if where to save downloaded file config is enabled on your browser). This can happen immediately after you start fetching files when you enable data streaming before fetching the files.

When the download (read streaming) starts, you will see something similar to the below image on your download pane in your browser. You should not close the Drive Explorer tab. This is not like downloading any other file on the internet. The tab should stay open to continue writing files.

drive explorer streaming data to a csv file

Enable data streaming

When you are sure there will be hundreds of thousands of files inside the folder you open with Drive Explorer, it's recommended to enable data streaming.

When you enable data streaming before fetching files, a CSV file will start downloading immediately. This means the data is written to that file instead of holding it in the RAM memory. You are not supposed to close the Drive Explorer tab to continue fetching and writing files.

drive explorer enabling data streaming

Folder size

When the file data is directly written to the CSV file, it will not have folder size column. Usually after fetching all the files, Drive Explorer updates the folder size of each folder. That's not possible in this case as the data is already exported to the file.

So, the folder size data is exported separately in another CSV file, after the fetching process is completed. That CSV file will have Folder name, Path, Size, Size in bytes (for sorting by size) & Link columns.

Authentication

Drive Explorer after it starts fetching files, it can continue to process files without interruption for an hour. After that, the access token expires. It should be fetched again. So, an authentication pop-up opens and waits for your approval.

Once the approval is provided, Drive Explorer will continue to fetch files from where it left.