Download#
pyinaturalist_convert.download
Helper utilities for downloading and extracting files, with progress bars
- class pyinaturalist_convert.download.CSVProgress(*filenames, **kwargs)#
Bases:
MultiProgress
Track progress of processing CSV files
- class pyinaturalist_convert.download.FlatTarFile(name=None, mode='r', fileobj=None, format=None, tarinfo=None, dereference=None, ignore_zeros=None, encoding=None, errors='surrogateescape', pax_headers=None, debug=None, errorlevel=None, copybufsize=None)#
Bases:
TarFile
Extracts all archive contents to a flat base directory, ignoring archive subdirectories
- extract(member, path='', **kwargs)#
Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. member’ may be a filename or a TarInfo object. You can specify a different directory using `path’. File attributes (owner, mtime, mode) are set unless `set_attrs’ is False. If `numeric_owner is True, only the numbers for user/group names are used and not the names.
The filter function will be called before extraction. It can return a changed TarInfo or None to skip the member. String names of common filters are accepted.
- class pyinaturalist_convert.download.JobProgress(progress, task, task_description=None)#
Bases:
object
- progress#
- task#
- task_description#
- class pyinaturalist_convert.download.MultiProgress(totals, total_progress=None, job_progress=None, task_description='Loading')#
Bases:
object
Track progress of multiple processes run in serial, plus overall combined progress
- advance(advance=1)#
- start_job(name)#
- class pyinaturalist_convert.download.ParallelMultiProgress(total=0, total_progress=None)#
Bases:
object
Track progress of multiple processes run in parallel, plus overall combined progress
- advance(name, advance=1)#
- log(msg)#
- log_job(msg, task_description, name, task_size=-1)#
- refresh()#
Recreate table with current jobs
- start_job(name, total, task_description='Loading')#
- stop_job(name)#
- class pyinaturalist_convert.download.ProgressIO(path, *args, callback=None, description='Extracting', **kwargs)#
Bases:
FileIO
File object wrapper that updates progress on read and write.
- Parameters:
- read(size)#
Read at most size bytes, returned as bytes.
Only makes one system call, so less data may be returned than requested. In non-blocking mode, returns None if no data is available. Return an empty bytes object at EOF.
- write(b)#
Write buffer b to file, return number of bytes written.
Only makes one system call, so not all of the data may be written. The number of bytes actually written is returned. In non-blocking mode, returns None if the write would block.
- class pyinaturalist_convert.download.ZipProgress(archive, **kwargs)#
Bases:
MultiProgress
Track progress of extracting files from a zip archive
- pyinaturalist_convert.download.check_download(dest_file, url=None, bucket=None, key=None, release_interval=None)#
Check if a locally downloaded file exists and is up to date
- Return type:
- pyinaturalist_convert.download.download_file(url, dest_file)#
Download a file from a URL, with progress bar
- pyinaturalist_convert.download.download_s3_file(bucket_name, key, dest)#
Download a file from S3, with progress bar
- pyinaturalist_convert.download.estimate_next_release(remote_mtime, release_interval)#
Get estimated time until the next update
- pyinaturalist_convert.download.get_progress(**kwargs)#
Default progress bar format
- Return type:
Progress
- pyinaturalist_convert.download.get_progress_dl(**kwargs)#
Track progress of processing a file in bytes
- Return type:
Progress
- pyinaturalist_convert.download.get_progress_spinner(description='Loading')#
Get a spinner-type progress bar (for tasks that can’t be estimated)
- Return type:
Progress
- pyinaturalist_convert.download.untar_progress(archive_path, dest_dir)#
Extract a tar file with progress
- pyinaturalist_convert.download.unzip_progress(archive_path, dest_dir)#
Extract a zip file with progress