Download#

pyinaturalist_convert.download

Helper utilities for downloading and extracting files, with progress bars

class pyinaturalist_convert.download.CSVProgress(*filenames, **kwargs)#

Bases: MultiProgress

Track progress of processing CSV files

class pyinaturalist_convert.download.FlatTarFile(name=None, mode='r', fileobj=None, format=None, tarinfo=None, dereference=None, ignore_zeros=None, encoding=None, errors='surrogateescape', pax_headers=None, debug=None, errorlevel=None, copybufsize=None)#

Bases: TarFile

Extracts all archive contents to a flat base directory, ignoring archive subdirectories

extract(member, path='', **kwargs)#: Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. member’ may be a filename or a TarInfo object. You can specify a different directory using `path’. File attributes (owner, mtime, mode) are set unless `set_attrs’ is False. If `numeric_owner is True, only the numbers for user/group names are used and not the names.

class pyinaturalist_convert.download.JobProgress(progress, task, task_description=None)#

Bases: object

is_complete()#

Return type:: bool

progress#

task#

task_description#

class pyinaturalist_convert.download.MultiProgress(totals, total_progress=None, job_progress=None, task_description='Loading')#

Bases: object

Track progress of multiple processes run in serial, plus overall combined progress

advance(advance=1)#

start_job(name)#

class pyinaturalist_convert.download.ParallelMultiProgress(total=0, total_progress=None)#

Bases: object

Track progress of multiple processes run in parallel, plus overall combined progress

advance(name, advance=1)#

property job_names#

log(msg)#

log_job(msg, task_description, name, task_size=-1)#

refresh()#: Recreate table with current jobs

start_job(name, total, task_description='Loading')#

stop_job(name)#

class pyinaturalist_convert.download.ProgressIO(path, *args, callback=None, description='Extracting', **kwargs)#

Bases: FileIO

File object wrapper that updates progress on read and write.

Parameters:

path – Path to file
callback (Optional[Callable]) – Progress callback function; a new one will be created if not provided
description (str) – Description of the task being performed

read(size)#

Read at most size bytes, returned as bytes.

Only makes one system call, so less data may be returned than requested. In non-blocking mode, returns None if no data is available. Return an empty bytes object at EOF.

write(b)#

Write buffer b to file, return number of bytes written.

Only makes one system call, so not all of the data may be written. The number of bytes actually written is returned. In non-blocking mode, returns None if the write would block.

class pyinaturalist_convert.download.ZipProgress(archive, **kwargs)#

Bases: MultiProgress

Track progress of extracting files from a zip archive

pyinaturalist_convert.download.check_download(dest_file, url=None, bucket=None, key=None, release_interval=None)#

Check if a locally downloaded file exists and is up to date

Return type:: bool

pyinaturalist_convert.download.download_file(url, dest_file)#: Download a file from a URL, with progress bar

pyinaturalist_convert.download.download_s3_file(bucket_name, key, dest)#: Download a file from S3, with progress bar

pyinaturalist_convert.download.estimate_next_release(remote_mtime, release_interval)#: Get estimated time until the next update

pyinaturalist_convert.download.get_progress(**kwargs)#

Default progress bar format

Return type:: Progress

pyinaturalist_convert.download.get_progress_dl(**kwargs)#

Track progress of processing a file in bytes

Return type:: Progress

pyinaturalist_convert.download.get_progress_spinner(description='Loading')#

Get a spinner-type progress bar (for tasks that can’t be estimated)

Return type:: Progress

pyinaturalist_convert.download.untar_progress(archive_path, dest_dir)#: Extract a tar file with progress

pyinaturalist_convert.download.unzip_progress(archive_path, dest_dir)#: Extract a zip file with progress