Download¶

pyinaturalist_convert.download

Helper utilities for downloading and extracting files, with progress bars

class pyinaturalist_convert.download.CSVProgress(*filenames, **kwargs)¶

Bases: MultiProgress

Track progress of processing CSV files

class pyinaturalist_convert.download.FlatTarFile(name=None, mode='r', fileobj=None, format=None, tarinfo=None, dereference=None, ignore_zeros=None, encoding=None, errors='surrogateescape', pax_headers=None, debug=None, errorlevel=None, copybufsize=None)¶

Bases: TarFile

Extracts all archive contents to a flat base directory, ignoring archive subdirectories

extract(member, path='', **kwargs)¶

Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. member’ may be a filename or a TarInfo object. You can specify a different directory using `path’. File attributes (owner, mtime, mode) are set unless `set_attrs’ is False. If `numeric_owner is True, only the numbers for user/group names are used and not the names.

The filter function will be called before extraction. It can return a changed TarInfo or None to skip the member. String names of common filters are accepted.

class pyinaturalist_convert.download.JobProgress(progress, task, task_description=None)¶

Bases: object

is_complete()¶

Return type:: bool

progress¶

task¶

task_description¶

class pyinaturalist_convert.download.MultiProgress(totals, total_progress=None, job_progress=None, task_description='Loading')¶

Bases: object

Track progress of multiple processes run in serial, plus overall combined progress

advance(advance=1)¶

start_job(name)¶

class pyinaturalist_convert.download.ParallelMultiProgress(total=0, total_progress=None)¶

Bases: object

Track progress of multiple processes run in parallel, plus overall combined progress

advance(name, advance=1)¶

property job_names¶

Return type:: Iterable[str]

log(msg)¶

log_job(msg, task_description, name, task_size=-1)¶

refresh()¶: Recreate table with current jobs

start_job(name, total, task_description='Loading')¶

stop_job(name)¶

class pyinaturalist_convert.download.ProgressIO(path, *args, callback=None, description='Extracting', **kwargs)¶

Bases: FileIO

File object wrapper that updates progress on read and write.

Parameters:

path – Path to file
callback (Optional[Callable]) – Progress callback function; a new one will be created if not provided
description (str) – Description of the task being performed

read(size)¶

Read at most size bytes, returned as bytes.

Only makes one system call, so less data may be returned than requested. In non-blocking mode, returns None if no data is available. Return an empty bytes object at EOF.

write(b)¶

Write buffer b to file, return number of bytes written.

Only makes one system call, so not all of the data may be written. The number of bytes actually written is returned. In non-blocking mode, returns None if the write would block.

class pyinaturalist_convert.download.ZipProgress(archive, **kwargs)¶

Bases: MultiProgress

Track progress of extracting files from a zip archive

pyinaturalist_convert.download.check_download(dest_file, url=None, bucket=None, key=None, release_interval=None)¶

Check if a locally downloaded file exists and is up to date

Return type:: bool

pyinaturalist_convert.download.download_file(url, dest_file)¶: Download a file from a URL, with progress bar

pyinaturalist_convert.download.download_s3_file(bucket_name, key, dest)¶: Download a file from S3, with progress bar

pyinaturalist_convert.download.estimate_next_release(remote_mtime, release_interval)¶: Get estimated time until the next update

pyinaturalist_convert.download.get_progress(**kwargs)¶

Default progress bar format

Return type:: Progress

pyinaturalist_convert.download.get_progress_dl(**kwargs)¶

Track progress of processing a file in bytes

Return type:: Progress

pyinaturalist_convert.download.get_progress_spinner(description='Loading')¶

Get a spinner-type progress bar (for tasks that can’t be estimated)

Return type:: Progress

pyinaturalist_convert.download.untar_progress(archive_path, dest_dir)¶: Extract a tar file with progress

pyinaturalist_convert.download.unzip_progress(archive_path, dest_dir)¶: Extract a zip file with progress