o a&i^@sdZddlZddlZddlZddlmZzddlmZWn ey'dZYnwzddlZWn ey9dZYnwdddZ Gd d d Z Gd d d Z Gd ddZ GdddZ ddZddZddZdS)z1 The classes that actually handle the downloads. N) parse_url)tqdmFcCs^tttttd}t|}|d|vr$td|dd|d|d||d|d}|S)a Choose the appropriate downloader for the given URL based on the protocol. Parameters ---------- url : str A URL (including protocol). progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. Returns ------- downloader A downloader class, like :class:`pooch.HTTPDownloader`, :class:`pooch.FTPDownloader`, or :class: `pooch.SFTPDownloader`. Examples -------- >>> downloader = choose_downloader("http://something.com") >>> print(downloader.__class__.__name__) HTTPDownloader >>> downloader = choose_downloader("https://something.com") >>> print(downloader.__class__.__name__) HTTPDownloader >>> downloader = choose_downloader("ftp://something.com") >>> print(downloader.__class__.__name__) FTPDownloader >>> downloader = choose_downloader("doi:DOI/filename.csv") >>> print(downloader.__class__.__name__) DOIDownloader )ftphttpshttpsftpdoiprotocolzUnrecognized URL protocol 'z' in ''. Must be one of .) progressbar) FTPDownloaderHTTPDownloaderSFTPDownloader DOIDownloaderr ValueErrorkeys)urlr Zknown_downloaders parsed_url downloaderr`__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. chunk_size : int Files are streamed *chunk_size* bytes at a time instead of loading everything into memory at one. Usually doesn't need to be changed. **kwargs All keyword arguments given when creating an instance of this class will be passed to :func:`requests.get`. Examples -------- Download one of the data files from the Pooch repository: >>> import os >>> from pooch import __version__, check_version >>> url = "https://github.com/fatiando/pooch/raw/{}/data/tiny-data.txt" >>> url = url.format(check_version(__version__, fallback="main")) >>> downloader = HTTPDownloader() >>> # Not using with Pooch.fetch so no need to pass an instance of Pooch >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> os.path.exists("tiny-data.txt") True >>> with open("tiny-data.txt") as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> os.remove("tiny-data.txt") Authentication can be handled by passing a user name and password to :func:`requests.get`. All arguments provided when creating an instance of the class are forwarded to :func:`requests.get`. We'll use ``auth=(username, password)`` to use basic HTTPS authentication. The https://httpbin.org website allows us to make a fake a login request using whatever username and password we provide to it: >>> user = "doggo" >>> password = "goodboy" >>> # httpbin will ask for the user and password we provide in the URL >>> url = f"https://httpbin.org/basic-auth/{user}/{password}" >>> # Trying without the login credentials causes an error >>> downloader = HTTPDownloader() >>> try: ... downloader(url=url, output_file="tiny-data.txt", pooch=None) ... except Exception: ... print("There was an error!") There was an error! >>> # Pass in the credentials to HTTPDownloader >>> downloader = HTTPDownloader(auth=(user, password)) >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> with open("tiny-data.txt") as f: ... for line in f: ... print(line.rstrip()) { "authenticated": true, "user": "doggo" } >>> os.remove("tiny-data.txt") FcKs4||_||_||_|jdurtdurtddSdSNT2Missing package 'tqdm' required for progress bars.)kwargsr chunk_sizerrselfr rrrrr__init__s zHTTPDownloader.__init__c Cs$|j}|ddt|d }|rt|d}zqtj|fi|}||j|j d}t |j dd}|j durMt tjdk} t|d | d ddd } n |j rV|j } || _|D]} | rn|| ||j rn| |j qX|j r| | || W|r|d Sd S|r|ww) a Download the given URL over HTTP to the given output file. Uses :func:`requests.get`. Parameters ---------- url : str The URL to the file you want to download. output_file : str or file-like object Path (and file name) to which the file will be downloaded. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. streamTwritew+b)rzcontent-lengthrwin32OBtotalZncolsasciiunitZ unit_scaleZleaveN)rcopy setdefaulthasattropenrequestsgetraise_for_status iter_contentrintheadersr boolsysplatformrr*r$flushupdateresetclose) r!r output_filepoochrispathresponsecontentr* use_asciiprogresschunkrrr__call__sP          zHTTPDownloader.__call__NFr__name__ __module__ __qualname____doc__r"rFrrrrrSs L rc@s0eZdZdZ       d dd Zd d ZdS) ra Download manager for fetching files over FTP. When called, downloads the given file URL into the specified local file. Uses the :mod:`ftplib` module to manage downloads. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to customize the download of files (for example, to use authentication or print a progress bar). Parameters ---------- port : int Port used for the FTP connection. username : str User name used to login to the server. Only needed if the server requires authentication (i.e., no anonymous FTP). password : str Password used to login to the server. Only needed if the server requires authentication (i.e., no anonymous FTP). Use the empty string to indicate no password is required. account : str Some servers also require an "account" name for authentication. timeout : int Timeout in seconds for ftp socket operations, use None to mean no timeout. progressbar : bool If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. **Custom progress bars are not yet supported.** chunk_size : int Files are streamed *chunk_size* bytes at a time instead of loading everything into memory at one. Usually doesn't need to be changed.  anonymousNFrcCsL||_||_||_||_||_||_||_|jdur"tdur$tddSdSr) portusernamepasswordaccounttimeoutr rrr)r!rPrQrRrSrTr rrrrr" s zFTPDownloader.__init__c s>t|}tj|jd}|j|d|jdtd }|r"tdzp|j|j |j |j dd|d}|j rz| d t||d}ttjd k} t|d | d d d dfdd} |j|| |jdWdn1stwYn |j|j|jdW||rdSdS||rww)a Download the given URL over FTP to the given output file. Parameters ---------- url : str The URL to the file you want to download. output_file : str or file-like object Path (and file name) to which the file will be downloaded. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. )rTnetloc)hostrPr$r%)userpasswdZacctzRETR pathzTYPE Ir&r'r(Tr)cst||dSz+Update the progress bar and write to outputN)r;lenr$)datar>rDrrcallbackHsz(FTPDownloader.__call__..callback) blocksizeN)rftplibFTPrTconnectrPr/r0loginrQrRrSr voidcmdr5sizer7r8r9rZ retrbinaryrr$quitr=) r!rr>r?rrr@commandrerCr^rr]rrF!sH      zFTPDownloader.__call__)rMrNrOrONFrrHrrrrrs& rc@s.eZdZdZ      d ddZd d ZdS) ra Download manager for fetching files over SFTP. When called, downloads the given file URL into the specified local file. Requires `paramiko `__ to be installed. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to customize the download of files (for example, to use authentication or print a progress bar). Parameters ---------- port : int Port used for the SFTP connection. username : str User name used to login to the server. Only needed if the server requires authentication (i.e., no anonymous SFTP). password : str Password used to login to the server. Only needed if the server requires authentication (i.e., no anonymous SFTP). Use the empty string to indicate no password is required. timeout : int Timeout in seconds for sftp socket operations, use None to mean no timeout. progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. rNrONFcCsh||_||_||_||_||_||_g}|jr tdur |dtdur)|d|r2t d |dS)Nrz7Missing package 'paramiko' required for SFTP downloads. ) rPrQrRrSrTr rappendparamikorjoin)r!rPrQrRrSrTr errorsrrrr"ws   zSFTPDownloader.__init__c s$t|}tj|d|jfd}d}zr|j|j|jdtj|}|j | _ |j rEt ||dj}ttjdk}t|d|dd d d |j rkfd d } |j|d|| d Wdn1sewYn||d|W||dur|dSdS||dur|ww)aQ Download the given URL over SFTP to the given output file. The output file must be given as a string (file name/path) and not an open file object! Otherwise, paramiko cannot save to that file. Parameters ---------- url : str The URL to the file you want to download. output_file : str Path (and file name) to which the file will be downloaded. **Cannot be a file object**. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. rU)sockN)rQrRrYr&r'r(Tr)cs"t|_t|jdSrZ)r5r*r;n)currentr*rDrrr^s z)SFTPDownloader.__call__..callback)r^)rrk TransportrPrbrQrRZ SFTPClientZfrom_transportrTZ get_channel settimeoutr r5statst_sizer7r8r9rr2r=) r!rr>r?r connectionrrerCr^rrqrrFsD      zSFTPDownloader.__call__)rhrNrOrONFrHrrrrrVs" rc@r) ra Download manager for fetching files from Digital Object Identifiers (DOIs). Open-access data repositories often issue Digital Object Identifiers (DOIs) for data which provide a stable link and citation point. The trick is finding out the download URL for a file given the DOI. When called, this downloader uses the repository's public API to find out the download URL from the DOI and file name. It then uses :class:`pooch.HTTPDownloader` to download the URL into the specified local file. Allowing "URL"s to be specified with the DOI instead of the actual HTTP download link. Uses the :mod:`requests` library to manage downloads and interact with the APIs. The **format of the "URL"** is: ``doi:{DOI}/{file name}``. Notice that there are no ``//`` like in HTTP/FTP and you must specify a file name after the DOI (separated by a ``/``). Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to be able to download files given the DOI instead of an HTTP link. Supported repositories: * `figshare `__ * `Zenodo `__ .. attention:: DOIs from other repositories **will not work** since we need to access their particular APIs to find the download links. We welcome suggestions and contributions adding new repositories. Parameters ---------- progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. chunk_size : int Files are streamed *chunk_size* bytes at a time instead of loading everything into memory at one. Usually doesn't need to be changed. **kwargs All keyword arguments given when creating an instance of this class will be passed to :func:`requests.get`. Examples -------- Download one of the data files from the figshare archive of Pooch test data: >>> import os >>> downloader = DOIDownloader() >>> url = "doi:10.6084/m9.figshare.14763051.v1/tiny-data.txt" >>> # Not using with Pooch.fetch so no need to pass an instance of Pooch >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> os.path.exists("tiny-data.txt") True >>> with open("tiny-data.txt") as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> os.remove("tiny-data.txt") Same thing but for our Zenodo archive: >>> url = "doi:10.5281/zenodo.4924875/tiny-data.txt" >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> os.path.exists("tiny-data.txt") True >>> with open("tiny-data.txt") as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> os.remove("tiny-data.txt") FrcKs||_||_||_dS)N)rr rr rrrr"s zDOIDownloader.__init__c Csttd}t|}|d}t|}t|d}||vr*td|dt|d||||ddd|d } td |j |j d |j } | | ||d S) a7 Download the given DOI URL over HTTP to the given output file. Uses the repository's API to determine the actual HTTP download URL from the given DOI. Uses :func:`requests.get`. Parameters ---------- url : str The URL to the file you want to download. output_file : str or file-like object Path (and file name) to which the file will be downloaded. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. )z figshare.comz zenodo.orgrUzInvalid data repository 'r zx. To request or contribute support for this repository, please open an issue at https://github.com/fatiando/pooch/issuesrY/) archive_url file_namer )r rNr) figshare_download_urlzenodo_download_urlr doi_to_urlrlistrsplitrr rr) r!rr>r? convertersrr ry repository download_urlrrrrrFs0  zDOIDownloader.__call__NrGrHrrrrrs P rcCsLtd|}|j}d|jkrdkr$n|Std|d|d|S)z Follow a DOI link to resolve the URL of the archive. Parameters ---------- doi : str The DOI of the archive. Returns ------- url : str The URL of the archive in the data repository. zhttps://doi.org/iiXzArchive with doi:z not found (see z). Is the DOI correct?)r1r2r status_coder)r rArrrrr}Hsr}cCsl|dd}td|}dd|dD}||vr,td|d|d |d ||d d }|S)  Use the API to get the download URL for a file given the archive URL. Parameters ---------- archive_url : str URL of the dataset in the repository. file_name : str The name of the file in the archive that will be downloaded. doi : str The DOI of the archive. Returns ------- download_url : str The HTTP URL that can be used to download the file. rwrxzhttps://zenodo.org/api/records/cSi|]}|d|qS)keyr.0itemrrr wz'zenodo_download_url..filesFile '' not found in data archive  (doi:).linksr!)rr1r2jsonr)ryrzr article_idarticlerrrrrr|asr|cCstd|d}|d}td|d}|dd|D}||vr8td|d |d |d ||d }|S) rz)https://api.figshare.com/v2/articles?doi=ridz%https://api.figshare.com/v2/articles/z/filescSr)namerrrrrrrz)figshare_download_url..rrrrr)r1r2rr3r)ryrzr rrrArrrrrr{s r{)F)rLr8r`r1utilsrr ImportErrorrkrrrrrr}r|r{rrrrs2     7oo