8000 GitHub - EOFarm/CDSETool: Download data from the Copernicus Data Space Ecosystem (CDSE) · GitHub
[go: up one dir, main page]

Skip to content
  • EOFarmPublic
    forked from CDSETool/CDSETool

    About

    Download data from the Copernicus Data Space Ecosystem (CDSE)

    Resources

    Stars

    0 stars

    Watchers

    0 watching

    Forks

     
     

    Latest commit

     

    History

    372 Commits

    Folders and files

    Repository files navigation

    CDSETool

    About CDSETool

    This script downloads copernicus data from the Copernicus Data Space Ecosystem

    Quick start

    from cdsetool.query import query_features, shape_to_wkt
    from cdsetool.credentials import Credentials
    from cdsetool.download import download_features
    from cdsetool.monitor import StatusMonitor
    from datetime import date
    
    features = query_features(
        "SENTINEL-1",
        {
            "contentDateStartGe": "2020-12-20",
            "contentDateStartLe": date(2020, 12, 25),
            "processingLevel": "LEVEL1",
            "productType": "IW_GRDH_1S",
            "geometry": shape_to_wkt("path/to/shapefile.shp"),
        },
    )
    
    list(
        download_features(
            features,
            "path/to/output/
    8000
    folder/",
            {
                "concurrency": 4,
                "monitor": StatusMonitor(),
                "credentials": Credentials("username", "password"),
            },
        )
    )

    Or use the CLI:

    cdsetool query search SENTINEL-2 --search-term contentDateStartGe=2020-01-01 --search-term contentDateStartLe=2020-01-10 --search-term productType=S2MSI2A
    
    cdsetool download SENTINEL-2 PATH/TO/DIR --concurrency 4 --search-term contentDateStartGe=2020-01-01 --search-term contentDateStartLe=2020-01-10 --search-term productType=S2MSI2A

    Table of Contents

    Installation

    Install cdsetool using pip:

    pip install cdsetool==0.3.3

    Usage

    Querying features

    Querying is always done in batches, returning len(results) <= top records each time. A local buffer is filled and gradually emptied as results are yielded. When the buffer is empty, more results will be requested and the process repeated until no more results are available, or the iterator is discarded.

    Important: The API has a pagination limit of 10,000 results per query. If your query returns more results, you'll need to narrow your search criteria (e.g., use smaller date ranges).

    Since downloading features is the most common use-case, query_features assumes that the query will run till the end. Because of this, the batch size is set to 1000, which is the size limit set by CDSE.

    from cdsetool.query import query_features
    
    collection = "SENTINEL-2"
    search_terms = {
        "top": 100,  # batch size, between 1 and 1000 (default: 1000)
        "contentDateStartGe": "2024-01-01",
        "productType": "S2MSI1C"
    }
    
    # wait for a single batch to finish, yield results immediately
    for feature in query_features(collection, search_terms):
        # do something with feature
    
    # wait for all batch requests to complete, returning list
    features = list(query_features(collection, search_terms))
    
    # manually iterate
    iterator = query_features(collection, search_terms)
    
    featureA = next(iterator)
    featureB = next(iterator)
    # ...

    Query Options

    Product Format

    Query results are returned directly from the Copernicus API. Each product has the following structure:

    product = {
        "Id": "uuid-string",
        "Name": "S2A_MSIL2A_20240110T105421_...",
        "Collection": "SENTINEL-2",
        "ContentDate": {"Start": "2024-01-10T10:54:21Z", "End": "..."},
        "Online": True,
        "ContentLength": 1043654649,
        "GeoFootprint": {"type": "Polygon", "coordinates": [...]},
        "Attributes": [
            {"Name": "productType", "Value": "S2MSI2A"},
            {"Name": "cloudCover", "Value": 5.2},
            ...,
        ],
    }

    Expand Product Attributes

    By default, query results do not include product attributes (productType, cloudCover, platform, instrument, etc.). To include product attributes, you need to request this from the server using the option expand_attributes and can then access the attributes using the function get_product_attribute():

    from cdsetool.query import query_features, get_product_attribute
    
    features = query_features(
        "SENTINEL-2",
        {"contentDateStartGe": "2024-01-01"},
        options={"expand_attributes": True},
    )
    feature = features[0]
    
    # Access basic properties directly
    print(feature["Name"])  # Product name
    print(feature["Id"])  # Product UUID
    
    # Access attributes using helper function
    cloud_cover = get_product_attribute(feature, "cloudCover")
    product_type = get_product_attribute(feature, "productType")

    Querying by geometry

    To query by shapes, you must first convert your shape to Well Known Text (WKT). The included shape_to_wkt can solve this.

    from cdsetool.query import query_features, shape_to_wkt
    
    geometry = shape_to_wkt("path/to/shape.shp")
    features = query_features("SENTINEL-3", {"geometry": geometry})

    Querying by lists of parameters

    Most search terms only accept a single argument. To query by a list of arguments, loop the arguments and pass them one by one to the query function.

    from cdsetool.query import query_features
    
    tile_ids = ["32TPT", "32UPU", "32UPU", "31RFL", "37XDA"]
    
    for tile_id in tile_ids:
        features = query_features("SENTINEL-2", {"tileId": tile_id})
        for feature in features:
            # do things with feature

    Querying by dates

    Its quite common to query for features created before, after or between dates.

    Search terms support comparison operator suffixes:

    Suffix Meaning Example
    Eq equals (=) contentDateStartEq
    Gt greater than (>) contentDateStartGt
    Ge greater than or equal (>=) contentDateStartGe
    Lt less than (<) contentDateStartLt
    Le less than or equal (<=) contentDateStartLe

    Eq can be applied on any field but the other suffixes can only be applied to numeric and date fields.

    Interval syntax is only allowed on the base name, not on suffixed variants, and only on numeric and date fields.

    Interval notation Suffixes to combine Meaning
    [a, b] Ge + Le a <= value <= b (closed)
    (a, b) Gt + Lt a < value < b (open)
    [a, b) Ge + Lt a <= value < b (half-open)
    (a, b] Gt + Le a < value <= b (half-open)
    from cdsetool.query import query_features
    from datetime import date, datetime
    
    date_from = date(
        2020, 1, 1
    )  # or datetime(2020, 1, 1, 23, 59, 59, 123456) or "2020-01-01" or "2020-01-01T23:59:59.123456Z"
    date_to = date(2020, 12, 31)
    
    features = query_features(
        "SENTINEL-2",
        {
            "contentDateStartGe": date_from,
            "contentDateStartLe": date_to,
            "cloudCover": "[0, 30]",
        },
    )

    Listing search terms

    To get a list of all search terms for a given collection, you may either use the describe_collection function or use the CLI:

    from cdsetool.query import describe_collection
    
    search_terms = describe_collection("SENTINEL-2").keys()
    print(search_terms)

    And the CLI:

    $ cdsetool query search-terms SENTINEL-2

    Downloading features

    Authenticating

    An account is required to download features from the Copernicus distribution service.

    To authenticate using an account, instantiate Credentials and pass your username and password

    from cdsetool.credentials import Credentials
    
    username = "konata@izumi.com"
    password = "password123"
    credentials = Credentials(username, password)

    Alternatively, Credentials can pull from ~/.netrc when username and password are left blank.

    # ~/.netrc
    machine https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token
    login konata@izumi.com
    password password123
    
    # main.py
    from cdsetool.credentials import Credentials
    
    credentials = Credentials()

    The credentials object may then be passed to a download function. If left out, the download functions will default to using .netrc.

    credentials = Credentials()
    
    download_features(features, "/some/download/path", {"credentials": credentials})

    Credentials can be validated using the validate_credentials function which will return a boolean.

    from cdsetool.credentials import validate_credentials
    
    validate_credentials(username="user", password="password")

    If None are passed to username and password, validate_credentials will validate .netrc

    Concurrently downloading features

    CDSETool provides a method for concurrently downloading features. The concurrency level should match your accounts privileges. See CDSE quotas

    The downloaded feature ids are yielded, so its required to await the results.

    from cdsetool.query import query_features
    from cdsetool.download import download_features
    
    features = query_features(
        "SENTINEL-2",
        {"contentDateStartGe": "2024-01-01", "contentDateStartLe": "2024-01-10"},
    )
    
    download_path = "/path/to/download/folder"
    downloads = download_features(features, download_path, {"concurrency": 4})
    
    for id in downloads:
        print(f"feature {id} downloaded")
    
    # or
    
    list(downloads)

    Sequentially downloading features

    Its possible to download features sequentially in a single thread if desired.

    from cdsetool.query import query_features
    from cdsetool.download import download_feature
    
    features = query_features(
        "SENTINEL-2",
        {"contentDateStartGe": "2024-01-01", "contentDateStartLe": "2024-01-10"},
    )
    
    download_path = "/path/to/download/folder"
    for feature in features:
        download_feature(feature, download_path)

    Download specific files within features

    It's possible to download specific files within products bundles using Unix filename pattern matching.

    It can be used in CDSETool:

    • Through the filter_pattern option of download_features and download_feature:

      from cdsetool.query import query_features
      from cdsetool.download import download_features
      
      features = query_features(
          "SENTINEL-2",
          {"contentDateStartGe": "2024-01-01", "contentDateStartLe": "2024-01-10"},
      )
      
      download_path = "/path/to/download/folder"
      filter_pattern = "*TCI.jp2"
      downloads = download_features(
          features, download_path, {"filter_pattern": filter_pattern}
      )
      
      for id in downloads:
          print(f"feature {id} downloaded")
    • Or through the CLI:

      cdsetool download SENTINEL-2 PATH/TO/DIR --filter-pattern *TCI.jp2 --concurrency 4 --search-term contentDateStartGe=2024-01-01 --search-term contentDateStartLe=2024-01-10 --search-term productType=S2MSI2A
      

    Roadmap

    • Query schema validation
    • High-level API
      • Query features
      • Download features
        • Download single feature
        • Download list of features
        • Download by ID
        • Download by URL
    • Command-Line Interface
      • Update to match the high-level API
      • Better --help messages
      • Quickstart guide in README.md
    • Test suite
      • Query
      • Credentials
      • Download
      • Monitor
    • Strategy for handling HTTP and connection errors

    Contributing

    Any contributions you make are greatly appreciated.

    If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".

    1. Fork the Project
    2. Create your Feature Branch (git checkout -b feature/cool-new-feature)
    3. Commit your Changes (git commit -m 'Add some feature')
    4. Push to the Branch (git push origin feature/cool-new-feature)
    5. Open a Pull Request

    LICENSE

    Distributed under the MIT License. See LICENSE for more information.

    About

    Download data from the Copernicus Data Space Ecosystem (CDSE)

    Resources

    Stars

    0 stars

    Watchers

    0 watching

    Forks

    Releases

    Packages

    Contributors

    Languages

    0