8000 Can not list/retrieve specific request with keyword "timespan" · Issue #163 · ecmwf/fdb · GitHub
[go: up one dir, main page]
Skip to content

Can not list/retrieve specific request with keyword "timespan" #163

New issue
New issue
Open
Open
Can not list/retrieve specific request with keyword "timespan"#163
Labels
bugSomething isn't working

Description

@pgeier
pgeier
opened on Aug 21, 2025
8000

What happened?

I am testing encoding output from ERA6 with the additional keyword timespan from metkit feature/mtg2_merge. I have output in a local FDB (/lus/h2resw01/scratch/mapg/5220/mapg-20250721-FORTRAN-20250820-093123-68e8e8e5f5793386b953a291b9cab93d-cmp_dev/fdbs/fdb/etc/fdb/config.yaml on ATOS).

I'm not able to retrieve list/retrieve specific fields if the keyword "timespan" is contained.
Without timespan I can list and retrieve, but the DataRetriever returns me only an empty bytearray() and not the whole grib message.
(I'm interested in the grib message, not the contained values because we do not store any values for testing metadata encoding).

What are the steps to reproduce the bug?

With pyfdb I am listing up all fields with fdb.list(keys=True).
For inspection and comparison I am picking an entry to perform a retrieve:

fdb.retrieve({'param': '162112',
 'levtype': 'ml',
 'date': '20220531',
 'expver': '5220',
 'time': '1800',
 'timespan': 'fs',
 'type': 'fc',
 'class': 'e6',
 'domain': 'g',
 'levelist': '115',
 'year': '2022',
 'step': '2',
 'stream': 'lwda',
 'month': '5',
 'anoffset': '3'})

This gives me an very unexpected error due to an failing ASSERT from fdb (see Relevant LOG Output)

Taking the same request in fdb.list(entry) returns an empty list (although this entry has been returned by fdb before).
Removing timespan from the entry solves the issue and lists the field again:

[{'path': '/lus/h2resw01/scratch/mapg/ERA6-LWDA/mapg-20250721-CPP-20250820-105456-481e7474ac647ce014657e7c08618b22-cmp_dev/fdbs/fdb01/root/e6:5220:lwda:2022:g/5:fc:ml:3:fs.20250820.105851.ad6-198.bullx.13080662202384405.data',
  'offset': 10895664,
  'length': 2964}]

However with the DataRetriever return from d = fdb.retrieve(entry) I'm only getting on empty bytearray(b''). Also d.size() returns 0, although the list showed a length of 2964.

Version

develop

Platform (OS and architecture)

ATOS

Relevant log output

Unexpected error on fdb.retrieve(entry):

Assertion failed: ax (/etc/ecmwf/nfs/dh2_perm_a/mapg/multio-dev/source/fdb5/src/fdb5/types/TypeStep.cc:75 getValues)
backtrace [9] stack has 50 addresses
(/perm/mapg/multio-dev/install/lib/../lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x1af
(/perm/mapg/multio-dev/install/lib/../lib/libeckit.so+eckit::handle_assert(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, eckit::CodeLocation const&))0x5bd
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::TypeStep::getValues(metkit::mars::MarsRequest const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string
<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, fdb5::Notifier const&, fdb5::CatalogueReader const*) const)0x17a
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::MultiRetrieveVisitor::values(metkit::mars::MarsRequest const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, fdb5::TypesRegistry const&, s
td::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&))0x6d
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Rule::findMatchingKeys(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&) const)0xa5
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::RuleDatum::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&, fdb5::Key&) const)0x24
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::RuleIndex::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&, fdb5::Key&) const)0xf9
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::RuleDatabase::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&) const)0x81
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Schema::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&) const)0x48
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Inspector::inspect(metkit::mars::MarsRequest const&, fdb5::Schema const&, fdb5::Notifier const&) const)0xaa
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Inspector::inspect(metkit::mars::MarsRequest const&) const)0x39
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::FDB::retrieve(metkit::mars::MarsRequest const&))0x12e

Accompanying data

/lus/h2resw01/scratch/mapg/5220/mapg-20250721-FORTRAN-20250820-093123-68e8e8e5f5793386b953a291b9cab93d-cmp_dev/fdbs/fdb/etc/fdb/config.yaml

Organisation

ECMWF

Activity

  1. pgeier
    added
    bugSomething isn't working
    on Aug 21, 2025

  2. caraghbiner commented on Aug 21, 2025

    @caraghbiner
    caraghbiner
    on Aug 21, 2025
    Member

    Hi @pgeier , I'll try to reproduce

  3. caraghbiner commented on Aug 21, 2025

    @caraghbiner
    caraghbiner
    on Aug 21, 2025
    Member

    Hi @pgeier , I think this might be because metkit is expanding fs to from-start on retrieve, but not on archive (as you pointed out in your other issue). I'd be curious to know if you can retrieve timespan=1h from your FDB.

    I have noticed using your config file I can list 1h:

    fdb-list param=162112,levtype=ml,date=20220531,expver=5220,time=1800,class=e6,timespan=1h
Listing for request
list,
        class=e6,
        levtype=ml,
        date=20220531,
        time=1800,
        expver=5220,
        param=162112,
        timespan=1h


{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=100,param=162112}
{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=101,param=162112}
{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=102,param=162112}
{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=103,param=162112}
...

    but not fs

    $: /ec/res4/scratch/maby/fdb_github_163/bin/fdb-list param=162112,levtype=ml,date=20220531,expver=5220,time=1800,class=e6,timespan=fs
Listing for request
list,
        class=e6,
        levtype=ml,
        date=20220531,
        time=1800,
        expver=5220,
        param=162112,
        timespan=from-start <---!!!!

  4. pgeier commented on Aug 21, 2025

    @pgeier
    pgeier
    on Aug 21, 2025
    ContributorAuthor

    Well, that strongly relates to #161. If expand and validation had happened the error would not be here?

  5. caraghbiner commented on Aug 21, 2025

    @caraghbiner
    caraghbiner
    on Aug 21, 2025
    Member

    @pgeier and I discussed on teams.

    In summary the issue arises because metkit is assuming from-start to be the canonical form of fs, whereas eccodes seems to use fs. This means the metadata extracted from eckit::message on archive uses timespan=fs. FDB assumes eckit will give it the canonical representation.

    Updating the metkit language.yaml fixes the issue:

        before: [from-start, fs,fromstart]
    after:  [fs, from-start, fromstart]

    But it is a data governance question whether or not this is the correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions
      0