Problem Retrieving 1 time series via Python API

Hello,

I’ve been using your Python API successfully for some ttime, but I’m unable to retrieve this specific dataset: https://api.db.nomics.world/v22/series?observations=1&series_ids=BEA/NIPA-T11706/A191RX-Q.

When debugging, I found that series with data are initially retrieved fine, but then in your “fetch_series_by_api_link” method (lines 233-236) it breaks in a loop where it tries to retrieve “Industry” dimension, which, I think, maybe was not populated properly for these time-series.

Could you advise me how I could retrieve these time series with Python?

Hello,

Thanks for your feedback. If I understand well, the problem is:

df = fetch_series('BEA/NIPA-T11706/A191RX-Q')

KeyError  Traceback (most recent call last)
<ipython-input-9-da4c1b56e5b7> in <module>
----> 1 df = fetch_series('BEA/NIPA-T11706/A191RX-Q')

~/dev/jailbreak/dbnomics/dbnomics-python-client/dbnomics/__init__.py in fetch_series(provider_code, dataset_code, series_code, dimensions, series_ids, max_nb_series, api_base_url, editor_api_base_url, filters)
    167             raise ValueError("When you use `series_ids`, you must not specifiy `provider_code` nor `dataset_code`.")
    168         api_link = series_base_url + '?observations=1&series_ids={}'.format(','.join(series_ids))
--> 169         return fetch_series_by_api_link(api_link, filters=filters, max_nb_series=max_nb_series,
    170                                         editor_api_base_url=editor_api_base_url)
    171 

~/dev/jailbreak/dbnomics/dbnomics-python-client/dbnomics/__init__.py in fetch_series_by_api_link(api_link, max_nb_series, editor_api_base_url, filters)
    233         if 'dimensions_values_labels' in dataset_dimensions:
    234             for dimension_code, dimension_label in dataset_dimensions_labels.items():
--> 235                 dimension_value_code = series_dims_by_dataset_code[complete_dataset_code][series['series_code']][dimension_code]
    236                 flat_series[dimension_label] = dict(dataset_dimensions['dimensions_values_labels']
    237                                                     [dimension_code])[dimension_value_code]

KeyError: 'industry'

I created an issue in our Gitlab to treat this. I can’t tell when we’ll plan to treat this. Probably not this week unfortunately.

But, as DBnomics is open software, if you identify the bug (and apparently you did well :+1:) and find a way to solve this one, we’re very open to pull requests ! You can communicate with us here to ask questions if you have some.

Have a good day

Hi @bruno.duye ,

Thank you very much for your quick reply.

Yes, I believe this is the problem, but I’m not sure yet how it could be resolved (DBNomics is a new thing to me, I need to spend some time understanding how it works). But I will think about it and will let you know if I get any ideas.

In the meantime, I think I found a lot more examples that fail in the same for loop that we identified above. Not sure if it helps to resolve the issue, but I’ll summarise my findings below just in case.

  1. I cannot retrieve any except Total index (i.e. FED/G17_IP_MARKET_GROUPS/IP.B50001.S) time series from this data group: https://db.nomics.world/FED/G17_IP_MARKET_GROUPS?dimensions={"FREQ"%3A["129"]%2C"SA"%3A["SA"]}

  2. I cannot retrieve some time series in this data group: https://db.nomics.world/FED/G17_CAPUTL?dimensions={"FREQ"%3A["129"]%2C"SA"%3A["SA"]}. Examples:
    – FED/G17_CAPUTL/CAPUTL.G211.S
    – FED/G17_CAPUTL/CAPUTL.G212.S
    – FED/G17_CAPUTL/CAPUTL.G2122.S
    – FED/G17_CAPUTL/CAPUTL.G2123.S
    – FED/G17_CAPUTL/CAPUTL.G213.S
    – FED/G17_CAPUTL/CAPUTL.G311.S
    – FED/G17_CAPUTL/CAPUTL.G312.S
    – FED/G17_CAPUTL/CAPUTL.G313.S
    – FED/G17_CAPUTL/CAPUTL.G314.S
    – FED/G17_CAPUTL/CAPUTL.G315.S
    – FED/G17_CAPUTL/CAPUTL.G316.S
    – FED/G17_CAPUTL/CAPUTL.G325212.S
    – FED/G17_CAPUTL/CAPUTL.G32522.S
    – FED/G17_CAPUTL/CAPUTL.G3311A2.S
    – FED/G17_CAPUTL/CAPUTL.G3341.S
    – FED/G17_CAPUTL/CAPUTL.G3342.S
    – FED/G17_CAPUTL/CAPUTL.G3344.S
    – FED/G17_CAPUTL/CAPUTL.G336.S
    – FED/G17_CAPUTL/CAPUTL.G33611.S
    – FED/G17_CAPUTL/CAPUTL.HITEK2.S
    – FED/G17_CAPUTL/CAPUTL.HITEK2.S
    – FED/G17_CAPUTL/CAPUTL.N325211.S
    – FED/G17_CAPUTL/CAPUTL.X4HTK2.S
    – FED/G17_CAPUTL/CAPUTL.X4HTMV.S
    – FED/G17_CAPUTL/CAPUTL.X50HTK.S

  3. I cannot retrieve many (maybe all, haven’t tested all) from this data group: Consumer Credit Outstanding [G19_CCOUT], examples:

  • FED/G19_CCOUT/DTCTL.M
  • FED/G19_CCOUT/DTCTLN.M
  • FED/G19_CCOUT/DTCTLR.M

I think all of these are related to the same for loop issue.

Thanks a lot for looking into this and your work developing this amazing database! Once I get something useful out of my economics research, I’ll make sure to share the credit :slight_smile:

Regards,
Aiste

Hi Aiste,

I can explain your FED problem, which is different from the BEA problem. We had to modify a bit the series code of FED series, sorry (the Fed gave sometimes more than ones series even when we specified a value for each dimension).

Now you can download all the series if you use the mask, as explained here (https://git.nomics.world/dbnomics/dbnomics-python-client/blob/master/index.ipynb, Fetch time series by code mask). In your example 1, you should write : fetch_series("FED", "G17_IP_MARKET_GROUPS", series_code="129..SA") and you will obtain the 63 different series, as in the DBnomics website.

Equivalently, you could use the Fetch time series by dimension :
fetch_series("FED", "G17_IP_MARKET_GROUPS", dimensions={ "FREQ": ["129"], "SA": ["SA"], })
and you will obtain the 63 series.

Thank you for your interest in DBnomics!
Thomas

Not sure if this is the same issue, but some things are kind of weird in the FED/G17_CAPUTL area. For example the series CAPUTL.B50001.S. If you download this (JSON) with

complete_missing_periods=1

you get 53 valid observations (out of 637 periods). The data are supposed to be monthly, but these values seem to pertain to the 28th of February in each year, with other months’ data all missing. But if you use

complete_missing_periods=0

you get 637 (apparently) non-missing observations. This is not easy to understand!

Hi @thomasbrand,

Thank you very much! I will be looking into this again this week. I’ll try your workaround and will you know if it works.

Many thanks,
Aiste