About the Stata plugin


#1

Hi all,

Love this initiative! I have some experience with Stata so I would like to offer my help in developing the API plugin.

As it looks the two straightforward entry points would be the JSON API (I read somewhere that it should be already active or soon to be), on the basis of the Stata libjson library and the insheetjson command by Erik Lindsley. Alternatively, it may be interesting to integrate the API with the existing command sdmxuse from Sébastien Fontenay, which already allows to import data from WB, IMF, OECD, Eurostat (and via tweaking other services sharing the same API specs, e.g. the Belgian National Bank).

The first solution may be more elegant as sdmxuse uses a lot of shell calls which tend to make import process rather “flashy”. Another important aspect to consider is whether to allow insecure requests: only the two latest versions of Stata (14 & 15) support the https protocol, but I reckon earlier versions are still widely used.

Keep up the good work, can’t wait to see all the new sources you are planning to include up and working.

Thanks,
Simone


#2

Hello Simone,

Love this initiative! I have some experience with Stata so I would like to offer my help in developing the API plugin.

Thank you very much for your interest in DB.nomics. Your message pleased our team!

I’m the author of the Python client for DB.nomics which you can find here: https://git.nomics.world/dbnomics/dbnomics-python-client
As stated in the README, you can open it in a dynamically generated environment by opening this link.

I don’t know Stata at all and can’t give you any hint about which library to use.

You can look at the source code of dbnomics-python-client, this Python file in particular, and reproduce the main ideas.

It calls the Web API, especially the /dataframe endpoint, which is documented here, and its source code is here. It transforms the response JSON into a Pandas DataFrame. I think you can do an equivalent transformation, but targeting a Stata tabular data structure.

Another important aspect to consider is whether to allow insecure requests: only the two latest versions of Stata (14 & 15) support the https protocol, but I reckon earlier versions are still widely used.

We did not state yet about allowing http:// requests, but please let us know when you have the actual problem.

Best regards,

Christophe


#3

Great! Thanks a lot for the tips, I’ll get back here as soon as I can put together a workable version based on your python client.

Simone


#4

Hi!
Just to keep you informed that we push a new version of the python client for DBnomics, with more features : https://git.nomics.world/dbnomics/dbnomics-python-client/tree/master. I hope it can help. We are currently adapting the R client.


#5

Hi Thomas,

Thanks a lot, I had a look at the new apidocs. I have been on the Stata plugin on and off in the past weeks, I hope to push an initial release for testing soon. I’m not looking for a full port from python or R just yet but something that would look more native to the way Stata works (I might change my opinion on this in the future).

I noticed there is a {series_code}.csv endpoint now, that will make it much easier to integrate in Stata since its handling of JSON objects is rather basic without external dependencies (so far I am working only with the libjson library).

More in general, do you have a roadmap for future updates? I’d be happy to know and plan ahead the addition of new features.

Best,
Simone


#6

Hi Simone,
It will be possible very soon (in February) to download several series in csv format (for one given frequency), so maybe it will easier for you to develop your Stata plugin. Moreover, we plan to put in production the new version of DBnomics (https://next.nomics.world/) in March.
If you want to know more about our discussions and roadmap (and participate!), you can have a look here : https://git.nomics.world/dbnomics-fetchers/management/boards.


#7

Hi Thomas,

Many thanks for the update. I’m almost done on my side, I built routines for almost all endpoints of the API. It would be nice at some point to allow to extract series using masks (e.g. FRA+BEL.3+99…0.0.PIGOT), which is something that can be done now but requires the user to know dimension names in advance and is thus a bit more challenging to automate.

By the way, querying series at the moment seems only to work with the /{provider_code}/{dataset_code}, despite being allowed also for /series/{provider_code}/{dataset_code} (this option does not seem to work at the moment as it always returns all series allowed by the limit option).

I’ll be back soon with a first prototype version.
Best,

Simone


#8

Hi Simone,

We will implement masks as your example very soon. However, such masks are similar to SDMX requests and could not be extended to all the providers in DBnomics. Moreover, we will provide on the website a direct access to a dictionary of the codes and labels of dimensions for each dataset, to use more easily the API.

Best,
Thomas


#9

Hi Simone,

These 2 endpoints of the API could be merged at some point. The big picture is that the first responds a list of series (codes and names only) and how much series correspond to each dimension (called facets counts in Apache Solr indexer), and the second responds series with their observations.

The clients call the second one to build the dataframes. I didn’t understand your problem with the limit param. Perhaps you missed the offset param, which you can use to fetch the next “pages”.

Have a look at this code section in the Python client: it loops until all the pages are downloaded, and increments the offset param accordingly.

Hope it helps.

Regards


#10

Hi Christophe, Thomas,

It took me a while, but I am happy to say that the first release of the Stata plugin is ready! Have a look at https://dreameater89.github.io/dbnomics/.

Aside from de-bugging, as a next step it would be great in the future to try and write an article about the command (and the platform) to publish on the Stata Journal, perhaps present it at future Stata conferences. It would definitely help promoting the platform as well as the tool itself. I understand you may prefer to reach a certain milestone version before thinking about that, so I’m happy discuss more in the future. In the meantime, let me know if you spot any bug or would like to see things changed.

Best,
Simone


#11

Hi Simone,
Great news! I will use it and go back to you soon (I have to install Stata before).
Definitely it could be very useful to present it at conferences. We expect to have a version to do that very soon (end of March).
Thanks again for your contribution.
Thomas


#12

Hello everyone, Any news from this wonderful project? I just tried DBnomics (v1.0.2) but got some error messages. For example:

dbnomics providers, clear
Downloading list of providers…
jsoncolsArray(): 3010 attempt to dereference NULL pointer
json2table(): - function returned error
: - function returned error

Did I miss anything?


#13

Hi Nathalex,

The reason why you get such error (pretty much on any dbnomics subcommand) is because I last updated the routine on version 0.18.0 of the API. I see now the API got updated to version 0.21.5. Likely there are some structural changes to the API response which I am not parsing correctly.

Haven’t had much time to keep the version update in the last weeks and months, but I will try to do so in the next days. Keep following this thread and the github page https://github.com/dreameater89/dbnomics for new updates!

Best,
Simone


#14

Hi Nathalex,

The stata command was just updated to work with version 0.21.5 of the API. Could you please update (net install dbnomics, from("https://dreameater89.github.io/dbnomics") replace) and let me know if the updated version works? I plan a couple more improvements in the next weeks, this was just a temporary patch to make things work with the new version of the API.

Best,
Simone


#15

Great news Simone, thank you very much for your contribution!


#16

Hi @signorsignore,

Thanks for your work!

I just wanted to inform you that we updated the DBnomics web-site to add a link to the Stata client in the footer.


#17

Hi all,

Thank you for your great work in the past months, in particular for having pushed version 22 of the API. I finally have some spare time to get back to this project and update the Stata routine. I hope to be able to push a new version by the end of the month.

In the meantime, I have been proposed to present the Stata command to the upcoming 25th UK Stata Conference in London (September 2019), which I think would be a great opportunity to showcase not only my API client but the overall project.

However, I wanted to check with you first to get your consent (I could work independently towards a draft presentation and share it with you first, if interested). Let me know if this would be interesting for you.

Best,
Simone


#18

Hi Simone,
Thank you for your message! Your initiative is really helpful for us. I attach a basic presentation we’ve made, if it can help (DBnomics_presentation.pdf (292.1 KB)).
Best,
Thomas