Pulling JSON from a URL Data value in Rust.

Pulling JSON from a URL data value

Rust is definitely a much more difficult language than I initially thought. I didn’t expect to run into a huge difference between pulling a key that already had a displayable value like the language key versus going to that key’s data value that’s a url and grabbing the JSON from there.

If that was confusing, basically, the Github API for grabbing starred repositories has a lot of keys in which the value is simple - you can just display it:

Example of Starred repo JSON

description: "Resumes generated using the GitHub informations"  
fork: false  
url: "https://api.github.com/repos/resume/resume.github.com"  
forks_url: "https://api.github.com/repos/resume/resume.github.com/forks"  
keys_url: "https://api.github.com/repos/resume/resume.github.com/keys{/key_id}"  
....etc

The description and fork keys have easy to grab values, for example.

But it also has a lot of keys in which the value is a URL such as url, forks_url, keys_url and other keys. If you go to that URL, you get an unstructured key-value JSON object:

JSON displayed by going to a URL value expressed in the Starred repo

JavaScript: 19277  
HTML: 15229  
CSS: 7329  
Ruby: 179

supernova doesn’t know what languages the repo will have because that’s all dependant on the starred repo in question. And we can’t predict what languages there will be, so we can’t use keys like in the previous JSON structure.


Rust really tries to keep you safe and from doing dangerous things on a systems level, which from my C/C++ background is quite difficult to adjust to. I like doing potentially ridiculous things with my code and then fixing it before anyone has seen what I’m doing. rustc, Rust’s compiler, felt all of a sudden like a suddenly talkative (and critical) acquaintance hovering over my shoulder.

I’m very used to just playing around aimlessly and enjoying the Eureka moment from realizing how to go about a problem. Rust is not one of those languages. Granted, I had to get this work done for the end of my semester and I was in a rush, but this is a language that requires a lot of care and respect. You need to take that Rust Programming Language book lying on your bed and read it front to back.

On top of that, this is the first time I’m working with pulling information from an API. There’s quite a bit to learn in terms of how deserialization and serialization works, how to properly format JSON objects, how to use them in a meaningful way and do all of this in a language I’ve never worked with, with concepts I’ve never experienced before. So unsurprisingly I’ve had quite a few moments of frustration.

I do think this is worth writing about, I feel that I’ve learned quite a bit from this currently not working pull request.

Issue:

Using the Github API to grab programming language information on the starred repositories.

Solution (But mostly side issues):

This is my first ever Work In Progress [WIP] pull request. It’s disheartening to put up work that isn’t working. However, it is actually really nice to get detailed feedback from others, especially if you’re unable to find solutions online or get feedback via IRC. Getting properly directed and seeing what the maintainers are thinking in terms of design especially on such a young project is great.

Challenges:

  • Using the existing url data and trying to grab JSON from it
    • Some things to note: keeping Repository struct in scope and using that data value in the first place
    • Do this everytime a starred repo is pulled and repopulates the Repository struct: Every repo has it’s own languages_url with its own data from that url (this isn’t very good in terms of performance but at the moment this is all we can do)
    • Understanding the Serde framework and its related Rust ideas (so that the compiler can be satisfied)

Most likely this will get even more complicated because logical progression is to allow the user to request this data when they want to via argument flags.

Also, the information passed in the value url JSON formats the amount of a programming language in bytes. Meaning there needs to be work to convert it into something more coherent.

As well, it could be great to have future enhancements where you can type a programming language you want to use and have the starred repos utilizing those pop up. But this in particular is speculation on my part.

Benefit:

  • More useful information to the user of the tool
  • Sets up the way to pull info from data value urls for even more future use
  • Having that setup makes it easier for newcomers of supernova to contribute enhancements

Here’s the Pull Request.