RadioShop
RadioShop is a company that supplies background music to shops and businesses.
To make music management easier, the two of us had to put a Big Data architecture into production and build a REST API in Node.js.
Project presentation
(10 minutes)
After introducing us to the RadioShop team, my supervisor explained that I had to build an API: a program that communicates with, processes or modifies the data stored in the database.
To get there properly, my supervisor had me write small programs that would later be merged into the API.
Listening to and downloading music
(5 days)
After teaching myself Node.js (the open platform for running JavaScript), my supervisor gave me a music playback program a previous developer had written. I set out to understand how it worked so I could take it over and improve it.
My first program played a track from my computer in a web page as a stream: sending small chunks of the track to the page in order to save bandwidth.
Read, insert and update programs
(2 days)
The API communicates with a database, so being able to read, insert and update it is essential. As the database was MongoDB (NoSQL), I had to learn the query system it uses (JSON).
Starting all the MongoDB servers
(2 days)
The person in charge of starting the MongoDB servers was running into difficulties, so I joined in to help.
That is how I discovered replication, a group of three servers holding the same data: if the primary stops, one of the other two takes over and is elected primary. I also discovered sharding, which splits the database into chunks, one chunk on one replica set and another on a different one.
Building a first version of the API
(1 day)
The first version of the API only let us check that everything I inserted was correctly passed to MongoDB and could be read back. So it only allowed a simple insert, update or read. The API is used through an HTTP request.
Music insertion program
(7 days)
This program inserts a track (as binary) and its metadata into the database.
To do so I wrote a program that “downloads” a track from a link, keeping only its binary, then encrypts it so that MongoDB understands the value is binary, and builds a JSON document with all the track metadata. Finally the program sends the JSON document to the API to insert it.
OpLog
(Until the end of the internship)
The oplog is a MongoDB file that keeps servers in sync. It gave us a lot of trouble until the end of the internship.
My insertion program sends data very fast; the oplog, which records the latest inserts, updates and deletes, could not keep up and brought the servers down.
The API, its final version
(5 days)
To lighten my workload, a teammate had written the insertion and display of album artwork and song lyrics. On my side, I handled playback and display of mono tracks.
Once all the programs were written, I merged them into the API and built a utility that formats the lyrics correctly in the database.
The performance tests compared download times for several hundred tracks across RadioShop’s servers. Once my programs were finished, I wrote technical and user documentation for each of them.