Python Music Recommender: A Practical Guide Part-3

Inspiration to continue the project

Yes, I am back with the same project but this time it's a little bit different.

When using music streaming services I often find that the recommendations simply aren't very good. Perhaps it's because the music I listen to often doesn't follow a specific theme or consistent genre. Instead, I often listen to music tied to a specific place and time in my life. Music for me unlocks the memories and feelings from that period in my life.

Unfortunately, music streaming services don't seem well-equipped to do this. After listening to certain types of music for a while you get into a valley of recommendations.

My hope is to create a playlist that speaks to your own time and place using a bit of data and some APIs.

---

Data Collection

In search of data to explore more, I found this Wikipedia collection of the Billboard Hot 100 Top-Ten singles for every year from 1958 to 2022. I collected it all into a dataset of 4,251 songs.

Here is a sample:

---

Data Exploration (Exploratory Data Analysis)

After pulling down these features, I did some exploration. Given that I had tracks from so many years, I thought it would be interesting to see how music had changed over time.

import pandas as pd
import numpy as np
import math

import matplotlib.pyplot as plt
from matplotlib import gridspec
from matplotlib.offsetbox import (OffsetImage, AnnotationBbox)

plt.style.use("df-style.mplstyle") # <-- Find this file in my github repo
df = pd.read_csv("songs.csv")

I made averages for each track feature by year and plotted them. Several things caught my attention.

year = df.groupby(by='year').mean(numeric_only=True).drop(columns='peak').reset_index()
year.head()

Output:

Feature Plotting

# Feature Plotting
N = len(year.columns[1:])
cols = 4
rows = int(math.ceil(N / cols))

gs = gridspec.GridSpec(rows, cols)
fig = plt.figure()

for n, column in zip(range(N), year.columns[1:]):
    ax = fig.add_subplot(gs[n])
    ax.plot(year['year'], year[column])
    ax.set_title(column)

fig.tight_layout()

Output:

Some clear features have changed over time:

• Loudness: The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing the relative loudness of tracks.

• Danceability (0-1): Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity.

• Energy (0-1): Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.

• Mode: Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived.

• Speechiness (0-1): Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audiobook, poetry), the closer to 1.0 the attribute value.

• Acousticness: A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.

• Duration: The duration of the track in milliseconds.

• Valence: A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).

Duration:

# Duration
fig, ax = plt.subplots()
ax.plot(year['year'], (year['duration_ms'] / 1_000) / 60)
ax.set_xlim(1958, 2026)
ax.set_ylim(0,6.5)

# Add in title and subtitle
ax.set_title("""Nobody got time for Music?""")
ax.text(x=.08, y=.86, 
        s="Average duration in minutes. Features from top ten singles, 1958-2022.", 
        transform=fig.transFigure, 
        ha='left', 
        fontsize=20, 
        alpha=.8)

Output:

First, from the mid-1970s until the early 2000s, the average track duration was over 4 minutes. Maybe attention spans are decreasing, but as far as I know, music from that time was often too long. Perhaps notably, the first iPod was introduced in October 2001.

Loudness:

# Loudness
fig, ax = plt.subplots()
ax.plot(year['year'], year['loudness'])
ax.set_xlim(1958, 2026)
ax.set_ylim(-15, -3)

# Add in title and subtitle
ax.set_title("""Hearing protection recommended""")
ax.text(x=.08, y=.86, 
        s="Average loudness in decibels (-60-0). Features from top ten singles, 1958-2022.", 
        transform=fig.transFigure, 
        ha='left', 
        fontsize=20, 
        alpha=.8)

Output:

Next, I found that loudness increased dramatically from the mid-90s and has only recently started trending down. After researching a bit, I found that there was such a thing as the loudness war.

This became such a problem that Spotify (and other music services) started normalizing the audio of tracks.

Instrumentalness & Speechiness:

# Instrumentalness & Speechiness
fig, ax = plt.subplots()
ax.plot(year['year'], year['instrumentalness'])
ax.plot(year['year'], year['speechiness'])
ax.set_xlim(1958, 2025)
ax.set_ylim(0,.32)

ax.set_yticks(np.arange(0, .35, 0.05))

# Add in title and subtitle
ax.set_title("They don't sing like they used to..")
ax.text(x=.08, y=.86, 
        s="Average features from top ten singles, 1958-2022, Scale 0-1", 
        transform=fig.transFigure, 
        ha='left', 
        fontsize=20, 
        alpha=.8)

# Label the lines directly
ax.text(x=.57, y=.17, s="""Instrumentalness""", 
        transform=fig.transFigure, ha='left', fontsize=20, alpha=.7)
ax.text(x=.72, y=.52, s="""Speechiness""", 
        transform=fig.transFigure, ha='left', fontsize=20, alpha=.7)

Output:

Spotify has other features they track, like "speechiness," which is basically a measure of how much like pure speech a track sounds like. Values above .33 contain both music and speech.

Tracks aren't reaching the point of being just speech as they are well below the .33 level. There has been a notable increase in the average speechiness of tracks. There was a similar bump in the 2000s, so this could be a trend.

At the same time, instrumentalness, the prediction of whether a track has no vocals, has gone to near 0. This tells me that more instrumental songs used to have a chance at hitting the top 10, whereas now they have much less chance.

Valence & Acousticness:

# Valence & Acousticness
fig, ax = plt.subplots()
ax.plot(year['year'], year['valence'])
ax.plot(year['year'], year['acousticness'])
ax.set_xlim(1958, 2025)
ax.set_ylim(0,.85)

# Add in title and subtitle
ax.set_title("""I've got a negative feeling...""")
ax.text(x=.08, y=.86, 
        s="Average features from top ten singles, 1958-2022, Scale 0-1", 
        transform=fig.transFigure, 
        ha='left', 
        fontsize=20, 
        alpha=.8)

# Label the lines directly
ax.text(x=.78, y=.66, s="""Valence""", 
        transform=fig.transFigure, ha='left', fontsize=20, alpha=.7)
ax.text(x=.73, y=.30, s="""Acousticness""", 
        transform=fig.transFigure, ha='left', fontsize=20, alpha=.7)

Output:

Some features obviously trended downward. Valence, the musical positiveness of a track, has been steadily decreasing. At the same time, acousticness (i.e., music produced with a physical instrument that vibrates the air) has been decreasing.

I associate acoustic music with more positive feelings. I'm not sure if this is generally true. A good change is that acousticness jumped in 2019.

Danceability & Energy:

# Danceability & Energy
fig, ax = plt.subplots()
ax.plot(year['year'], year['danceability'])
ax.plot(year['year'], year['energy'])
ax.set_xlim(1958, 2025)
ax.set_ylim(0,.85)

# Add in title and subtitle
ax.set_title('Hmm... 10% increase in both')
ax.text(x=.08, y=.86, 
        s="Average features from top ten singles, 1958-2022, Scale 0-1", 
        transform=fig.transFigure, 
        ha='left', 
        fontsize=20, 
        alpha=.8)

# Label the lines directly
ax.text(x=.7, y=.795, s="""Energy""", 
        transform=fig.transFigure, ha='left', fontsize=20, alpha=.7)
ax.text(x=.685, y=.61, s="""Danceability""", 
        transform=fig.transFigure, ha='left', fontsize=20, alpha=.7)

Output:

Last, both energy and danceability (how danceable a track is) have been trending up, although not as dramatically as we have seen in other features. Top tracks today are more than 10% danceable since 1958.

Energy has been closely correlated with danceability. Something in me wants to investigate where the lines cross (the mid-2010s, early 2000s, and late 90s) to see if there is a signal there of something. I'll leave that for another day.

Unfortunately, all of this exploration didn't get me closer to understanding how to find better music for myself. I did get some important perspective, though.

---

This is not the end of the Project we now need to create a playlist of Top songs for each year. The notebook can be found here.

I am also thinking of deploying it in an app using Streamlit and containerizing the application using Docker. So I think all this will take two more parts to complete.

Until then take care, Happy Learning!

Thanks for Reading!