TF-IDF (Term Frequency Time Inverse Document Frequency) Concept

Content-based systems, such as Wikipedia, rely on user input. When you such a certain item, most likely several items will appear, some not related to what is searched. To select the most relevant information, we use the concept of TF-IDF (Term Frequency Time Inverse document frequency).

It identifies the number of times a certain word appears in a text. It calculates and determines whether the word is found in a document and returns an inverse of how many documents have it over those that do not. The log that results from the computation narrows down the search to a few results, as shown in the code below:

The system highlighted below uses metadata to suggest movies for a user based on their history of rating movies. To build this system, we will answer these three questions: What metrics are we using to rate the movies? What is the average score for the rated movies? How do we sort the movies in regard to the score? Load the following dataset into your compiler.

In this case, we are going to use the user’s rating as the primary source of data to come up with -the recommendations. Although, this system has a challenge as the movie’s popularity is not taken into account. As a result, it is easy to rate a movie with fewer voters with a higher rating than with many voters but with a lower rating. Fewer people rating a movie gives a rather skewed reflection of the movie and not the actual one. Thus, the more voters there are, the more accurate the feedback will be. Bearing such in mind, it is imperative to develop a weighted rating that considers not only the rating but also the number of voters. The following mathematical formula will be used to calculate the weighted rating:

• v – amount of votes;
• m is the minimum votes required to be listed in the chart;
• R is the average rating of the movie, And
• C is the mean vote across the whole report

Having been provided with the values of v (vote_count) and R (vote_average), the system will be prompted to calculate the value of C. As a system developer, one is tasked with the responsibility of determining the least number of votes that a movie must garner so as to qualify to be listed. The value of M differs from one person to another. It is based on the opinion of an individual or a collective agreement. For this particular project, we will utilize the 95^th percentile as our cut-off. In short, for a movie to be listed, it must have garnered more votes than 95% of the listed movies. You will realize that the lower the percentile, the higher the number of movies and the higher the percentile, the lower the number of movies. To begin with, the value of C is calculated as shown in the figure below. In IMDB, on a scale of 10, the average rating of a movie is 5.6.

The succeeding step is to enumerate the amount of votes, m, garnered by a movie in the 95^th percentile. Through the panda library, this is achieved through the quantile() method as shown below:

Then, the movies that meet the set threshold are listed based on their average rating score on the chart. The code below illustrates how the system is set to filter out the movies that meet the set requirements.

So as to ascertain that the DataFrame, q_movies, created is not dependent on the initial metadata, the copy () method is used. Thus, any alterations enacted upon the q_movies DataFrame have no significant effect on the metadata. There are over four thousand movies that satisfy the requirements to be listed; thus, each qualified movie’s metric must be calculated. In order to achieve this, a weighted_rating () function is defined. Also, another feature score, weighted () rating, must be defined so as to compute the value in the data frame.

The final stage is to arrange the data frame based on the ratings of the top 5 movies, detailing their titles, average ratings, and number of voters.

In some instances, a user may want to view movies whose plots are closely related, such as aliens and zombies, among many others. This system will try to recommend movies whose plots are closely related but list them on the chart based on the average weighted score garnered using the formula discussed earlier on. The description of the plot is displayed under the overview heading in the dataset, specifically under the metadata using the method below:

Applying this program will result in the output being coherent with the top five movies displayed on IMDB.