Q1. As we know that Lasso regression can be used for feature selection while ridge cannot be, why should we even consider using ridge?

Ans: Though lasso regression can be used for feature selection while ridge regression cannot be, it comes at a huge computational cost. Since it does not convert into a nice invertible function, it is to be solved using an iterative process which has significantly more computational requirements compared to ridge regression which demands a simple tweak to the simple linear regression solution and can be converted to an invertible matrix and can thus be solved using matrix operations and thus has significantly lower computational costs associated with it.

Q2. Consider a situation where the target variable is not linearly related to a predictor variable, can you create a linear regression model for the same? What exactly does ‘linear’ in linear regression mean?

Ans: The ‘linear’ in linear regression does not stand for the relation between the target variable and the predictor variable. In fact, it stands for the coefficients of the predictor terms in the linear regression solution. It is entirely logical to have a linear regression solution as z= a*sinx + b*cosy, however, it will not be a linear regression model if the relation can be defined as

y=2a*y*x^*sinx + y^2*x^2*sinxy.

Q3: Explain the process of Best Subset Selection.

Ans: In the Best Subset Selection algorithm, we start with 0 features, i.e. a null model M0 with no features. Now, as we increase the number of features, we consider every model that has all combinations of a certain number of features. and select a model which results in the least RSS (or largest R2). This gives us a model Md with d features. We continue this iteration by increasing the value of d by one till you reach d is equal to the number of features in the dataset and find the models M0, M1, M2,....., Mp. Out of all these models M0, M1, M2,....., Mp, select the best one, as measured by measures such as Cp, AIC, BIC, Adjusted R2 or mean cross-validated error.