The common returns, which are calculated for a backtest, are simply the correlation of the net portfolio returns to five common factors (momentum, size, value, short term reversal, and volatility). It's measuring how much of a portfolio's returns can be explained by the 'well known' or common factors. The specific returns are simply the excess returns beyond this common exposure. On a given day, the total returns = common returns + specific returns.

The questions was "Is [specific returns] relative to the money invested or is it relative to if you had all your cash invested all the time in the benchmark?". The specific returns are calculated from the actual net return of the entire portfolio. If the portfolio value increased 1% then that is what is used to calculate specific returns. It is independent of leverage and actual cash value and doesn't look at how those returns were realize. It simply uses the net daily end of day returns.

Hope that makes sense.

Thanks for your response.
But I'm pretty shure it is not entirely correct. Here is my result with (an exagerated amount of leverage)-I go allin all the time.
The result over 1 year (2019) is : 122. 159% (over 100000%), most of which are specific returns . I'm pretty shure this cannot be reached in the real world and this not the result of net return according the definition you gave above.
Leverage plays a (huge) role. Therefor once more the question: how exactly are the returns determined?

The daily return of a backtest is simply the gain in portfolio value from the previous day to the current current. It's calculated after the close each day. It can be represented like this

portfolio_value = sum_of_long_positions + sum_of_short_positions + cash  
daily_return = (portfolio_value_today - portfolio_value_yesterday) / abs(portfolio_value_yesterday)

Notice this is an arithmetic return and not a log return. To get the total return, the log of the daily returns are summed.

This definition of 'daily returns' is independent from whether the strategy trades or even if it has any positions. In fact, if the portfolio didn't trade at all and was always in cash, then the daily return would be zero. One other point to notice. Dividends are automatically added to the portfolio by the backtest engine which increases cash. Likewise, commissions are subtracted from cash.

The reason you are seeing huge returns is the algorithm is using huge leverage. In theory, if you could borrow enough to support that leverage, you would indeed probably see those returns.

Maybe, the sentence above is poorly worded [specific returns are] independent of leverage and actual cash value and doesn't look at how those returns were realized . I meant it to imply the calculation for specific returns doesn't take into account leverage or cash value. It only takes into account the portfolio returns. However, both those will generally impact returns so, indirectly, specific returns can be impacted by leverage. In this case a lot.

Once again, thanks for your explanation. This made it a lot clearer for me.

I think I can see where the problem is and, correct me if I'm wrong, the calculations are not correct when using leverage.

When leverage is used it is added directly to the portfolio_value_today and not to the portfolio_value_yesterday. This is of course corrrect in an absolute sense because this was the value of the portfolio yesterday but if you add it to the portfolio_value_today then you create returns out of nothing.
E.g. if my portfoliovalue (yesterday) is 100 and I would invest 50 in a new security (with leverage) and this security and my portfolio of 100 would stay at the same price during that day I would have made a return of 50% according to the definition above. So return value is added out of nothing.

I think this would be an improved calculation for daily_return:
daily_return = (portfolio_value_today - (portfolio_value_yesterday + leverage_today)) / abs(portfolio_value_yesterday + leverage_today)

So the excess cash or leverage used to invest above your portfolio_value should be taken into account for calculating the returns. Otherwise leverage is added to the returns directly which cannot be correct.

I would like to know your view on this.

When stocks are purchased the portfolio cash is always reduced by the purchase price plus any commission. When stocks are sold the portfolio cash is increased by the sale price less any commission. This is exactly what would happen in a real brokerage account.

Therefore the comment above isn't exactly correct

if my portfolio value (yesterday) is 100 and I would invest 50 in a new security (with leverage) and this security and my portfolio of 100 would stay at the same price during that day I would have made a return of 50% according to the definition above. So return value is added out of nothing.

Let's follow the chain of events when purchasing a stock (long). For simplicity assume (for now) that the end of day closing price of our stock XYZ stays the same at $100/share. Also assume commissions are $0.

day 1 - opening balance of $100,000  
        cash: $100,000  
        stock value: $0  
        portfolio value: $100,000  (100000 + 0)  
        leverage: 0   (0 / 100000)

day 2 - purchase shares with existing cash  
        buy 1000 shares XYZ @ $100/share  
        cash: $0  
        stock value: $100,000  
        portfolio value: $100,000    (0 + 100000)  
        leverage: 1  (100000 / 100000)  
        daily returns: 0%   (100000 - 100000) / 100000

day 3 - purchase shares on margin  
        buy 1000 shares XYZ @ $100/share  
        cash:  -$100,000  
        stock value: $200,000  
        portfolio value: $100,000    (-100000 + 200000)  
        leverage: 2     (200000 / 100000)  
        daily returns: 0%    ((100000 - 100000) / 100000)

day 4 - purchase more shares on margin  
        buy 1000 shares XYZ @ $100/share  
        cash:  -$200,000  
        stock value: $300,000  
        portfolio value: $100,000    (-200000 + 300000)  
        leverage: 3    (300000 / 100000)  
        daily returns: 0%    ((100000 - 100000) / 100000)

day 5 - sell all the shares  
        sell 3000 shares XYZ @ $100/share  
        cash:  $100,000  
        stock value: $0  
        portfolio value: $100,000   (100000 + 0)  
        leverage: 0   (0 / 100000)  
        daily returns: 0%    ((100000 - 100000) / 100000)

A few things to note about the above. 1) the portfolio value always stays at $100,000 even as more shares are bought with margin and leverage increases. 2) the daily returns, and net return over the 5 days, is 0%. 3) the cash in the account goes negative when leverage is over 1. A negative cash amount implies funds which are borrowed and need to be paid back.

So, what does a similar transaction look like if the stock price goes up? Again assume no commissions but now our XYZ goes up to $101. First with a leverage of 1.

day 1 - opening balance of $100,000  
        cash: $100,000  
        stock value: $0  
        portfolio value: $100,000 (100000 + 0)  
        leverage: 0   (0 / 100000)

day 2 - purchase shares with existing cash  
        buy 1000 shares XYZ @ $100/share  
        cash: $0  
        stock value: $100,000  
        portfolio value: $100,000   (0 + 100000)  
        leverage: 1  (100000 / 100000)  
        daily returns: 0%  (100000 - 100000) / 100000

day 3 - sell all the shares at $101  
        sell 1000 shares XYZ @ $101/share  
        cash:  $101,000  
        stock value: $0  
        portfolio value: $101,000   (101000 + 0)  
        leverage: 0   (0 / 101000)  
        daily returns: 1%   ((101000 - 100000) / 100000)

Looks good. We made 1%. Now look at the same transaction but buy on margin for a leverage of 2.

day 1 - opening balance of $100,000  
        cash: $100,000  
        stock value: $0  
        portfolio value: $100,000   (100000 + 0)  
        leverage: 0   (0 / 100000)

day 2 - purchase 2000 shares using margin  
        buy 2000 shares XYZ @ $100/share  
        cash: $-100,000  
        stock value: $200,000  
        portfolio value: $100,000   (-100000 + 200000)  
        leverage: 2   (200000 / 100000)  
        daily returns: 0%   (100000 - 100000) / 100000

day 3 - sell all the shares at $101  
        sell 2000 shares XYZ @ $101/share  
        cash:  $102,000  ((2000x101) + (-100000))  
        stock value: $0  
        portfolio value: $102,000   (102000 + 0)  
        leverage: 0   (0 / 102000)  
        daily returns: 2%   ((102000 - 100000) / 100000)

Notice that by adding leverage we multiply our returns. Even though we do not directly increase our net portfolio value with leverage we do multiply any gains. This is why using margin can be a great tool. However, notice I intentionally used the word multiply and not increase. If one had daily losses (eg -1%) leverage will multiply those and be -2%.

When comparing strategies it's typically best to have an 'apples to apples' comparison and ensure one's leverage is the same. Try to keep leverage around 1. In the real world, individuals are typically limited to a leverage of 2 by one's broker (and the SEC). Additionally, borrowing money is not free. The backtester doesn't take any borrowing costs into account. So, while the returns on highly leveraged portfolios are technically correct they would probably not be realized in live trading.