The syntax above is close but (as you noticed) not exactly right. I'll try to step through it...

First is to define the custom factor. It's most common for factors to have a single output value but they can have multiple output values each with different names. One could have a single output and name it but this isn't typical. Both of the following result in the same factor but just referenced a bit different

class MyFactor(CustomFactor):  
    inputs = [Fundamentals.ebit.latest,  
              Fundamentals.net_ppe.latest]  
    outputs = ['return_on_capital']  
    window_length = 1  
    def compute(self, today, asset_ids, out, ebit, net_tangible):  
        out.return_on_capital[:] = ebit/net_tangible

# The above named output factor is instantiated like this  
roc = MyFactor().return_on_capital

class Return_On_Capital(CustomFactor):  
    inputs = [Fundamentals.ebit.latest,  
              Fundamentals.net_ppe.latest]  
    window_length = 1  
    def compute(self, today, asset_ids, out, ebit, net_tangible):  
        out[:] = ebit/net_tangible

# The above typical single output factor is instantiated like this  
roc = Return_On_Capital()  

To make a filter out of these one can apply simple logical operators. This was correct in the original code. I think it didn't like the columns definition.

def make_pipeline():  
   return_on_capital = MyFactor().return_on_capital

   pipe = Pipeline(  
           columns={  
                'return_on_capital':  return_on_capital  
            },  
           screen=(return_on_capital>0.5)  
        )  
   return pipe