Time Complexity of Detach() in torch "[R]"

Hi,

So I have an empirical observation that when I train a large model vs. the same model in a staggered fashion, .i.e. some layers are frozen and others receive a gradient update, the latter takes more training time although the number of trainable parameters are less. This leads me to suspect that the detach() operation is the culprit. I cannot find much resources online to help me with understanding the time-complexity of the detach() operation in torch. Did anyone …

complexity machinelearning