The two ideas are not mutually exclusive.

Taking up your cross can mean following Jesus to overcome your own human weaknesses and faults. That is a difficult process.

In the case of the early Christians it was even more literal. Tradition is that most of the apostles died on the cross.