Methods of Optimization and Systems Analysis for Problems of Transcomputational Complexity