ABSTRACT In this paper, performance of joint transmit and receive antenna selection in each hop of dual hop amplify‐and‐forward relay network is analyzed over flat and asymmetric Nakagami‐ m fading channels. In the network, source, relay, and destination are equipped with multiple antennas. By considering relay location, we derive exact closed‐form cumulative distribution function, moment generating function, moments of end‐to‐end signal‐to‐noise ratio and closed form symbol error probability expressions for fixed and channel state information‐based relay gains. We also derive the asymptotical outage probability and symbol error probability expressions to obtain diversity order and array gain of the network. Analytical results are validated by the Monte Carlo simulations. Copyright © 2012 John Wiley & Sons, Ltd.