If the Apple TV doesn't respond to commands from a device with the remote app installed, the device may be unpaired or it may need to be re-paired. 

Device Is Unpaired

You can check to see if your phone or tablet is listed under 'Remote App and Devices' by going to Settings > Remotes and Devices > Remote App and Devices. If it is not listed, you will need to pair the device. 

Open the remote app and select 'Add a Device.' Open 'Remote App and Devices' on the Apple TV and select the device. Enter the four-digit number provided by the Apple TV into your mobile device. 

Device Needs To Be Re-Paired

Select the device in 'Remote App and Devices' and choose the 'Forget' option. Next, using the steps listed above, re-pair the device.