Users are able to connect to report manager but not able to connect to report server.

You need to register IIS before installing SQL Server Reporting Services. However, there can be situations where you need to re-register it.  In order to re-register ISI follow the steps below:
1. Open the Command Prompt
2. Change directory as follows:
For 32-bit machines change to:
<WindowsDir>\Microsoft.NET\Framework\<version number>\
For 64-bit machines change to:
<WindowsDir\Microsoft.NET\Framework64\<version number>\
3. Then run the command ‘aspnet_regiis.exe -i’ and press enter