<ReportServerUrl></ReportServerUrl>
<ReportServerVirtualDirectory>ReportServer</ReportServerVirtualDirectory>
<ReportBuilderTrustLevel>FullTrust</ReportBuilderTrustLevel>
After changing the ReportServerUrl tag to contain both the server URL and the virtual directory, it could then connect to the remote server.
<ReportServerUrl>http://Server/ReportServer/</ReportServerUrl>
<ReportServerVirtualDirectory></ReportServerVirtualDirectory>
<ReportBuilderTrustLevel>FullTrust</ReportBuilderTrustLevel>
No IP addresses were added or was any IIS settings that I could tell. Leaving the ReportServerUrl tag blank should allow for it to find the server locally.