First you will need to make a copy of the default layout page AccessDenied.aspx which is located at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS
For example, you can call it AccessDeniedNew.aspx. This new page layout should stay in “LAYOUTS” folder.
Login to your server as a farm administrator. Start SharePoint 2010 Management Shell. Run the following code:
$site = get-spsite "http://YOUR_SITE_URL"
$webApp = $site.WebApplication
$webapp.UpdateMappedPage(1, "/_layouts/AccessDeniedNew.aspx")
$webapp.Update()
To switch back to the default page, you can use the following code
$site = get-spsite "http://YOUR_SITE_URL"
$webApp = $site.WebApplication
$webapp.UpdateMappedPage(1, $null)
$webapp.Update()
You can get more information about the custom page from Yaroslav’s blog
In the method of $webapp.UpdateMappedPage, 1 is for Access Denied Page. See the following table for other type of pages
AccessDenied = 1
Confirmation = 2
Error = 3
Login = 4
RequestAccess = 5
Signout = 6
WebDeleted = 7
You can use Set-SPCustomLayoutsPage and Get-SPCustomLayoutsPage to reach the same goal as well.