cURL CLI Emulate Web Browser With User-Agent Proxy Cookies And Custom Headers

In some cases, while downloading files from a remote server, or just ping URL or a web page using cURL command, we get a fail message i.e. Error 403 The access is forbidden error.

The most reason comes, when the server applies some integrity checks for the user client app before allowing it to access the server contents or applying web firewall rules.

Any way we can apply the following adjustments to the cURL CLI to emulate the web browser as possible, to use in some specific request cases.

Prepare CURL Header Parameters

You can prepare cURL command parameters as the following

cURL and Collect browser data

1- Visit your target web page from a chromium-based desktop web browser as Google-Chrome.

2- Right-Click on the web page, and select “Inspect” from the appearing context menu, as the example shown in the below screenshot. which will open the chrome Developer toolbar.

3- From the Developer toolbar select the “Network” tab, Then refresh your page again while the “Network” tab is active.

4- Press on the first request URL from the “Network” tab Under the “Name” label, and Right-Click Then Select “Copy” from the pop-up menu, Then “Copy as cURL (bash)” as shown in the below screenshot.

The copies request will be the same as the outputs here

curl 'https://www.chromium.org/chromium-projects/' \
  -H 'authority: www.chromium.org' \
  -H 'pragma: no-cache' \
  -H 'cache-control: no-cache' \
  -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-user: ?1' \
  -H 'sec-fetch-dest: document' \
  -H 'referer: https://www.chromium.org/' \
  -H 'accept-language: en-US,en;q=0.9,ar;q=0.8,tr;q=0.7' \
  -H 'cookie: _ga=GA1.2.579855328.1643135453; _gid=GA1.2.1955147752.1645531691' \
  --compressed

As we can see, here we collect good cURL header parameters to use in our command at the CLI terminal, It includes a Browser User-agent, Accept-Encoding, and a true Cookies value.

cURL Add Proxy Parameters

We can append the proxy parameters to the cURL output above as the following

--proxy http://proxy_server:proxy_port --proxy-user proxyusername:proxypassword

When using Socks Proxy

--socks5 socks_server_ip:socks_port --proxy-user proxyusername:proxypassword

cURL Skip SSL Certificate Verification

We can append the -k / –insecure parameter to skip the cURL SSL Certificate Verification and allow insecure server connections when using SSL URLs.

--insecure

cURL Allow Cookies

To allow a website application to write cookies on our side, we will append –cookie-jar parameter with the path to the cookie file where to store the cookie value.

--cookie-jar /path/to/cookiefile.txt

And we can send a cookie value to the server-side with our request by pointing to the same cookie file too.

--cookie  /path/to/cookiefile.txt

cURL User-Agent

We can send the User-Agent string to the server-side as an individual parameter, by appending the following to our command.

--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

The Complete cURL Command

curl 'https://www.chromium.org/chromium-projects/' \
  -H 'authority: www.chromium.org' \
  -H 'pragma: no-cache' \
  -H 'cache-control: no-cache' \
  -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-user: ?1' \
  -H 'sec-fetch-dest: document' \
  -H 'referer: https://www.chromium.org/' \
  -H 'accept-language: en-US,en;q=0.9,ar;q=0.8,tr;q=0.7' \
  -H 'cookie: _ga=GA1.2.579855328.1643135453; _gid=GA1.2.1955147752.1645531691' \
  --compressed \
  --proxy http://proxy_server:proxy_port --proxy-user proxyusername:proxypassword \
  --insecure \
  --cookie  /path/to/cookiefile.txt \
  --cookie-jar /path/to/cookiefile.txt \
  --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" https://The_Target_URL

You can take a look at the cURL PHP library by visiting the tutorial: PHP Lib cURL workshop and action tips